标签归档:CentOS

Python使用Kerberos认证查询Impala

最近做QoS报告,数据来源于Impala,客户端认证采用的是Kerberos。
Impala是Cloudera公司开发并开源的一款基于HDFS/Hbase的MPP SQL引擎,它提供SQL语义,能够查询存储在Hadoop的HDFS和HBase中的PB级大数据。Kerberous本身是一个网络认证授权协议,借由中心服务器认证,对通信双方的客户端/服务端进行授权而不需要传递双方的密码。Kerberos的认证流程比较有意思,分为三个阶段

  • 客户端认证
    • 1 客户端发送自己用户名
    • 2 认证服务器返回使用客户端密钥加密的Client/TGS会话密钥和使用票据授权服务器密钥加密的TGT, 包括sessions key,用户信息及有效期
    • 3 客户端使用自己的密钥解密出Client/TGS会话密钥
  • 服务授权
    • 1 客户端发送两条消息:接收到的TGT和所请求的服务ID;使用Client/TGS会话密钥加密的用户ID和时间戳
    • 2 票据授权服务器使用自己的密钥解密TGT得到客户端的Client/TGS会话密钥,然后使用它解密出用户ID并进行认证。返回使用所请求服务端密钥加密的client-server票据和使用Client/TGS会话密钥加密的Client/Server会话密钥
    • 3 客户端使用Client/TGS会话密钥(Client/TGS Session Key)解密出Client/Server会话密钥
  • 服务请求
    • 1 客户端发送两条消息:使用所请求服务端密钥加密的client-server票据及使用Client/Server会话密钥加密的用户ID和时间戳
    • 2 服务端使用自己的密钥解密client-server票据从而得到Client/Server会话密钥,使用该密钥解密获得用户信息并认证。返回使用Client/Server会话密钥的新时间戳
    • 3 客户端使用Client/Server会话密钥解密该消息,认证结束并请求服务
    • 4 服务端提供服务

在CentOS上安装Kerberos

yum install krb5-devel pam_krb5 krb5-libs krb5-workstation

编辑配置

vim /etc/krb5.conf

配置KDC,认证服务器

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tkt_enctypes = rc4-hmac
default_tgs_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
[realms]
EXAMPLE.COM = {
default_domain = example.com
kdc = kdc01.example.com
kdc = kdc02.example.com
admin_server = adc01.example.com
admin_server = adc02.example.com
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

测试一下

[root@localhost rc]# kinit abc.xyz@EXAMPLE.COM
Password for abc.xyz@EXAMPLE.COM:

注意这个配置文件每行前面的空格被删掉,是因为在VirtualBox里面每行开头有莫名其妙的乱码,Linux下并不可见,在EditPlus下面才发现,否则会乱报错

kinit: Improper format of Kerberos configuration file while initializing Kerberos 5 library
kinit: Cannot find KDC for realm "EXAMPLE.COM" while getting initial credentials

查看一下认证的ticket

[root@localhost vagrant]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: abc.xyz@EXAMPLE.COM

Valid starting       Expires              Service principal
09/21/2017 08:30:50  09/21/2017 18:30:50  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 09/28/2017 08:30:42

这个ticket在28号就会过期了,到时候又要输入密码,这样也不便于自动化程序使用。可以使用ktutil创建keytab文件

$ ktutil
ktutil:  addent -password -p abc.xyz@EXAMPLE.COM -k 1 -e RC4-HMAC
Password for abc.xyz@EXAMPLE.COM:
ktutil:  wkt abc.xyz.keytab
ktutil:  q
$ ls
abc.xyz.keytab

测试一下

$ kinit -kt abc.xyz.keytab abc.xyz@EXAMPLE.COM
$ klist -k abc.xyz.keytab
Keytab name: FILE:abc.xyz.keytab
KVNO Principal
---- --------------------------------------------------------------------------
  1 abc.xyz@EXAMPLE.COM

之后便可以使用kinit自动更新ticket了。注意,如果更换了密码,需要重新生成新的keytab。
另外,相同用户生成的授权ticket在任意一台机器上都是相同的, kinit时会自动同步回来的。
公司的大数据平台使用Hue来提供基于web界面的查询,Impala也支持使用ODBC方式查询。在Python里使用的是impyla来查询,首先安装sasl的依赖

yum install libgsasl-devel cyrus-sasl-devel cyrus-sasl-gssapi
pip install impyla thrift_sasl

测试脚本

from impala.dbapi import connect
conn = connect(host="impalad.example.com", port=21050, auth_mechanism='GSSAPI', kerberos_service_name='impala', database='acme')
cur =  conn.cursor()
cur.execute(r'SELECT * FROM acme WHERE dt="2017-09-12" LIMIT 5')
print(cur.fetchall())

运行下

python test.py

如下报错,则是服务器不能连接,检查一下网络,DNS/hosts及VPN

thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not connect to ('impalad.example.com', 21050)")

如下报错,CentOS则是需要cyrus-sasl-gssapi模块

thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found'")

参考链接:
Impala:新一代开源大数据分析引擎
大数据时代快速SQL引擎-Impala
CDH 5.2中Impala认证集成LDAP和Kerberos
Kerberos
Configuring Kerberos Authentication for Windows
Speaking Kerberos with KNIME Big Data Extensions

Vagrant CentOS 共享目录挂载问题解决

一直以来都是使用Vagrant与VirtualBox运行CentOS系统来搭建环境,然而有一天突然出现Windows下面的目录无法映射进去了,报错:

D:\project\vagrant\centos64php56>vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
  default: Adapter 1: nat
  default: Adapter 2: hostonly
==> default: Forwarding ports...
  default: 80 (guest) => 8080 (host) (adapter 1)
  default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
  default: SSH address: 127.0.0.1:2222
  default: SSH username: vagrant
  default: SSH auth method: private key
  default: Warning: Connection reset. Retrying...
  default: Warning: Connection aborted. Retrying...
  default: Warning: Connection reset. Retrying...
  default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
[default] GuestAdditions versions on your host (5.1.26) and guest (4.3.6) do not matc
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirrors.btte.net
 * epel: repo.fedoralinux.ir
 * extras: mirrors.btte.net
 * updates: mirrors.btte.net
No package kernel-devel-2.6.32-358.23.2.el6.x86_64 available.
Package gcc-4.4.7-18.el6.x86_64 already installed and latest version
Package binutils-2.20.51.0.2-5.47.el6_9.1.x86_64 already installed and latest version
Package 1:make-3.81-23.el6.x86_64 already installed and latest version
Package 4:perl-5.10.1-144.el6.x86_64 already installed and latest version
Package bzip2-1.0.5-7.el6_0.x86_64 already installed and latest version
Nothing to do
Copy iso file C:\Program Files\Oracle\VirtualBox\VBoxGuestAdditions.iso into the box
Installing Virtualbox Guest Additions 5.1.26 - guest version is 4.3.6
Verifying archive integrity... All good.
Uncompressing VirtualBox 5.1.26 Guest Additions for Linux...........
VirtualBox Guest Additions installer
Removing installed version 5.1.26 of VirtualBox Guest Additions...
vboxadd.sh: Stopping VirtualBox Additions.
Copying additional installer modules ...
Installing additional modules ...
vboxadd.sh: Starting the VirtualBox Guest Additions.
Failed to set up service vboxadd, please check the log file
/var/log/VBoxGuestAdditions.log for details.
An error occurred during installation of VirtualBox Guest Additions 5.1.26. Some func
In most cases it is OK that the "Window System drivers" installation failed.
vboxadd.sh: Starting the VirtualBox Guest Additions.
vboxadd.sh: failed: Look at /var/log/vboxadd-install.log to find out what went wrong.
vboxadd.sh: failed: modprobe vboxguest failed.
==> default: Checking for guest additions in VM...
  default: The guest additions on this VM do not match the installed version of
  default: VirtualBox! In most cases this is fine, but in rare cases it can
  default: prevent things such as shared folders from working properly. If you see
  default: shared folder errors, please make sure the guest additions within the
  default: virtual machine match the version of VirtualBox you have installed on
  default: your host and reload your VM.
  default:
  default: Guest Additions Version: 4.3.6
  default: VirtualBox Version: 5.1
==> default: Configuring and enabling network interfaces...
  default: SSH address: 127.0.0.1:2222
  default: SSH username: vagrant
  default: SSH auth method: private key
==> default: Mounting shared folders...
  default: /home/rc => D:/project/vagrant/centos64php56
Vagrant was unable to mount VirtualBox shared folders. This is usually
because the filesystem "vboxsf" is not available. This filesystem is
made available via the VirtualBox Guest Additions and kernel module.
Please verify that these guest additions are properly installed in the
guest. This is not a bug in Vagrant and is usually caused by a faulty
Vagrant box. For context, the command attempted was:

mount -t vboxsf -o uid=500,gid=500 home_rc_ /home/rc

The error output from the command was:

/sbin/mount.vboxsf: mounting failed with the error: No such device

这里提示虚拟机里CentOS的VBoxGuestAdditions与VirtualBox的版本匹配,需要升级:

[default] GuestAdditions versions on your host (5.1.26) and guest (4.3.6) do not matc

应该是升级了VirtualBox导致的。然而自动安装新插件失败:

vboxadd.sh: Starting the VirtualBox Guest Additions.
Failed to set up service vboxadd, please check the log file
/var/log/VBoxGuestAdditions.log for details.
An error occurred during installation of VirtualBox Guest Additions 5.1.26. Some func
In most cases it is OK that the "Window System drivers" installation failed.
vboxadd.sh: Starting the VirtualBox Guest Additions.
vboxadd.sh: failed: Look at /var/log/vboxadd-install.log to find out what went wrong.
vboxadd.sh: failed: modprobe vboxguest failed.

导致共享目录无法映射。但是虚拟机仍然是启动成功的,可以ssh进去,或者使用sftp挂载。
Google了下有说是VirtualBox bug的,也有说是Windows问题的,各种折腾不能解决。升级插件也无效:vagrant plugin install vagrant-vbguest。重新安装VirtualBox和Vagrant,并不会影响现有虚拟机及网络配置,但不能解决问题。
直到看到这篇文章,决定从CentOS入手解决。
启动虚拟机后,使用sftp上传VBoxGuestAdditions.iso,ssh进入手动安装:

$ sudo
$ mount VBoxGuestAdditions.iso -o loop /mnt
$ cd /mnt
$ sh VBoxLinuxAdditions.run

安装失败,查看日志:

Building the main Guest Additions module                   [FAILED]
(Look at /var/log/vboxadd-install.log to find out what went wrong)

vboxadd-install.log日志:

/tmp/vbox.0/Makefile.include.header:97: *** Error: unable to find the sources of your current Linux kernel. Specify KERN_DIR=<directory> and run Make again.  Stop.

参照这篇文章,查看kernel版本

$ rpm -qa kernel\* | sort
kernel-2.6.32-358.23.2.el6.x86_64
kernel-devel-2.6.32-696.10.2.el6.x86_64
kernel-firmware-2.6.32-358.10.2.el6.noarch
kernel-headers-2.6.32-696.10.2.el6.x86_64

$ uname -r
2.6.32-358.10.2.el6.x86_64

其实一开始是更多版本不匹配的,尝试更新kernel:

yum update
yum install kernel-headers kernel-devel

结果部分更新失败:

Warning: No matches found for: kernel-devel
No Matches found

参照这里解除版本锁定,设置enabled = 0:

$ vim /etc/yum/pluginconf.d/versionlock.conf

再次运行升级kernel就可以了。安装成功后,重启后,再次运行sh VBoxLinuxAdditions.run 就可以了。事实上Vagrant启动时就会自动安装VBoxGuestAdditions:

[default] GuestAdditions 5.1.26 running --- OK.

Vagrant升级到后发现vagrant up初始化下载box卡住了,那是你的vagrant版本太高与对应的powershell版本对应不上,可以下载最新的powershell安装即可。
如果box下载很慢,可以参照这里的方法自我映射取得url单独下载:

https://app.vagrantup.com/box-cutter/boxes/centos73
==> https://atlas.hashicorp.com/box-cutter/boxes/centos73/versions/2.0.21/providers/virtualbox.box

参考链接:
Guest Additions Version error on VirtualBox5
VirtualBox下挂载共享文件目录问题处理
Resolving GuestAdditions version mismatch in vagrant/homestead vm (failed to mount shared folders / modprobe vboxsf failed)
Problem installing virtualBox guest additions
Warning: No matches found for: kernel-devel
Vagrant up hangs forever on Windows 7, Vagrant 1.9.7, VirtualBox 5.1.22.r11512

Centos 6.4 安装 Python 2.7

终于又开始学Python了,不过这次是在Centos 6.4 上面,也碰到了好多问题。Centos6.4
并不能通过yum安装Python 2.7,系统自带的yum等使用的都是Python2.6.6,将系统的Python软链接指向2.7版本会有各种问题,包括依赖库等等;或者只能创建新的可执行命令,如Python27。最终按照这篇文章的介绍成功安装了Python2.7。
首先安装相关的工具,要不然等下编译Python会报各种各样的错:

$sudo yum groupinstall "Development tools"
$sudo yum install zlib-devel
$sudo yum install bzip2-devel
$sudo yum install openssl-devel
$sudo yum install ncurses-devel
$sudo yum install sqlite-devel

然后下载Python并安装,注意这里是make altinstall而不是make install,参考这里

$ wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tar.xz
$ tar xf Python-2.7.11.tar.xz
$ cd Python-2.7.11
$ ./configure --prefix=/usr/local
$ sudo make 
$ sudo make altinstall

检查一下是不是安装到了/usr/local/bin/python2.7下面去了,后面Python 2.7相关的库也将安装到这里

$ ls -ltr /usr/local/bin/python*

检查一下Python2.6是不是还在/usr/bin/下面

$ ls -ltr /usr/bin/python*

检查一下系统路径变量PATH,保证/usr/local/bin在/usr/bin之前,然后将Python的软链接指向2.7。

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
$ ln -s /usr/local/bin/python2.7 /usr/local/bin/python

这样子普通用户登录的时候就可以使用Python2.7了,而root用户(sudo)仍然使用Python2.6,yum等才不会出错。

$ which python
/usr/local/bin/python
$ python -V
Python 2.7.11

$ sudo -s
which python
#/usr/bin/python
python -V
#Python 2.6.6
exit

安装Python 2.7的包管理工具

$ wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
$ sudo /usr/local/bin/python2.7 ez_setup.py
$ sudo /usr/local/bin/easy_install-2.7 pip

检查一下是不是对了

$ which pip
/usr/local/bin/pip

$ which easy_install
/usr/local/bin/easy_install

在普通用户下面操作使用easy_install 就是安装到Python2.7的目录下面去了

$ easy_install requests

这时候会报错,因为相关的Python 2.7目录并没有写的权限

$ sudo chmod 664 /usr/local/bin

或者

sudo /usr/local/bin/easy_install-2.7 requests

注意:如果是在root用户或者sudo命令下,使用的仍然是Python 2.6,所以必须要指明使用那个版本的easy_install。
接下来就可以愉快的使用pip安装Python2.7相关的库了。

参考链接:
Installing python 2.7 on centos 6.3. Follow this sequence exactly for centos machine only
How To Set Up Python 2.7.6 and 3.3.3 on CentOS 6.4
How to install Python 2.7 and Python 3.3 on CentOS 6
Centos 6.4 python 2.6 升级到 2.7
CENTOS 6.5 安装 Python 2.7 总结
Difference in details between “make install” and “make altinstall”
Common Python Tools: Using virtualenv, Installing with Pip, and Managing Packages