标签归档：impala

Django + Celery处理异步任务/计划任务

在Web开发的过程中，难免有一些操作是耗时任务，有时候我们不想阻塞Web进程，希望提交到后台处理，有结果了再通知前端；有时候用户不需要实时结果，可以异步处理的，比如发送邮件通知；有时候我们希望定时执行某些任务，复用已有的一些Web代码。对于第一种情况，可以是RPC或调用其他接口处理；第二种情况则是放入队列，由后台进程异步处理；第三种情况可以是在定时任务内处理，也可以是触发消息放入队列，由后台进程任务处理，同第二种情况。对于第一种情况，也可以是放入队列由后台进程处理，Web前端，定时轮询队列/接口是否有结果。这样三种情况都可以统一为一种情况，即不同来源事件（用户/定时器）触发消息，放入队列由后台任务进程异步处理，异步查询结果。
之前的一个Python项目即类似。不管代码是PHP还是Python思想都是类似，只是不同语言有不同工具/优势来处理。Web前端展示统计报表，后台进程定时查询Impala，分析统计数据；Web前端也可以触发事件比如统计、发送邮件报告等，后台进程监听并处理。Web前端选择Django，是Python写的一个MVT框架，提供了登录认证、管理后台模块，还有命令行助手工具可以生成项目，易于快速搭建并扩展Web站点。后台任务处理选择Celery，是Python写的一个分布式的任务处理平台，支持各种消息队列，比如RabbbitMQ、Redis，并提供Flower监控工具。Django本身是一个Web框架，生产环境最好使用性能更好的HTTP服务器，Gunicorn是Python写的一个WSGI HTTP服务器，用来监听HTTP请求，调用Django。Gunicorn前端最好使用Nginx来做代理，分发请求，结构如下：

安装Python 3和Django，并生成项目框架代码。注意：由于机器上面本来就有Pyton 2.7，所以要重命名下。

yum -y update
yum -y install yum-utils
yum -y groupinstall development
yum install zlib-devel bzip2-devel openssl-devel ncurese-devel expat-devel gdbm-devel readline-devel sqlite-devel libffi-devel
wget https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tar.xz
tar Jxvf Python-3.6.2.tar.xz
mv Python-3.6.2 /opt/Python-3.6.2
cd /opt/Python-3.6.2
./configure --enable-shared --prefix=/usr/local/python3
make && make install
vim /etc/ld.so.conf
/usr/local/python3/lib/
/sbin/ldconfig -v
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python3.6
ln -s /usr/bin/python3.6 /usr/bin/python3
#ln -s /usr/local/python3/bin/python3.6 /usr/bin/python
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
python3 -V
pip3 -V
pip3 install Django
python3 -m django --version
ln -s /usr/local/python3/bin/django-admin /usr/bin/django-admin/
cd /home/rc
django-admin startproject qrd
cd qrd
python3 manage.py startapp master
vim qrd/settings.py

编辑settings.py，监听相应域名、IP的请求

ALLOWED_HOSTS = ['dev.example.com','localhost','127.0.0.1','10.1.*.*]
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'master',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

测试一下

1	`python3 manage.py runserver 0:80`

然后安装Gunicorn

$ pip3 install gunicorn
$ pip3 install greenlet
$ pip3 install ConfigParser
$ ln -s /usr/local/python3/bin/gunicorn /usr/bin/gunicorn

测试运行

$ cd ~/qrd/qrd
$ gunicorn -w 4 qrd.wsgi --bind  unix:/tmp/gunicorn.sock
$ sudo vim /etc/systemd/system/gunicorn.service

创建Gunicorn服务

[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=root
Group=root
WorkingDirectory=/home/rc/qrd/qrd
ExecStart=/usr/bin/gunicorn --access-logfile - --workers 3 --bind unix:/tmp/gunicorn.sock qrd.wsgi
[Install]
WantedBy=multi-user.target

开机启动

systemctl start gunicorn
systemctl enable gunicorn
#systemctl stop gunicorn

接下来安装Nginx

yum install nginx
vim /etc/nginx/conf.d/default.conf

配置Nginx与Gunicorn通过UNIX Sockect通信

server {
    listen       80;
    server_name  localhost;
    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;
    #location / {
    #   root   /usr/share/nginx/html;
    #   index  index.html index.htm;
    #
    location / {
        proxy_pass http://unix:/tmp/gunicorn.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

启动Nginx

systemctl start nginx
systemctl enable nginx

安装MySQL及RabbitMQ

yum install mysql-community-devel
pip3 install mysqlclient
 
 
yum install rabbitmq-server
rabbitmq-server -detached
rabbitmqctl status

创建RabbitMQ vhost

$ rabbitmqctl add_user qrdweb <password>
Creating user "qrdweb" ...
...done.
$ rabbitmqctl add_vhost qrdweb
Creating vhost "qrdweb" ...
...done.
$ rabbitmqctl set_user_tags qrdweb management
Setting tags for user "qrdweb" to [management] ...
...done.
$ rabbitmqctl set_permissions -p qrdweb qrdweb ".*" ".*" ".*"
Setting permissions for user "qrdweb" in vhost "qrdweb" ...
...done.
$ netstat -apn | grep rabbitmq
$ rabbitmqctl status

安装Celery

pip3 install celery
ln -s /usr/local/python3/bin/celery /usr/bin/celery

测试Celery的异步任务worker及计划任务schedule

$ cd /home/qrd/qrd/
$ ls
dashboard  db.sqlite3  manage.py  master  qrd  static
$ celery -A qrd worker -l info
/usr/local/python3/lib/python3.6/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the -u option.
User information: uid=0 euid=0 gid=0 egid=0
  uid=uid, euid=euid, gid=gid, egid=egid,
 -------------- celery@localhost.localdomain v4.1.0 (latentcall)
---- **** -----
--- * ***  * -- Linux-3.10.0-693.2.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core 2017-09-22 08:19:39
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         qrd:0x7fda62e705c0
- ** ---------- .> transport:   amqp://qrdweb:**@localhost:5672/qrdweb
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
  
[tasks]
  . dashboard.tasks.debug_task
[2017-09-22 08:19:39,769: INFO/MainProcess] Connected to amqp://qrdweb:**@127.0.0.1:5672/qrdweb
[2017-09-22 08:19:39,781: INFO/MainProcess] mingle: searching for neighbors
[2017-09-22 08:19:40,811: INFO/MainProcess] mingle: all alone
[2017-09-22 08:19:40,860: WARNING/MainProcess] /usr/local/python3/lib/python3.6/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2017-09-22 08:19:40,860: INFO/MainProcess] celery@localhost.localdomain ready.
[2017-09-22 08:20:55,023: INFO/MainProcess] Received task: dashboard.tasks.debug_task[71e6c0e1-92e1-494e-b5e9-163eeb7bd24e]
[2017-09-22 08:20:55,027: INFO/ForkPoolWorker-1] Task dashboard.tasks.debug_task[71e6c0e1-92e1-494e-b5e9-163eeb7bd24e] succeeded in 0.001253978000022471s: 'debug_task'
[2017-09-22 08:22:21,179: INFO/MainProcess] Received task: dashboard.tasks.debug_task[b81fe9a0-1725-4702-ba0e-13196c9b5977]
[2017-09-22 08:22:21,180: INFO/ForkPoolWorker-1] Task dashboard.tasks.debug_task[b81fe9a0-1725-4702-ba0e-13196c9b5977] succeeded in 0.00018433199147693813s: 'debug_task'
  
  
$ celery -A qrd beat -l info -s /tmp/celerybeat-schedule
celery beat v4.1.0 (latentcall) is starting.
__    -    ... __   -        _
LocalTime -> 2017-09-24 04:20:37
Configuration ->
    . broker -> amqp://qrdweb:**@localhost:5672/qrdweb
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> /tmp/celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 minutes (300s)
[2017-09-24 04:20:37,823: INFO/MainProcess] beat: Starting...
[2017-09-24 04:20:37,866: INFO/MainProcess] Scheduler: Sending due task add every 10 (qrd.celery.test)
[2017-09-24 04:20:47,856: INFO/MainProcess] Scheduler: Sending due task add every 10 (qrd.celery.test)
[2017-09-24 04:20:57,858: INFO/MainProcess] Scheduler: Sending due task add every 10 (qrd.celery.test)
[2017-09-24 04:20:57,861: INFO/MainProcess] Scheduler: Sending due task qrd.celery.test('world') (qrd.celery.test)
[2017-09-24 04:21:07,858: INFO/MainProcess] Scheduler: Sending due task add every 10 (qrd.celery.test)
[2017-09-24 04:21:17,859: INFO/MainProcess] Scheduler: Sending due task add every 10 (qrd.celery.test)

运行成功，可以使用Supervisord来守护监控Celery的运行，参考这里。
Django项目结果如下

先配置Celery使用RabbitMQ作为Broker，使用Django DB来保存调用结果settings.py

import os
from configparser import RawConfigParser
 
#https://code.djangoproject.com/wiki/SplitSettings
config = RawConfigParser()
config.read('/home/qrd/setting/settings.ini')
 
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
 
CELERY_BROKER_URL = 'amqp://usr:pwd@localhost:5672/qrdweb'
CELERY_RESULT_BACKEND = 'django-db'

然后在Django项目下创建celery.py文件，

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from celery.schedules import crontab
 
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'qrd.settings')
 
app = Celery('qrd')
 
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
 
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
 
 
app.conf.beat_schedule = {
    'hue-tasks-debug_task': {
        'task': 'hue.tasks.debug_task',
        'schedule': 10.0,
        'args': ()
    },
}

并且在__init__.py引入Celery即可集成

from __future__ import absolute_import, unicode_literals
 
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
 
__all__ = ['celery_app']

Django的异步任务只能定义在各个app的task.py文件里，比如qrd.hue.tasks定义了一个定时任务

from celery import task
 
@task
def debug_task():
    #print(arg)
    return 'debug_task'

也可以在其模块里面调用

from tasks import debug_task
 
def save(data):
    debug_task.delay()

顺便推荐一个Bootstrap管理后台模板：gentelella。

参考链接：
异步任务神器 Celery
Django配置celery执行异步任务和定时任务
 淺談 Gunicorn 各個 worker type 適合的情境

Python使用Kerberos认证查询Impala

发表评论

最近做QoS报告，数据来源于Impala，客户端认证采用的是Kerberos。
Impala是Cloudera公司开发并开源的一款基于HDFS/Hbase的MPP SQL引擎，它提供SQL语义，能够查询存储在Hadoop的HDFS和HBase中的PB级大数据。Kerberous本身是一个网络认证授权协议，借由中心服务器认证，对通信双方的客户端/服务端进行授权而不需要传递双方的密码。Kerberos的认证流程比较有意思，分为三个阶段：

客户端认证

1 客户端发送自己用户名
2 认证服务器返回使用客户端密钥加密的Client/TGS会话密钥和使用票据授权服务器密钥加密的TGT，包括sessions key，用户信息及有效期
3 客户端使用自己的密钥解密出Client/TGS会话密钥

服务授权

1 客户端发送两条消息：接收到的TGT和所请求的服务ID；使用Client/TGS会话密钥加密的用户ID和时间戳
2 票据授权服务器使用自己的密钥解密TGT得到客户端的Client/TGS会话密钥，然后使用它解密出用户ID并进行认证。返回使用所请求服务端密钥加密的client-server票据和使用Client/TGS会话密钥加密的Client/Server会话密钥
3 客户端使用Client/TGS会话密钥(Client/TGS Session Key)解密出Client/Server会话密钥

服务请求

1 客户端发送两条消息：使用所请求服务端密钥加密的client-server票据及使用Client/Server会话密钥加密的用户ID和时间戳
2 服务端使用自己的密钥解密client-server票据从而得到Client/Server会话密钥，使用该密钥解密获得用户信息并认证。返回使用Client/Server会话密钥的新时间戳
3 客户端使用Client/Server会话密钥解密该消息，认证结束并请求服务
4 服务端提供服务

在CentOS上安装Kerberos

1	`yum` `install` `krb5-devel pam_krb5 krb5-libs krb5-workstation`

编辑配置

1	`vim` `/etc/krb5.conf`

配置KDC，认证服务器

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = true
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tkt_enctypes = rc4-hmac
default_tgs_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
[realms]
EXAMPLE.COM = {
default_domain = example.com
kdc = kdc01.example.com
kdc = kdc02.example.com
admin_server = adc01.example.com
admin_server = adc02.example.com
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM

测试一下

[root@localhost rc]# kinit abc.xyz@EXAMPLE.COM
Password for abc.xyz@EXAMPLE.COM:

注意这个配置文件每行前面的空格被删掉，是因为在VirtualBox里面每行开头有莫名其妙的乱码，Linux下并不可见，在EditPlus下面才发现，否则会乱报错

kinit: Improper format of Kerberos configuration file while initializing Kerberos 5 library
kinit: Cannot find KDC for realm "EXAMPLE.COM" while getting initial credentials

查看一下认证的ticket

[root@localhost vagrant]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: abc.xyz@EXAMPLE.COM
 
Valid starting       Expires              Service principal
09/21/2017 08:30:50  09/21/2017 18:30:50  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 09/28/2017 08:30:42

这个ticket在28号就会过期了，到时候又要输入密码，这样也不便于自动化程序使用。可以使用ktutil创建keytab文件

$ ktutil
ktutil:  addent -password -p abc.xyz@EXAMPLE.COM -k 1 -e RC4-HMAC
Password for abc.xyz@EXAMPLE.COM:
ktutil:  wkt abc.xyz.keytab
ktutil:  q
$ ls
abc.xyz.keytab

测试一下

$ kinit -kt abc.xyz.keytab abc.xyz@EXAMPLE.COM
$ klist -k abc.xyz.keytab
Keytab name: FILE:abc.xyz.keytab
KVNO Principal
---- --------------------------------------------------------------------------
  1 abc.xyz@EXAMPLE.COM

之后便可以使用kinit自动更新ticket了。注意，如果更换了密码，需要重新生成新的keytab。
另外，相同用户生成的授权ticket在任意一台机器上都是相同的， kinit时会自动同步回来的。
公司的大数据平台使用Hue来提供基于web界面的查询，Impala也支持使用ODBC方式查询。在Python里使用的是impyla来查询，首先安装sasl的依赖

yum install libgsasl-devel cyrus-sasl-devel cyrus-sasl-gssapi
pip install impyla thrift_sasl

测试脚本

from impala.dbapi import connect
conn = connect(host="impalad.example.com", port=21050, auth_mechanism='GSSAPI', kerberos_service_name='impala', database='acme')
cur =  conn.cursor()
cur.execute(r'SELECT * FROM acme WHERE dt="2017-09-12" LIMIT 5')
print(cur.fetchall())

运行下

1	`python` `test.py`

如下报错，则是服务器不能连接，检查一下网络，DNS/hosts及VPN

thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not connect to ('impalad.example.com', 21050)")

如下报错，CentOS则是需要cyrus-sasl-gssapi模块

thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not start SASL: b'Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found'")

参考链接：
Impala：新一代开源大数据分析引擎
 大数据时代快速SQL引擎-Impala
CDH 5.2中Impala认证集成LDAP和Kerberos
Kerberos
Configuring Kerberos Authentication for Windows
Speaking Kerberos with KNIME Big Data Extensions

勇气

逆水行舟，不进则退

标签归档：impala

Django + Celery处理异步任务/计划任务

Python使用Kerberos认证查询Impala