标签归档：Docker

API 网关 Kong

1条回复

从前开发一个互联网服务程序，大概可以在一台机器上完成：数据库、应用都在一起。随着业务发展壮大，会把数据库独立出来，以便扩展拆分。然后再把一部分公用业务独立出来扩展，譬如文件存储、缓存等。接着业务也才拆分，比如会员、商品。微服务大行其道，各个团队维护着许多服务、API。这么多服务，前端业务逻辑该怎么接入呢？
如今单一的前端UI也可能是多个团队共同开发的结果。当你在本地开发一个小功能时，甚至会牵扯到多个前端UI/后端API。一个前端页面除了加载自己的资源外，还加载V2/V3的API（为什么会同时存在V1/V2/V3的API），甚至嵌入了另外一个页面（React看起来也不错），该怎么调试这种混合开发呢？cookie都传递不过去，好吧，使用JWT代替，再设置一下跨源资源共享（CORS）。。。但是那些旧的应用怎么办？
当我们启动一个Node.js应用时npm run start，默认监听3000端口。当然我们也可以让它监听80端口。但是当我们多开几个应用时，只好让它们都监听不同端口了，怎么样才能统一端口监听呢？当然Node.js可以通过诸如node-http-proxy来转发这些请求，但是代理转发跟应用业务无关吧？如果是其他编程语言呢？都重复这些代理配置/开发吗？Don’t Repeat Yourself。
Faas也日渐流行，比如AWS Lambda， 强调仅仅专注某个功能，根据事件驱动进行计算，那么每个服务也要开发一整套路由分发/认证/日志吗？
这些问题可以使用反向代理来解决，提供统一的服务入口，对前端/客户端隐藏背后的细节，最简单的当然是Nginx。Nginx监听来自某个端口（比如80）的请求，然后根据不同的来源/端口/域名/url分发给不同的后端服务器。这看起来跟云服务厂商各自开发的负载均衡器差不多，比如AWS ELB。
在本地开发的时候，我们当然不会使用ELB来解决。直接使用Nginx当然也没啥问题，编辑一下配置文件，重启应用。但是如果有个UI就更好了，如果还有API就非常好了-这样就能方便的在线注册/更改路由而不需要重启服务器，在这个快速发展、弹性开发的年代更需要这个能力。
Kong是一个基于OpenResty的网关服务器，可以进行路由（转发/负载），插件（日志/认证/监控）管理，并提供RESTful API。而OpenResty 是一个基于Nginx与Lua的高性能Web平台，使用Lua来构建动态网关。听起来像是在Nginx上面编程，这跟在Apache上面使用PHP模块进行编程有什么区别？最大的区别在于，这里的编程对象是Nginx（或者公共模块），扩展Nginx能力，比如负载均衡/日志/认证/监控，而不是输出web页面/业务逻辑。这些东西抽出来以后，就不需要每个模块再重复开发了，比如认证/安全。
在本地可以使用docker来运行Kong服务，麻烦在于数据库迁移工作。通常部署Kong需要几个步骤，比如初始化数据库，迁移升级等等。网上的配置大都过时了，这里可以使用官方提供docker-compose.yml来做数据库工作，并且使用Konga作为管理UI

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
➜  kong ls
POSTGRES_PASSWORD         data                      docker-compose.yml
➜  kong ls data
postgresql
➜  kong cat POSTGRES_PASSWORD
kong
➜  kong cat docker-compose.yml
version: '3.7'
 
volumes:
  kong_data: {}
 
networks:
  kong-net:
    external: false
 
services:
  kong-migrations:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations bootstrap
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
 
  kong-migrations-up:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations up && kong migrations finish
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
 
  kong:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    user: "${KONG_USER:-kong}"
    depends_on:
      - db
    environment:
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: '0.0.0.0:8001'
      KONG_CASSANDRA_CONTACT_POINTS: db
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    ports:
      - "8000:8000/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "8443:8443/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
 
  db:
    image: postgres:9.5
    environment:
      POSTGRES_DB: ${KONG_PG_DATABASE:-kong}
      POSTGRES_USER: ${KONG_PG_USER:-kong}
      POSTGRES_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "${KONG_PG_USER:-kong}"]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
    stdin_open: true
    tty: true
    networks:
      - kong-net
    volumes:
      - /Users/xxxx/docker/kong/data/postgresql:/var/lib/postgresql/data
 
  konga:
    image: pantsel/konga
    environment:
      TOKEN_SECRET: channing.token
      DB_ADAPTER: postgres
      DB_HOST: db
      DB_USER: ${KONG_PG_USER:-kong}
      DB_PASSWORD: kong
      DB_DATABASE: ${KONG_PG_DATABASE:-kong}
    ports:
     - 1337:1337
    networks:
     - kong-net
 
    depends_on:
      - db
 
secrets:
  kong_postgres_password:
    file: ./POSTGRES_PASSWORD

运行docker-composer up就可以看到

1
2
3
4
5
➜  kong docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                             PORTS                                                                                                NAMES
20ebe7885c0d        kong:latest                       "/docker-entrypoint.…"   2 hours ago         Up 11 seconds (healthy)            0.0.0.0:8000->8000/tcp, 127.0.0.1:8001->8001/tcp, 0.0.0.0:8443->8443/tcp, 127.0.0.1:8444->8444/tcp   kong_kong_1
4a7a39c863ae        pantsel/konga                     "/app/start.sh"          2 hours ago         Up 11 seconds                      0.0.0.0:1337->1337/tcp                                                                               kong_konga_1
aa732758fc51        postgres:9.5                      "docker-entrypoint.s…"   2 hours ago         Up 11 seconds (health: starting)   5432/tcp                                                                                             kong_db_1

访问http://127.0.0.1:1337/即可以进入Konga 管理界面了。8000/8443端口是需要监听转发的端口，8001/8444则是Kong RESTFUl管理API的端口。这里也可以把8000/8443改为80/443，这样访问的时候就可以直接使用域名/localhost而不必加端口了。
注册登录进去后首先要添加Kong RESTFUl管理API的地址，这里使用的是docker环境，IP是动态分配的，所以使用链接名就可以了

连接上Kong API后可以看到dashboard列出了支持的插件

Kong里面的管理对象是service，路由转发/插件都是围绕service展开的，添加一个service


在它上面添加一个路由，这里我们在服务只监听以/api开头的url，并转发到后端服务器去


注意这里在route里面的path里面也需要添加/api，否则转发的时候会出错（多拼api）。最简单就是service监听不指定path，在route里指定。一个service下面可以有多个转发路由，每个可以独立管理、设置超时等。
还可以为每个service绑定不同的插件进行处理，比如认证/安全/日志等等，这样就不用在不同的系统里面重复开发这些功能，使得各个团队更加专注也本业务开发


有效插件还可以进行流量限制、请求头/响应头修改等等。这些插件的功能需要基于consumer来开发管理，功能可以非常强大，详细可以参考文档。
Kong还有一项功能Upstream配置，与Nginx的ngx_http_upstream_module差不多，可以作为负载均衡、流量分发控制使用。可以用命令行测试基于hostname的路由，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev1.example.com'
HTTP/1.1 404 Not Found
Date: Tue, 27 Oct 2020 08:41:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.1.4
 
{"message":"no Route matched with those values"}                                                                                                                                                                                                                             ➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev.example.com'
 
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Tue, 27 Oct 2020 08:42:21 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.6.40
Set-Cookie: PHPSESSID=0uc3aoc735j21sk3ni5s72vjc1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Access-Control-Allow-Headers: X-CSRF-Token,Authorization,X-Accept-Charset,X-Accept,Content-Type
X-Kong-Upstream-Latency: 8397
X-Kong-Proxy-Latency: 0
Via: kong/2.1.4
 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...

作为一个API网关，Kong能做的很多。如果只是在本地简单的做反向代理，可以使用nginx-proxy-manager，这也是一个基于Nginx开发的网关，根据WEB UI动态生成Nginx配置文件，然后执行/usr/sbin/nginx -s reload生效。有些参数不能通过UI配置（比如超时设置），可以直接写Nginx配置，upstream则支持直接转发tcp请求，支持websockets代理转发，甚至集成了Let’s encrypt自动申请SSL证书（需要验证域名）


本地同样可以使用docker跑起来，默认监听80/443端口，81端口即管理界面

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
➜  nginx-proxy-manager ls -lah
total 16
drwxr-xr-x   6 xxxx domain users   192B Sep 29 10:24 .
drwxr-xr-x  15 xxxx domain users   480B Sep 29 09:15 ..
-rw-r--r--   1 xxxx domain users   2.3K Sep 29 09:42 config.json
drwxr-xr-x   8 xxxx domain users   256B Sep 29 09:31 data
-rw-r--r--   1 xxxx domain users   740B Sep 29 10:24 docker-compose.yml
drwxr-xr-x   3 xxxx domain users    96B Oct 27 16:38 letsencrypt
➜  nginx-proxy-manager cat docker-compose.yml
version: "3"
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: always
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
      # TCP Forward Example:
      - '8022:8022'
    volumes:
      # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: jc21/mariadb-aria
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql
➜  nginx-proxy-manager docker ps
CONTAINER ID        IMAGE                             COMMAND             CREATED             STATUS                PORTS                                                                    NAMES
3dd58e9cff1f        jc21/nginx-proxy-manager:latest   "/init"             4 weeks ago         Up 7 days (healthy)   0.0.0.0:80-81->80-81/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8022->8022/tcp   nginx-proxy-manager_app_1
a033151b28ed        jc21/mariadb-aria                 "/scripts/run.sh"   4 weeks ago         Up 7 days             3306/tcp                                                                 nginx-proxy-manager_db_1

nginx-proxy-manager会将配置写在data目录下面，可以直接编辑这些文件，Nginx reload之后便会生效，可以看到这些文件，具体的加载规则参考文档

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
➜  nginx-proxy-manager ls data/nginx
dead_host        default_host     default_www      dummycert.pem    dummykey.pem     proxy_host       redirection_host stream           temp
➜  nginx-proxy-manager cat data/nginx/proxy_host/1.conf
# ------------------------------------------------------------
# dev.example.com
# ------------------------------------------------------------
server {
  set $forward_scheme http;
  set $server         "192.168.33.14";
  set $port           80;
 
  listen 80;
listen [::]:80;
 
  server_name dev.example.com;
 
  access_log /data/logs/proxy_host-1.log proxy;
 
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
 
  location /Login {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://login.example.com:443;
 
  }
 
  location /api/v2 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.14:9070;
 
  }
 
  location /api/v3 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;
 
  }
 
  location /Chat {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;
 
  }
 
  location /Catalog {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;
 
  }
 
  location /static {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;
 
  }
 
  location /css {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;
 
  }
 
  location / {
 
    # Proxy!
    include conf.d/include/proxy.conf;
  }
 
  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}
 
➜  nginx-proxy-manager cat data/nginx/stream/1.conf
# ------------------------------------------------------------
# 8022 TCP: 1 UDP: 0
# ------------------------------------------------------------
server {
  listen 8022;
listen [::]:8022;
 
  proxy_pass 192.168.33.14:22;
 
  # Custom
  include /data/nginx/custom/server_stream[.]conf;
  include /data/nginx/custom/server_stream_tcp[.]conf;
}

在docker-compose.yml里面设置一下upstream监听转发的端口，就可以通过8022端口访问192.168.33.14的22端口了

1
2
3
4
5
6
➜  nginx-proxy-manager telnet 127.0.0.1 8022
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3
^C^C^C^C^C^C^CConnection closed by foreign host.

不论是Kong还是nginx-proxy-manager均有提供API，极大的增强服务网关的可编程性，为动态上线/弹性扩展/自动化运维提供了便利。

参考链接：
从IaaS到FaaS—— Serverless架构的前世今生
聊一聊微服务网关 Kong
KONG网关 — KongA管理UI使用
云原生架构下的 API 网关实践： Kong （二）
微服务 API 网关 -Kong 详解
Creating a web API with Lua using Nginx OpenResty
Nginx基于TCP/UDP端口的四层负载均衡（stream模块）配置梳理
Nginx支持TCP代理和负载均衡-stream模块
聊聊 API Gateway 和 Netflix Zuul
Envoy 是什么？

使用Composer和Docker创建Swoole项目

发表评论

最近有一个小功能使用swoole扩展来开发，并没有使用框架，从零开始。这里采用类用似Symfony的目录结构：

1
2
3
4
5
6
7
8
9
10
11
12
13
.
├── bin
├── config
├── src
├── tests
├── vendor
├── phpunit.xml
├── composer.json
├── .gitignore
├── Dockerfile
├── LICENSE
├── phpunit.xml
└── README.md

项目代码放置在src下面，测试代码则在tests里面，config目录存放配置文件，bin文件夹则是一些命令行工具，vendor目录则是composer安装第三方依赖库的目录。composer.json是借鉴k911/swoole-bundle，内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
{
    "name": "llitllie/swoole-sample",
    "type": "library",
    "description": "Swoole Sample",
    "keywords": [
        "Swoole"
    ],
    "license": "MIT",
    "homepage": "https://github.com/llitllie/swoole-sample.git",
    "authors": [{
        "name": "llitllie",
        "email": "xxx@yyy.zzz",
        "homepage": "https://github.com/llitllie/swoole-sample.git"
    }],
    "require": {
        "php": "^7.2",
        "ext-swoole": "^4.3.4"
    },
    "require-dev": {
        "phpunit/phpunit": "^8",
        "phpstan/phpstan": "^0.11.8",
        "friendsofphp/php-cs-fixer": "^2.15",
        "swoole/ide-helper": "@dev"
    },
    "scripts": {
        "static-analyse-src": [
            "phpstan analyze src -l 7 --ansi"
        ],
        "cs-analyse": [
            "php-cs-fixer fix -v --dry-run --diff --stop-on-violation --ansi"
        ],
        "analyse": [
            "@static-analyse-src",
            "@cs-analyse"
        ],
        "test": [
            "@analyse",
            "@unit-tests"
        ],
        "unit-tests": [
            "phpunit tests --testdox --colors=always"
        ],
        "fix": "php-cs-fixer fix -v --ansi"
    },
    "suggest": {
        "ext-uv": "^0.2.4",
        "ext-ev": "^1.0.6"
    }
}

这里require里面指定了PHP版本>=7.2和Swoole扩展，composer安装时会自动检查。require-dev里面指定了单元测试工具phpunit、代码分析工具phpstan和代码风格检查php-cx-fixer。phpstan可以帮助分析代码，检测语法，在代码运行前发现问题。php-cx-fixer则可以帮忙格式化代码，保持代码统一风格。scripts里面定义了调用它们的命令，可以使用composer运行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
$ ls -la vendor/bin/
total 4
drwxrwxr-x.  2 vagrant vagrant   69 Jul  1 07:24 .
drwxrwxr-x. 24 vagrant vagrant 4096 Jul  1 07:25 ..
lrwxrwxrwx.  1 vagrant vagrant   41 Jul  1 07:24 php-cs-fixer -> ../friendsofphp/php-cs-fixer/php-cs-fixer
lrwxrwxrwx.  1 vagrant vagrant   33 Jul  1 07:24 php-parse -> ../nikic/php-parser/bin/php-parse
lrwxrwxrwx.  1 vagrant vagrant   30 Jul  1 07:24 phpstan -> ../phpstan/phpstan/bin/phpstan
lrwxrwxrwx.  1 vagrant vagrant   26 Jul  1 07:24 phpunit -> ../phpunit/phpunit/phpunit
$ ./vendor/bin/phpunit --testdox tests
 
$ composer test
> phpstan analyze src -l 7 --ansi
 7/8 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓░░░░]  87%
 
 8/8 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%                                                                               
 [OK] No errors                                                                
                                                                                 
 
> php-cs-fixer fix -v --dry-run --diff --stop-on-violation --ansi
Loaded config default from "/home/ticket/.php_cs.dist".
Using cache file ".php_cs.cache".
SSSSSSSSSSS
Legend: ?-unknown, I-invalid file syntax, file ignored, S-Skipped, .-no changes, F-fixed, E-error
 
Checked all files in 0.070 seconds, 6.000 MB memory used
> phpunit tests --testdox --colors=always
PHPUnit 8.2.4 by Sebastian Bergmann and contributors.
 
Sim\Ticket\Node\Zookeeper
 ✓ Get id
 
Sim\Ticket\Number
 ✓ Load
 ✓ Get timestamp
 ✓ Get node id
 ✓ Generate
 ✓ Generate with zookeeper node
 
Sim\Zookeeper\Client
 ✓ Zookeeper
 ✓ Zookeeper extension
 
Time: 841 ms, Memory: 4.00 MB
 
OK (8 tests, 23 assertions)

phpuit.xml里面则是一些测试配置，可以在里面定义自动加载和变量

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<?xml version="1.0" encoding="UTF-8"?>
 
<!-- https://phpunit.readthedocs.io/en/7.3/configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/7.3/phpunit.xsd"
         colors="true"
         bootstrap="vendor/autoload.php"
>
    <filter>
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">src/</directory>
        </whitelist>
    </filter>
    <testsuites>
        <testsuite name="Unit tests suite">
            <directory>tests</directory>
        </testsuite>
    </testsuites>
    <php>
        <includePath>.</includePath>
        <const name="SERVICE" value="192.168.33.1"/>
        <env name="SERVICE" value="192.168.33.1"/>
    </php>
</phpunit>

PHP Composer支持使用已有的代码作为模板，快速克隆创建一个新项目，方便重复使用。将代码保存到GitHub上面然后在packagist.org上面提交就可以了。使用composer create-project：

1
composer create-project llitllie/swoole-sample example dev-master

这里使用Docker来构建运行代码，Dockerfile内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
ARG PHP_TAG="7.2-cli-alpine3.9"
 
FROM php:$PHP_TAG
 
ENV COMPOSER_ALLOW_SUPERUSER 1
 
RUN set -ex \
    && apk update \
    && apk add --no-cache --virtual .build-deps curl gcc g++ make build-base autoconf \
    && apk add libstdc++ openssl-dev libffi-dev \
    && docker-php-ext-install sockets \
    && docker-php-source extract \
    && printf "yes\nyes\nno\nyes\nno\n" | pecl install swoole \
    && docker-php-ext-enable swoole \
    && docker-php-source delete \
    && curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer \
    && apk del .build-deps \
    && rm -rf /tmp/*
 
WORKDIR /usr/src/app
COPY . ./
ARG COMPOSER_ARGS="install"
RUN composer ${COMPOSER_ARGS} --prefer-dist --ignore-platform-reqs --no-progress --no-suggest --no-scripts --ansi
 
EXPOSE 9501
CMD ["php", "bin/server.php"]

这里声明了swoole将会监听9501端口，默认运行bin目录下面的serverphp文件。使用pecl安装swoole会要求回答几个问题，比如是否启用socket/http2/mysqlnd等等，这里使用printf输出答案。构建，然后运行

1
2
3
docker build  -t llitllie/swoole-project .
docker run --name swoole -d -p 9501:9501 llitllie/swoole-project
#docker run --name web -dit -p 9501:9501 --mount type=bind,source=/Users/vagrant/example,target=/opt/app llitllie/swoole-project /bin/sh

这样一个swoole项目模板算是好了，可以继续往composer文件里面添加需要的库或脚本；也可以继承在已经构建好的docker镜像基础上继续添加所需的软件，更改监听端口等等。
可以在Docker Cloud上面配置对应的git，并配置对应自动化构建规则，这样就可以方便的自动构建了。

Nginx + Frp/Ngrok反向代理Webhook至本地

发表评论

跟第三方平台打交道，经常需要设置一个接受通知的Webhook，比如微信/Skype的回调。它们要求有一个可以在互联网上访问得了的入口，比如某个域名，如果是在本地开发的话，不好调试。通常使用花生壳来代理本地服务，但是花生壳有一些限制，比如端口。有些域名服务商，比如DNSPOD，Linode，提供相应的API，也可以自己搭建DDNS服务，但是也可能有端口访问限制。Frp/Ngrok都是Go语言开发的内网穿透工具，可以自己部署搭建。Frp是国人开发的一款反向代理软件，可以转发请求给位于NAT后面的机器，支持TCP，UDP，HTTP/HTTPS。Ngrok则是国外的一款内网穿透软件，也支持HTTP/HTTPS转发。这里使用Nginx作为反向代理服务器，接收互联网回调并转发给本地的Frp/Ngrok服务，由它们接收webhook请求并转发至本地开发环境。
前面使用OpenVpn搭建了私有网络，可以在Nginx里面配置转发给目标机器就可以了

1
vim /etc/nginx/conf.d/100-dev.example.conf

内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
server {
    listen 80;
    server_name dev.example.com;
    return 301 https://$host$request_uri;
}
 
server {
 
    listen 443;
    server_name dev.example.com;
 
    ssl_certificate           /etc/letsencrypt/live/example.com/cert.pem;
    ssl_certificate_key       /etc/letsencrypt/live/example.com/privkey.pem;
 
    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;
 
    location / {
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
 
      proxy_pass          http://10.9.0.2/;
      proxy_redirect off;
 
    }
}

这里使用了let’s encryt的泛域名证书，官方并没有对应的插件，但是DNSPOD有提供相应的API，第三方开发了一个插件自certbot-dns-dnspod，安装这个插件并且配置Dnspod的API Token：

1
2
3
4
5
6
7
8
9
$ yum install certbot python2-certbot-nginx
$ certbot --nginx
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ pip install certbot-dns-dnspod
$ vim /etc/letsencrypt/dnspod.conf
certbot_dns_dnspod:dns_dnspod_email = "123@163.com"
certbot_dns_dnspod:dns_dnspod_api_token = "123,ca440********"
 
$ chmod 600 /etc/letsencrypt/dnspod.conf

手动请求证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ certbot certonly -a certbot-dns-dnspod:dns-dnspod --certbot-dns-dnspod:dns-dnspod-credentials /etc/letsencrypt/dnspod.conf --server https://acme-v02.api.letsencrypt.org/directory -d example.com -d "*.example.com"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator certbot-dns-dnspod:dns-dnspod, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for example.com
dns-01 challenge for example.com
Starting new HTTPS connection (1): dnsapi.cn
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Resetting dropped connection: acme-v02.api.letsencrypt.org
 
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/example.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/example.com/privkey.pem
   Your cert will expire on 2019-08-04. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
*/1 * * * * /usr/local/qcloud/stargate/admin/start.sh > /dev/null 2>&1 &
 - If you like Certbot, please consider supporting our work by:
 
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
 
 
$ ls -la /etc/letsencrypt/live/example.com/
total 12
drwxr-xr-x 2 root root 4096 May  6 12:06 .
drwx------ 3 root root 4096 May  6 12:06 ..
lrwxrwxrwx 1 root root   34 May  6 12:06 cert.pem -> ../../archive/example.com/cert1.pem
lrwxrwxrwx 1 root root   35 May  6 12:06 chain.pem -> ../../archive/example.com/chain1.pem
lrwxrwxrwx 1 root root   39 May  6 12:06 fullchain.pem -> ../../archive/example.com/fullchain1.pem
lrwxrwxrwx 1 root root   37 May  6 12:06 privkey.pem -> ../../archive/example.com/privkey1.pem
-rw-r--r-- 1 root root  692 May  6 12:06 README

配置证书自动更新

1
0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew

Frp的开发者已经提供了编译好的frp服务端和客户端，下载即可使用。这里使用docker来运行Frp服务，使用这个Dockerfile，更改版本号为0.26.0，并编译

1
2
3
4
$ docker build . -t frps:0.26
$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
frps                 0.26                8a87cb91d4de        2 hours ago         21.1MB

测试一下SSH代理服务，创建服务端配置文件

1
2
mkdir -p frp/conf
vim frp/conf/frps.ini

frps.ini内容

1
2
[common]
bind_port = 7000

运行一下frp服务端

1
2
3
4
5
6
7
8
9
#清除先前运行的容器
$ docker rm frp-server
$ docker run --name frp-server -v /root/frp/conf:/conf -p 7000:7000 -p 6000:6000 frps:0.26
2019/04/22 06:41:17 [I] [service.go:136] frps tcp listen on 0.0.0.0:7000
2019/04/22 06:41:17 [I] [root.go:204] Start frps success
2019/04/22 06:41:27 [I] [service.go:337] client login info: ip [110.87.98.82:61894] version [0.26.0] hostname [] os [linux] arch [386]
2019/04/22 06:41:27 [I] [tcp.go:66] [e8783ecea2085e15] [ssh] tcp proxy listen port [6000]
2019/04/22 06:41:27 [I] [control.go:398] [e8783ecea2085e15] new proxy [ssh] success
2019/04/22 06:41:41 [I] [proxy.go:82] [e8783ecea2085e15] [ssh] get a new work connection: [110.*.*.*:61894]

这里映射了2个端口，端口7000是frp服务端监听的端口，以便客户端能够连接上；端口6000是需要服务端监听这个端口，以便提供反向代理服务，比如SSH。如果使用的是腾讯云，相应的端口需要在安全组放行。
客户端直接下对应的包，里面有配置试例。创建本地配置文件frpc.ini如下

1
2
3
4
5
6
7
8
9
[common]
server_addr = 123.*.*.*
server_port = 7000
 
[ssh]
type = tcp
local_ip = 127.0.0.1
local_port = 22
remote_port = 6000

这个配置即告诉服务端，将服务端的6000端口转发到本地的22端口。本地运行

1
2
3
4
$ ./frpc -c ./frpc.ini.ssh
2019/04/22 06:41:27 [I] [service.go:221] login to server success, get run id [e8783ecea2085e15], server udp port [0]
2019/04/22 06:41:27 [I] [proxy_manager.go:137] [e8783ecea2085e15] proxy added: [ssh]
2019/04/22 06:41:27 [I] [control.go:144] [ssh] start proxy success

然后在服务端连接客户端。这里连接的是服务端的6000端口，会被转发给远程（局域网内）主机

1
2
3
4
5
6
7
8
9
10
11
[rth@centos72]$ ssh -oPort=6000 vagrant@123.*.*.*
The authenticity of host '[123.*.*.*]:6000 ([123.*.*.*]:6000)' can't be established.
RSA key fingerprint is SHA256:NhBO/PDL***********************.
RSA key fingerprint is MD5:20:70:e2:*:*:*:*:*:*:*:*:*:*:*:*:*.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[123.*.*.*]:6000' (RSA) to the list of known hosts.
vagrant@123.*.*.*'s password:
Last login: Mon Apr 22 06:39:07 2019 from 10.0.2.2
[vagrant@centos64 ~]$ exit
logout
Connection to 123.*.*.* closed.

Frp转发http服务很简单。在conf目录下创建配置frps.ini监听本机来自8080端口的HTTP请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[common]
bind_port = 7000
vhost_http_port = 8080
 
[root@VM_1_218_centos frp]# docker run --name frp-server -v /root/frp/conf:/conf -p 7000:7000 -p 8080:8080 frps:0.26
2019/05/06 07:26:28 [I] [service.go:136] frps tcp listen on 0.0.0.0:7000
2019/05/06 07:26:28 [I] [service.go:178] http service listen on 0.0.0.0:8080
2019/05/06 07:26:28 [I] [root.go:204] Start frps success
2019/05/06 07:26:51 [I] [service.go:337] client login info: ip [123.*.*.*:56758] version [0.26.0] hostname [] os [linux] arch [386]
2019/05/06 07:26:51 [I] [http.go:72] [19f60a30aa924343] [web] http proxy listen for host [test.example.com] location []
2019/05/06 07:26:51 [I] [control.go:398] [19f60a30aa924343] new proxy [web] success
2019/05/06 07:27:05 [I] [proxy.go:82] [19f60a30aa924343] [web] get a new work connection: [123.*.*.*:56758]
2019/05/06 07:27:05 [I] [proxy.go:82] [19f60a30aa924343] [web] get a new work connection: [123.*.*.*:56758]
2019/05/06 07:27:06 [I] [proxy.go:82] [19f60a30aa924343] [web] get a new work connection: [123.*.*.*:56758]

然后配置Nginx转发请求

1
2
3
4
5
6
7
8
9
10
11
12
$ vim /etc/nginx/conf.d/100-dev.example.conf
 
    location / {
      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
 
      proxy_pass          http://127.0.0.1:8080/;
      proxy_redirect off;
 
    }

创建本地web传教客户端配置frpc.ini，将来自服务器dev.example.com:8080端口的HTTP请求转发至本地80端口

1
2
3
4
5
6
7
8
[common]
server_addr = 123.*.*.*
server_port = 7000
 
[web]
type = http
local_port = 80
custom_domains = dev.example.com

运行本地客户端

1
2
3
4
5
6
7
[root@vagrant-centos64 frp]# ./frpc -c ./frpc.ini
2019/05/06 07:26:51 [I] [service.go:221] login to server success, get run id [19f60a30aa924343], server udp port [0]
2019/05/06 07:26:51 [I] [proxy_manager.go:137] [19f60a30aa924343] proxy added: [web]
2019/05/06 07:26:51 [I] [control.go:144] [web] start proxy success
2019/05/06 07:27:37 [E] [control.go:127] work connection closed, EOF
2019/05/06 07:27:37 [I] [control.go:228] control writer is closing
2019/05/06 07:27:37 [I] [service.go:127] try to reconnect to server...

访问dev.example.com既可以看到本地web服务器页面。Frp还可以代理其他请求，也有在它基础上二次加工提供基于token认证的转发服务。
Ngrok 2.0以后不再开源，只能使用1.3版本的搭建。这里使用docker-ngrok来构建。Ngrok构建需要SSL证书，复制刚才生成的letsencypt证书并更改server.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ git clone https://github.com/hteen/docker-ngrok
$ cp /etc/letsencrypt/live/example.com/fullchain.pem myfiles/base.pem
$ cp /etc/letsencrypt/live/example.com/fullchain.pem myfiles/fullchain.pem
$ cp /etc/letsencrypt/live/example.com/privkey.pem myfiles/privkey.pem
 
$ vim server.sh
#!/bin/sh
set -e
 
if [ "${DOMAIN}" == "**None**" ]; then
    echo "Please set DOMAIN"
    exit 1
fi
 
if [ ! -f "${MY_FILES}/bin/ngrokd" ]; then
    echo "ngrokd is not build,will be build it now..."
    /bin/sh /build.sh
fi
 
 
${MY_FILES}/bin/ngrokd -tlsKey=${MY_FILES}/privkey.pem -tlsCrt=${MY_FILES}/fullchain.pem -domain="${DOMAIN}" -httpAddr=${HTTP_ADDR} -httpsAddr=${HTTPS_ADDR} -tunnelAddr=${TUNNEL_ADDR}

构建Ngrok镜像

1
2
3
4
5
6
[root@VM_1_218_centos docker-ngrok]# docker build -t ngrok:1.3 .
[root@VM_1_218_centos docker-ngrok]# docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
ngrok                1.3                 dc70190d6377        13 seconds ago      260MB
frps                 0.26                8a87cb91d4de        2 hours ago         21.1MB
alpine               latest              cdf98d1859c1        12 days ago         5.53MB

然后交叉编译生成Linux/Mac/Windows平台的客户端

1
2
3
4
5
6
7
8
9
$ rm -rf assets/client/tls/ngrokroot.crt
$ cp /etc/letsencrypt/live/example.com/chain.pem assets/client/tls/ngrokroot.crt
$ rm -rf assets/server/tls/snakeoil.crt
$ cp /etc/letsencrypt/live/example.com/cert.pem assets/server/tls/snakeoil.crt
$ rm -rf assets/server/tls/snakeoil.key
$ cp /etc/letsencrypt/live/example.com/privkey.pem assets/server/tls/snakeoil.key
$ GOOS=linux GOARCH=amd64 make release-client
$ GOOS=windows GOARCH=amd64 make release-client
$ GOOS=darwin GOARCH=amd64 make release-client

在服务器上运行Ngrok服务，将8090端口请求转发给容器的80端口，并且映射容器的4443端口到服务器的7000端口，以便客户端连接

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@VM_1_218_centos docker-ngrok]# docker run --name ngrok -e DOMAIN='example.com' -p 8090:80 -p 8091:443 -p 7000:4443 -v /root/docker-ngrok/myfiles:/myfiles ngrok:1.3 /bin/sh /server.sh
[09:18:21 UTC 2019/05/07] [INFO] (ngrok/log.(*PrefixLogger).Info:83) [registry] [tun] No affinity cache specified
[09:18:21 UTC 2019/05/07] [INFO] (ngrok/log.Info:112) Listening for public http connections on [::]:80
[09:18:21 UTC 2019/05/07] [INFO] (ngrok/log.Info:112) Listening for public https connections on [::]:443
[09:18:21 UTC 2019/05/07] [INFO] (ngrok/log.Info:112) Listening for control and proxy connections on [::]:4443
[09:18:21 UTC 2019/05/07] [INFO] (ngrok/log.(*PrefixLogger).Info:83) [metrics] Reporting every 30 seconds
[09:18:27 UTC 2019/05/07] [INFO] (ngrok/log.(*PrefixLogger).Info:83) [tun:18e8cd42] New connection from 123.*.*.*:50529
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [tun:18e8cd42] Waiting to read message
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [tun:18e8cd42] Reading message with length: 125
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [tun:18e8cd42] Read message {"Type":"Auth","Payload":{"Version":"2","MmVersion":"1.7","User":"","Password":"","OS":"linux","Arch":"amd64","ClientId":""}}
[09:18:27 UTC 2019/05/07] [INFO] (ngrok/log.(*PrefixLogger).Info:83) [ctl:18e8cd42] Renamed connection tun:18e8cd42
[09:18:27 UTC 2019/05/07] [INFO] (ngrok/log.(*PrefixLogger).Info:83) [registry] [ctl] Registered control with id 1957f20b9b3ce3b76c7d8fc8b16276ed
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [ctl:18e8cd42] [1957f20b9b3ce3b76c7d8fc8b16276ed] Writing message: {"Type":"AuthResp","Payload":{"Version":"2","MmVersion":"1.7","ClientId":"1957f20b9b3ce3b76c7d8fc8b16276ed","Error":""}}
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [ctl:18e8cd42] [1957f20b9b3ce3b76c7d8fc8b16276ed] Writing message: {"Type":"ReqProxy","Payload":{}}
[09:18:27 UTC 2019/05/07] [DEBG] (ngrok/log.(*PrefixLogger).Debug:79) [ctl:18e8cd42] [1957f20b9b3ce3b76c7d8fc8b16276ed] Waiting to read message

将刚才编译的客户端下载下来，创建grok.cfg，连接服务器的7000端口

1
2
server_addr: "example.com:7000"
trust_host_root_certs: false

指定要监听的域名，及本地web端口

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
./ngrok -config=ngrok.cfg -subdomain=dev 9010
 
ngrok                                                                                                                                                                                                                                                         (Ctrl+C to quit)
                                                                                                                                                                                                                                                                               
Tunnel Status                 online                                                                                                                                                                                                                                         
Version                       1.7/1.7                                                                                                                                                                                                                                        
Forwarding                    http://dev.flexkit.cn -> 127.0.0.1:9010                                                                                                                                                                                                        
Forwarding                    https://dev.flexkit.cn -> 127.0.0.1:9010                                                                                                                                                                                                       
Web Interface                 127.0.0.1:4040                                                                                                                                                                                                                                 
# Conn                        2                                                                                                                                                                                                                                              
Avg Conn Time                 46.84ms                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                               
 
 
HTTP Requests                                                        
-------------                                                        
                                                                       
GET /teams                    200 OK                  

请求dev.example.com即可以访问到本机9010端口的web服务。
附：ZeroTier是一个软件定义网络（SDN）软件，可以免费组建私有网络，当然也可以用来转发服务器请求至本地。

参考链接：：
CentOS7搭建ngrok服务器
inconshreveable/ngrok
hteen/ngrok
搭建自己的 Ngrok 服务器, 并与 Nginx 并存
使用Docker部署Ngrok实现内网穿透
Laravel DDNS package，可代替花生壳之类的软件
通过DNSPod API实现动态域名解析
借助dnspod-api定时更新域名解析获取树莓派公网ip
使用Let’s Encrypt生成通配符SSL证书
Letsencrypt使用DNSPOD验证自动更新证书
在 OpenWrt 环境下使用 DnsPod 来实现动态域名解析
利用ssh反向代理以及autossh实现从外网连接内网服务器
How To Configure Nginx with SSL as a Reverse Proxy for Jenkins

CentOS 7上安装OpenVPN

发表评论

之前做的一些树莓派机器因为分布分散，不好远程调试，最近看了CentOS 7.0 Set Up OpenVPN Server In 5 Minutes，决定备份一下。
首先需要有一台公网服务器，比如阿里云、腾讯云服务器，以便这些设备能连接上。阿里云、腾讯云的服务器外部IP并不是直接绑定在你的服务器上的，而是NAT转发到服务器上的，这与Linode不一样，虽然可以方便换IP。查看服务器公网IP，也可以在云管理后台查看

1
2
3
ip a show eth0
dig +short myip.opendns.com @resolver1.opendns.com
dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{ print $2}'

然后是一键安装脚步

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
yum update
wget https://raw.githubusercontent.com/Angristan/openvpn-install/master/openvpn-install.sh -O centos7-vpn.sh
chmod +x centos7-vpn.sh
./centos7-vpn.sh
 
Welcome to the OpenVPN installer!
The git repository is available at: https://github.com/angristan/openvpn-install
 
I need to ask you a few questions before starting the setup.
You can leave the default options and just press enter if you are ok with them.
 
I need to know the IPv4 address of the network interface you want OpenVPN listening to.
Unless your server is behind NAT, it should be your public IPv4 address.
IP address: 10.0.2.15
 
It seems this server is behind NAT. What is its public IPv4 address or hostname?
We need it for the clients to connect to the server.
Public IPv4 address or hostname: 110.*.*.*
 
Checking for IPv6 connectivity...
 
Your host does not appear to have IPv6 connectivity.
 
Do you want to enable IPv6 support (NAT)? [y/n]: n
 
What port do you want OpenVPN to listen to?
   1) Default: 1194
   2) Custom
   3) Random [49152-65535]
Port choice [1-3]: 1
 
What protocol do you want OpenVPN to use?
UDP is faster. Unless it is not available, you shouldn't use TCP.
   1) UDP
   2) TCP
Protocol [1-2]: 1
 
What DNS resolvers do you want to use with the VPN?
   1) Current system resolvers (from /etc/resolv.conf)
   2) Self-hosted DNS Resolver (Unbound)
   3) Cloudflare (Anycast: worldwide)
   4) Quad9 (Anycast: worldwide)
   5) Quad9 uncensored (Anycast: worldwide)
   6) FDN (France)
   7) DNS.WATCH (Germany)
   8) OpenDNS (Anycast: worldwide)
   9) Google (Anycast: worldwide)
   10) Yandex Basic (Russia)
   11) AdGuard DNS (Russia)
DNS [1-10]: 3
 
Do you want to use compression? It is not recommended since the VORACLE attack make use of it.
Enable compression? [y/n]: n
 
Do you want to customize encryption settings?
Unless you know what you're doing, you should stick with the default parameters provided by the script.
Note that whatever you choose, all the choices presented in the script are safe. (Unlike OpenVPN's defaults)
See https://github.com/angristan/openvpn-install#security-and-encryption to learn more.
 
Customize encryption settings? [y/n]: n
 
Okay, that was all I needed. We are ready to setup your OpenVPN server now.
You will be able to generate a client at the end of the installation.
Press any key to continue...

这个脚本省掉了许许多多配置，基本上按照默认一路enter就可以了，唯一需要输入的就是你的公网IP和生成的OpenVPN客户端连接文件名字，比如client.ovpn
然后启动服务

1
2
3
4
systemctl stop openvpn@server
systemctl enable openvpn@server
systemctl restart openvpn@server
systemctl status openvpn@server

查看生成的服务器配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@li846-239 ~]# cat /etc/openvpn/server.conf
port 1194
proto udp6
dev tun
user nobody
group nobody
persist-key
persist-tun
keepalive 10 120
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
#client-to-client
#push "dhcp-option DNS 1.0.0.1"
#push "dhcp-option DNS 1.1.1.1"
#push "redirect-gateway def1 bypass-dhcp"
#server-ipv6 fd42:42:42:42::/112
#tun-ipv6
#push tun-ipv6
#push "route-ipv6 2000::/3"
#push "redirect-gateway ipv6"
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key 0
crl-verify crl.pem
ca ca.crt
cert server_Yfej6xnJrDu3vs6K.crt
key server_Yfej6xnJrDu3vs6K.key
auth SHA256
cipher AES-128-GCM
ncp-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
status /var/log/openvpn/status.log
verb 3

10.8.0.0是默认的组网网段，可以改成别的。腾讯云并不支持IPV6，注释掉。push “dhcp-option DNS 1.0.0.1″是推送DNS服务器给客户端。push “redirect-gateway def1 bypass-dhcp”是重定向客户端所有流量到服务器，建议按需添加需要互访的子网段（默认只有10.8.0.0可以互相访问)，否则所有流量重定向，可能会引起连接问题，比如ssh腾讯云服务器公网22端口进不去。Push和Route还有高级应用，建议参考OpenVPN: 2x HOW TO
查看连接状况。

1
journalctl --identifier openvpn -f

配置腾讯云服务器安全组，开放1194端口


将刚才生成的client.ovpn下载下来，mac上双击，使用Tunnelblick打开导入即可。但是使用最新的OpenVPN 2.6提示连不上， 报错：tls-crypt unwrap error: packet too short。
在Linux上安装OpenVPN，启动连接

1
2
3
4
5
yum install openvpn
cp client.ovpn /etc/openvpn/client.conf #一定要复制到这里
openvpn --client --config /etc/openvpn/client.conf #测试一下
systemctl start openvpn@client
systemctl enable openvpn@client

测试一下

1
2
ping 10.8.0.1
ip route

如果有问题的话，可以查看iptable配置、网络及进程

1
2
3
4
5
6
7
8
9
10
11
cat /etc/iptables/add-openvpn-rules.sh
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
iptables -A INPUT -i tun0 -j ACCEPT
iptables -A FORWARD -i eth0 -o tun0 -j ACCEPT
iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT
iptables -A INPUT -i eth0 -p udp --dport 1194 -j ACCEPT
 
iptables -t nat -L -n -v
net.ipv4.ip_forward
netstat -tulpn | grep :1194
ps aux | grep openvpn

当再一次运行脚本的时候，会提示是否创建新用户配置，注意不同用户不能共享同一个配置。这里创建的用户可以是支持用户名密码也可以是无账号密码的。

1
2
3
4
5
6
7
8
9
10
11
12
[root@tc ~]# ./centos7-vpn.sh
Welcome to OpenVPN-install!
The git repository is available at: https://github.com/angristan/openvpn-install
 
It looks like OpenVPN is already installed.
 
What do you want to do?
   1) Add a new user
   2) Revoke existing user
   3) Remove OpenVPN
   4) Exit
Select an option [1-4]:

连上OpenVPN服务器都处于同一个网络了，可以使用内部IP访问。现在使用docker来部署OpenVPN服务也很方便，比如rancher-openvpn，支持LDAP验证。
如果是两个及以上的局域网互通，可以使用Openswan建立虚拟网络连接。
更新：发现resin更新成了balena，免费支持10台设备，并且开放了IOT相关的系统、软件。
发现一个新的组网服务zerotier，可以快速的创建属于自己的私有网络。

参考链接：
How To Set Up and Configure an OpenVPN Server on CentOS 7
Tunnel Your Internet Traffic Through an OpenVPN Server
OpenVPN configuration examples
云服务器 ECS Linux CentOS OpenVPN 配置概述
使用OpenVPN搞定远程办公
使用openvpn打通两个异地网络
How To Install Openswan And Create Site-to-Site VPN On CentOS 7
Setting up an IPSEC VPN using OpenSwan in cloud environments
Openswan L2TP/IPsec VPN client setup
在Raspberry Pi上面运行Docker
使用 Zerotier 组建虚拟局域网实现内网穿透
trying to route between two openvpn clients
Site-to-site VPN routing explained in detail

在Raspberry Pi上面运行Docker

发表评论

公司有个项目是运行在Raspberry Pi上面，这个项目涉及到Perl，Python，Java Applet，并且未来Pi的数量可能达到上千个。最近碰到了一样的代码遵循一样的部署脚本，在不同的树莓派上运行竟然到不同的结果。于是也想将Docker应用到树莓派上面去，使用docker主要优点：
1 消除环境差异，程序运行所依赖的环境已经一起打包，随身携带了
2 简化配置，只有docker相关配置，不再需要每个语言单独配置
3 重复利用
4 方便升级，之前有差异时，是重写镜像。。
5 方便管理，可以使用docker已有的工具来管理
6 性能接近原生应用

Raspberry Pi在jessie这个版本就已经支持了docker，检查版本：

1
2
3
4
5
6
$ lsb_release -da
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 8.0 (jessie)
Release:    8.0
Codename:  jessie

如果你的版本已经支持，直接执行以下命令就可以了：

1
curl -sSL https://get.docker.com | sh

如果你的版本不支持，你需要升级系统或重新刷镜像，可以采用hypriot

1
2
3
4
5
6
7
8
9
10
11
12
#下载
wget https://downloads.hypriot.com/hypriotos-rpi-v1.0.0.img.zip
#解压
unzip hypriotos-rpi-v1.0.0.img.zip
#查找 SD card ID
lsblk
#查找类似/dev/mmcblk0
 
#弹出SD卡
umount /run/media/mac/8734-1E4C
#将镜像写入 SD card
sudo dd if=hypriotos-rpi-v1.0.0.img of=/dev/mmcblk0 bs=1m

然后启动，默认的用户名/密码是pirate/hypriot。
当然也可以采用Debian或者是Alpine Linux来定制（Raspberry Pi仅支持 arm版本的系统，并且只能在arm平台上构建，或者这个）。
运行测试一下:

1
docker run -d -p 80:80 hypriot/rpi-busybox-httpd

访问以下树莓派的IP，就会看到一个非常神器的网页，而这个容器仅仅几M。在这里可以看到基于hypriot的好几个镜像，也可以搜索其他的raspberry pi镜像。
现在我们使用hypriot/rpi-python来构建一个使用Python Selenium测试带有Java Applet的网页：

1
2
docker run -idt -P --name web hypriot/rpi-python
docker attach web

进入到容器里面，执行以下命令：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
sudo apt-get update
#安装openjdk
sudo apt-get install openjdk-7-jdk
#安装Firefox
sudo apt-get install iceweasel
#安装Java Applet插件
sudo apt-get install icedtea-7-plugin
#安装虚拟桌面
sudo apt-get install xvfb
#安装Python包管理工具
sudo apt-get install python-pip
#安装selenium
sudo pip install selenium
#安装Python虚拟桌面库
sudo pip install pyvirtualdisplay

创建测试test.py：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from pyvirtualdisplay import Display
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
display = Display(visible=0, size=(800, 600))
display.start()
caps = DesiredCapabilities.FIREFOX
caps["marionette"] = False
#caps["binary"] = "/usr/bin/firefox"
firefox_binary = FirefoxBinary("/usr/bin/firefox")
fp = webdriver.FirefoxProfile()
fp.set_preference("security.enable_java", True )
fp.set_preference("plugin.state.java", 2)
driver = webdriver.Firefox(firefox_profile=fp,firefox_binary=firefox_binary)
driver.maximize_window()
driver.get("http://businessconnect.telus.com/bandwidth/en_us/index.html?voip=on&voiplines=1&testlength=15&codec=g711&_=1470641590&startTest=on")
time.sleep(120)
mos = driver.find_element_by_class_name("sectiontitle18").find_element_by_tag_name("span").get_attribute("innerHTML")
print mos
driver.close()
display.stop()

运行测试用例：

1
python test.py

如果仅仅是这样子，是行不通的，Java Applet的安全询问弹窗超出了浏览器范围是点击不到的，需要更为强大的组件来做，比如。
这里采用取巧的方法，首先把pyvirtualdisplay相关代码注释掉，在界面上观察具体的执行效果，并点击安全询问，运行通过。然后取消注释，再次运行，就可以了。最后将运行通过的配置文件/root/.icedtea目录拷贝出来，放到其他的树莓派里面就可以了。
如果碰到报错”selenium.common.exceptions.WebDriverException: Can’t load the profile”，通常是说Firefox或selenium版本太低，如果检查都不是，需要修改WebDriver的超时时间

1
nano /usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py

timeout从30秒改为120秒：

1
2
3
4
5
6
7
8
class WebDriver(RemoteWebDriver):
 
 
    # There is no native event support on Mac
    NATIVE_EVENTS_ALLOWED = sys.platform != "darwin"
 
 
    def __init__(self, firefox_profile=None, firefox_binary=None, timeout=120,

修改为容器后，提交变更：

1
2
3
4
5
root@black-pearl:/home/pirate# docker ps -l
CONTAINER ID        IMAGE                COMMAND            CREATED            STATUS              PORTS              NAMES
04762770df2d        hypriot/rpi-python  "bash"              2 days ago          Up 2 days                              web
root@black-pearl:/home/pirate# docker commit 04762770df2d hypriot/rpi-python
root@black-pearl:/home/pirate# docker rm web

运行新镜像

1
2
3
#docker run -idt -P --name web hypriot/rpi-python
#映射程序目录和配置文件进去
docker run -idt -P --name web -v /home/pi:/home/pi -v /home/pi/.icedtea:/root/.icedtea hypriot/rpi-python

程序正常运行后，可以将这个镜像打包：

1
docker save -o hypriot.tar hypriot/rpi-python:latest

分发给其他树莓派运行：

1
2
3
docker load --input hypriot.tar
docker images
docker run -idt -P --name web -v /home/pi:/home/pi -v /home/pi/.icedtea:/root/.icedtea hypriot/rpi-python

这样子便可以利用docker工具来管理这个应用程序，比起原本复杂的环境配置要简单很多。

在查找Pi可用镜像的时候，顺便发现了一个IOT（internet of things）管理平台：resin。

参考链接：
add support to install Docker on raspbian/jessie
Getting started with Docker on your Raspberry Pi
How to get Docker running on your Raspberry Pi using Linux
Getting selenium python to work on Raspberry Pi Model B