作者归档:admin

mTLS 认证

有时候虽然我们部署一个服务到了公网,但只想提供给部分用户而不想其他人(黑客/robot)使用,比如webhook或者跨机房的服务调用。虽然我们可以使用密码/token保护这些服务,但是程序可以被直接扫描到,仍然存在被攻击的风险。而如果把这些服务放在某个认证服务器后面,又存在二次登陆的问题。如何标识目标用户又不影响请求过程?答案是mTLS认证。
TLS认证是当前WEB通信的基石,没有加密的通信无异于裸奔:任何中间人都可以查看/串改通信内容。mTLS认证,亦即双向TLS认证,不但客户端校验服务器证书,服务器亦校验客户端证书,这意味着如果客户端证书不在服务器的信任列表里面,那么客户端也访问不了服务。请求认证过程如下

mTLS可以用来保护私有服务,加强通信过程,减少服务器攻击。由于客户端也需要提供证书,证书里面可以携带用户或者设备信息,可以识别仅授信用户,才授权访问 。由于服务器需要校验客户端信息,所以未提供有效证书用户不能访问,就算盗取了用户的密码/token也不可以。因为双方都互相校验证书,所以也可以抵抗中间人攻击。这样子我们便可以公开服务,但又限制某些用户使用了。
mTLS认证有别于应用程序认证。mTLS仅涉及证书校验,在网关出就可以进行,不需要涉及应用程序逻辑/数据库。mTLS也不依赖于请求头/请求体里面的内容,所以不会影响已有程序。
mTLS认证限制某些用户/设备访问,也有别于传统VPN网络。VPN总是要求服务端/客户端先建立连接,才能在同一内网内访问。一旦连接建立,便可以访问所有服务。而采用mTLS认证,则每一个服务都可以使用不同的客户端证书,限制不同的用户访问。零信任(ZeroTrust)亦建立在mTLS之上,即对每个请求都做双向认证。
mTLS并不是什么新鲜事物,许多互联网软件都支持,比如浏览器、SSH,SSH采用私钥登陆的过程,即双向认证:客户端先将自己的公钥上传至服务器,建立连接时,客户端询问是否信任服务器,服务器亦校验客户端。Git使用SSH key登陆也是类似双向认证。开发者调用Google云服务,除了app认证信息,也是需要双向认证的。有些互联网公司也采用双向认证来保护自己的应用,比如支付功能,甚至是所有页面-只允许自己的APP查看,其他免谈。
Nginx早已支持mTLS认证,除了服务器本身的证书,还需要配置客户端证书或其签发root CA证书。关于服务器证书获取之前已介绍基于DNS自动获取证书了,同样的方法也可以用来生成客户端证书,比如基于用户id来生成不同域名username.id.example.com的证书给客户端使用。这里介绍另外一种基于LEGO ACME在本地生成客户端证书的方法。
首先安装lego程序,如果有golang环境直接就好了,或者下载对应平台的程序

$ go install github.com/go-acme/lego/v4/cmd/lego@latest

然后就可以基于DNS厂商来生成证书,这里以腾讯云DNS,注意它的APP Key是不同于DNSPod的。如果网络不加,建议延长TIMEOUT时间。这里一并生成浏览器/系统可用的PKCS12文件(.pfx)

$ TENCENTCLOUD_SECRET_ID=*** TENCENTCLOUD_PROPAGATION_TIMEOUT=120 TENCENTCLOUD_POLLING_INTERVAL=5 \ 
TENCENTCLOUD_SECRET_KEY=***\
lego --email [email protected] --pem --pfx --pfx.pass *** --cert.timeout 120 --dns tencentcloud --domains user1.idt.exmaple.com run

2023/04/08 11:42:46 [INFO] [user1.idt.exmaple.com] acme: Obtaining bundled SAN certificate
2023/04/08 11:42:47 [INFO] [user1.idt.exmaple.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/217627103547
2023/04/08 11:42:47 [INFO] [user1.idt.exmaple.com] acme: Could not find solver for: tls-alpn-01
2023/04/08 11:42:47 [INFO] [user1.idt.exmaple.com] acme: Could not find solver for: http-01
2023/04/08 11:42:47 [INFO] [user1.idt.exmaple.com] acme: use dns-01 solver
2023/04/08 11:42:47 [INFO] [user1.idt.exmaple.com] acme: Preparing to solve DNS-01
2023/04/08 11:42:50 [INFO] [user1.idt.exmaple.com] acme: Trying to solve DNS-01
2023/04/08 11:42:50 [INFO] [user1.idt.exmaple.com] acme: Checking DNS record propagation using [127.0.0.53:53]
2023/04/08 11:42:55 [INFO] Wait for propagation [timeout: 2m0s, interval: 5s]
2023/04/08 11:42:55 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:01 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:06 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:11 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:17 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:22 [INFO] [user1.idt.exmaple.com] acme: Waiting for DNS record propagation.
2023/04/08 11:43:35 [INFO] [user1.idt.exmaple.com] The server validated our request
2023/04/08 11:43:35 [INFO] [user1.idt.exmaple.com] acme: Cleaning DNS-01 challenge
2023/04/08 11:43:37 [INFO] [user1.idt.exmaple.com] acme: Validations succeeded; requesting certificates
2023/04/08 11:43:39 [INFO] [user1.idt.exmaple.com] Server responded with a certificate.
$ ls -la .lego/certificates
total 40
drwx------ 2 channing channing 4096 4月 8 11:43 .
drwx------ 4 channing channing 4096 4月 8 10:58 ..
-rw------- 1 channing channing 5341 4月 8 11:43 user1.idt.exmaple.com.crt
-rw------- 1 channing channing 3751 4月 8 11:43 user1.idt.exmaple.com.issuer.crt
-rw------- 1 channing channing 244 4月 8 11:43 user1.idt.exmaple.com.json
-rw------- 1 channing channing 227 4月 8 11:43 user1.idt.exmaple.com.key
-rw------- 1 channing channing 5568 4月 8 11:43 user1.idt.exmaple.com.pem
-rw------- 1 channing channing 2992 4月 8 11:43 user1.idt.exmaple.com.pfx

由于这个证书的CA是Let’s Encrypt的,而Let’s Encrypt本身的证书又是Internet Security Research Group签发。而当Nginx配置ssl_client_certificate CA证书时只能是Root CA,也就是ISRG的证书,而不是中间厂商的证书,也就是不能用Let’s Encrypt的CA证书。ISRG的证书可以从浏览器导出或者网上下载,然后保存到服务器上

$ mv isrg_root.crt /etc/nginx/certs/
$ chomd 400 etc/nginx/certs/isrg_root.crt

Nginx配置如下,可以将日志级别设置为debug,调试/观察认证

$ vim /etc/nginx/conf.d/site-available/100-mtls.example.com
server {
    listen 80;
    server_name mtls.example.com;
    return 301 https://$host$request_uri;
}

server {

    listen 443 ssl;
    server_name mtls.example.com;

    ssl_certificate           /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_client_certificate    /etc/nginx/conf.d/certs/isrg_root.crt;
    ssl_verify_client on;
    ssl_verify_depth 2;

    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;
#    error_log /var/log/nginx/error.log debug;
    location / {
        root  /var/www/mtls.example.com/html;
        index  index.html index.htm;
    }
    error_page  500 502 503 504  /50x.html;
    location = /50x.html {
        root  /usr/share/nginx/html;
    }
}

使用curl测试一下,提示需要客户端证书

curl https://mtls.example.com
<html>
  <head>
    <title>400 No required SSL certificate was sent</title>
  </head>

  <body bgcolor="white">
    <center>
      <h1>400 Bad Request</h1>
    </center>
    <center>No required SSL certificate was sent</center>
    <hr>
    <center>nginx/1.14.0</center>
  </body>
</html>

当带上证书的时候就可以了

curl --cert user1.idt.exmaple.com.crt --key user1.idt.exmaple.com.key https://mtls.example.com
<html>
<head>
<title>Welcome to mtls.example.com!</title>
</head>
<body>
<h1>Success! The mtls server is working!</h1>
</body>
</html>

使用浏览器访问目标网站需要安装.pfx文件,否则也会报错,禁止访问。通常Windows/Mac双击.pfx文件即可自动安,也可以通过浏览器配置来安装。使用Chrome需要在从菜单设置->隐私设置和安全性->安全->管理设备证书->个人->导入。使用Firefox则是从从菜单设置->隐私与安全->安全->查看证书->您的证书->导入。iOS和Android可以通过邮件的方式将.pfx文件传输到手机上,点击.pfx文件按照系统提示安装即可。最后使用浏览器打开https://mtls.example.com,会提示使用刚才配置好的证书,选择即可。
上面Nginx配置的是公共CA的根证书(isrg_root.crt),这意味着所有该CA签发的证书都将可以访问https://mtls.example.com,这有点太开放了。如果我们自己生成Root CA证书,便可以做进一步的控制。
首先生成Root CA证书

$ openssl genrsa -des3 -out ca.key 4096
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
$ openssl req -new -x509 -days 3650 -key ca.key -out ca.crt39:06
Enter pass phrase for ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

$ cat ca.key > ca.pem 
$ cat ca.crt >> ca.pem
$ ls
ca.crt ca.key ca.pem

然后使用CA证书生成客户端

$ openssl genrsa -out user1.key
Enter PEM pass phrase:
$ openssl req -new -key user1.key -out user1.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
$ openssl x509 -req -days 365 -in user1.csr -CA ca.crt -CAkey ca.key -set_serial 01 -out user1.crt
Certificate request self-signature ok
Enter pass phrase for ca.key:
$ openssl pkcs12 -export -out user1.pfx -inkey user1.key -in user1.crt -certfile ca.crt
Enter Export Password:
Verifying - Enter Export Password:

然后将ca.crt上传至服务器/etc/nginx/certs/下面并配置ssl_client_certificate使用该证书即可。使用curl测试一下


curl --cert user1.crt --key user1.key https://mtls.example.com
<html>
<head>
<title>Welcome to mtls.example.com!</title>
</head>
<body>
<h1>Success! The mtls server is working!</h1>
</body>
</html>

同样可以成功访问。不过刚才生成的.pfx文件只有Firefox可以导入使用,其他系统/浏览器不可以,那么可以从Firefox中重新备份导出该文件,Chrome/其他系统就可以使用了。或者重新转换一下

openssl pkcs12 -in user1.pfx -out user1.pem -nodes #pfx证书转换为x.509格式的pem证书
openssl pkcs12 -in user1.pfx -nocerts -nodes -out user1.key #pfx证书中提取密钥对
openssl rsa -in user1.key -out user1_private.key #密钥对提取私钥
openssl pkcs12 -export -in user1.pem -inkey user1_private.key -out user1.pfx #使用x.509+私钥转换为win10可用的pfx证书

使用自定义CA可以有效控制证书的签发,也可以免CA直接签发客户端证书

$ openssl req -x509 -nodes -days 9999 -newkey rsa:2048 -keyout user2.key -out user2.crt
..+...+...
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []

无CA签名的证书,就直接将user2.crt上传至服务器/etc/nginx/certs/下面并配置ssl_client_certificate使用该证书即可。如果生成了多个,就合并成一个上传服务器就好了。使用curl测试一下,也是可以的

curl --cert user2.crt --key user2.key https://mtls.example.com
<html>
<head>
<title>Welcome to mtls.example.com!</title>
</head>
<body>
<h1>Success! The mtls server is working!</h1>
</body>
</html>

Nginx还支持将客户端证书及CA的DN传递给后端程序,再配合访问控制,便可以精细控制用户访问了。Nginx配置如下,注意这里ssl_verify_client是optional,认证失败不再返回400而是403

$ vim /etc/nginx/conf.d/site-available/100-mtls.example.com
server {
    listen 443 ssl;
    server_name mtls.example.com;

    ssl_certificate           /etc/letsencrypt/live/example.comfullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_client_certificate    /etc/nginx/conf.d/certs/ca.crt;
    ssl_verify_client optional;
    ssl_verify_depth 2;

    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;
#    error_log /var/log/nginx/error.log debug;
    location / {
#        if ($ssl_client_i_dn != "/C=EX/CN=Client/[email protected]") {
#          return 403;
#        }
        if ($ssl_client_verify != SUCCESS) { return 403; }
        proxy_set_header     SSL_Client_Issuer $ssl_client_i_dn;
        proxy_set_header     SSL_Client $ssl_client_s_dn;
        proxy_set_header     SSL_Client_Verify $ssl_client_verify;
        proxy_pass           http://127.0.0.1:3000/;
    }
    error_page  500 502 503 504  /50x.html;
    location = /50x.html {
        root  /usr/share/nginx/html;
    }
}
➜

NodeJS代码如下

const express = require('express');

const port = 3000;
const host = '127.0.0.1';
const options = { };

const app = express();

app.get('/', (req, res) => {
    res.status(200)
        .json(`Hello ${req.header("ssl_client")}, your certificate was issued by ${req.header("SSL_Client_Issuer")}!`);
});
app.listen(port, host, () => {
  console.log("listening");
})

使用curl测试一下便会输出客户端DN。

$ curl --cert user1.crt --key user1.key https://mtls.example.com/
"Hello emailAddress=user1@example,CN=exmaple,OU=server,C=exmaple, your certificate was issued by emailAddress=user1@example,CN=exmaple,OU=user1,C=exmaple!"

事实上不仅Nginx支持设置客户端认证,Kong网关,云原生应用传输traefik等也早已支持,编程语言层面更是支持开发mTLS应用了。

参考链接:
What is mutual TLS (mTLS)?
Mutual authentication
mTLS Golang Example
How to configure certification based client authentication with Nginx ?
How To Implement Two Way SSL With Nginx
mTLS with node
mTLS with NGINX and NodeJS
LEGO
Firefox导出PFX证书在Win10导入提示密码错误
NGinx SSL certificate authentication signed by intermediate CA (chain)

Android平板安装Termux和LunarVim

这次趁着双11买了个小米平板5pro,才发现不过是大号手机而已,系统是基于Android 11的MIUI 13。除了WPS,bilibili、浏览器都不过兼容桌面模式而已,然而却又失去了移动设备的便利性:touch事件不能精确响应,于是卸掉重新安装的普通版。终于体会到了雪球不明真相群众的那句话:平板上的东西已经在手机上进化完了。想想这个平板除了刷视频、看书、画画,还能干点啥?看代码应该也有优势,毕竟分辨率还不错。Android上并没有什么好用的代码App,虽然可以安装Github查看代码,但还是看不了几行,也不如IDE来的方便。幸好还有Termux,极大的拓展了Android的可用性,增加了超过1万个cli应用,比如比应用市场上任何服务器连接App都好用的ssh,强大的编程开发工具LunarVim
Termux是一个Android终端(terminal)模拟软件并提供Linux环境,可以安装许多Linux下常见软件,比如ssh、git、curl,和各种开发软件,比如nodejs、python、rust、golang等等。Termux既可以从GitHub上下载安装,也可以使用F-Droid安装。使用F-Droid的好处在于可以自动更新,和管理其他的扩展,比如Termux:Styling、Termux:API。安装完成后记得给予联网权限,如果安装了其他的扩展,记得在安全中心-联网管理中找到对应的扩展App也给予联网权限,否则Termux就上不了网。 安装完成后打开Termux,这里注意Termux重置了目录,所有目录都是在data/data/com.termux/files/下面,并提供了两个快捷访问$HOME: data/data/com.termux/files/home和$PREFIX: /data/data/com.termux/files/usr。运行下面的命令设置一下存储和文件夹软链接,这样方便Termux与外部目录(比如Downloads)互相复制粘贴

termux-setup-storage

也可以更改一些配置,比如软键盘快捷键之类的

vim ~/.termux/termux.properties
#重新加载配置
termux-reload-settings
#更改使用的源仓库,默认使用多仓库,可以改为使用单一仓库,比如清华源
termux-change-repo

然后安装一些基础软件

pkg install openssh git curl wget vim bat croc

然后安装oh-my-zsh和自动补全插件,其他插件的安装与正常安装无差别

pkg install zsh
sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

然后编辑~/.zshrc,更改主题和插件

# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes
#ZSH_THEME="robbyrussell"
ZSH_THEME="agnoster"

# Which plugins would you like to load?
# Standard plugins can be found in $ZSH/plugins/
# Custom plugins may be added to $ZSH_CUSTOM/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
# Add wisely, as too many plugins slow down shell startup.
plugins=(git zsh-autosuggestions)

如果zsh显示乱码的话,可以按住屏幕在弹出的more菜单里面更改nerd font字体,比如ubuntu。
现在可以开始安装LunarVim的依赖

pkg install neovim build-essential python vim-python nodejs rust fd ripgrep

安装完成完成后neovim, make,npm,pip,cargo也都安装好了。这时候就可以使用nvim来编辑了,可以参考:help nvim/:Tutor进行学习。这里顺便安装了pyhon vim的交互包,等下可以跳过python依赖的安装。还安装了两个rust写的神器:文件查找fd和文件内容搜索ripgrep,等下也可以跳过rust依赖安装。,
LunarVim是一个基于Neovim开发的IDE,打包了文件搜索、内容查找、代码补全,语法高亮等许多功能,这个可比简单vim强多了,更多功能、特点可以在官网视频学习。官方提供的安装脚本并不包含Termux的,我们下载下来看一下是否都已经安装好依赖、做一些改动

curl https://raw.githubusercontent.com/lunarvim/lunarvim/master/utils/installer/install.sh > install.sh

这里稍微改一下,匹配不到OS就使用pkg安装

function detect_platform() {
  case "$OS" in
    Linux)
      if [ -f "/etc/arch-release" ] || [ -f "/etc/artix-release" ]; then
        RECOMMEND_INSTALL="sudo pacman -S"
      elif [ -f "/etc/fedora-release" ] || [ -f "/etc/redhat-release" ]; then
        RECOMMEND_INSTALL="sudo dnf install -y"
      elif [ -f "/etc/gentoo-release" ]; then
        RECOMMEND_INSTALL="emerge -tv"
      else # it's termux
        RECOMMEND_INSTALL="pkg install -y"
      fi
      ;;

然后开始安装LunarVim

chmod +x install.sh
export LV_BRANCH='release-1.2/neovim-0.8'
./install.sh
      88\                                                   88\
      88 |                                                  \__|
      88 |88\   88\ 888888$\   888888\   888888\ 88\    88\ 88\ 888888\8888\
      88 |88 |  88 |88  __88\  \____88\ 88  __88\\88\  88  |88 |88  _88  _88\
      88 |88 |  88 |88 |  88 | 888888$ |88 |  \__|\88\88  / 88 |88 / 88 / 88 |
      88 |88 |  88 |88 |  88 |88  __88 |88 |       \88$  /  88 |88 | 88 | 88 |
      88 |\888888  |88 |  88 |\888888$ |88 |        \$  /   88 |88 | 88 | 88 |
      \__| \______/ \__|  \__| \_______|\__|         \_/    \__|\__| \__| \__|

--------------------------------------------------------------------------------
Detecting platform for managing any additional neovim dependencies
--------------------------------------------------------------------------------
Would you like to install LunarVim's NodeJS dependencies?
[y]es or [n]o (default: no) : y
Installing node modules with npm..
All NodeJS dependencies are successfully installed
--------------------------------------------------------------------------------
Would you like to install LunarVim's Python dependencies?
[y]es or [n]o (default: no) : y
Verifying that pip is available..
/usr/bin/python3: No module named ensurepip
Installing with pip..
Requirement already satisfied: pynvim in /usr/lib/python3/dist-packages (0.4.2)
All Python dependencies are successfully installed
--------------------------------------------------------------------------------
Would you like to install LunarVim's Rust dependencies?
[y]es or [n]o (default: no) : y
All Rust dependencies are successfully installed

这里注意,除了NodeJS依赖需要安装,Python、Rust都已经安装好了,选择No就可以了。安装完成后可以使用命令打开

#打开默认窗口
lvim
#打开指定文件
#lvim install.sh
#打开指定目录
#lvim downloads

默认界面长这样子,可以搜索文件、搜索内容,打开文件、项目等。如果有出现文字、图标显示乱码,那么需要安装nerd font,参考上面的字体配置。

按下空格键就会出现引导界面

比如空格+e打开目录

空格+s搜索

空格+L打开LunarVim配置,比如keyMaps

剩下就是neovim、LuarVim的操作了,比如ctrl+h/ctrl+l的文件目录与代码窗口左右切换,空格+b+p/n/e/j在打开的文件里面切换、打开、关闭,alt+1/2/3打开下边、右边、弹出窗口里面打开终端



如此便可以使用LunarVim写代码了。基于Neovim的编辑器还有NvChadSpaceVim,也都非常强大,不过LunarVim默认配置了许多开箱即用的功能。另外,还可以使用code server,不过这个方案直接安装是不可以,需要借助proot-distro安装其他Liunx才可以。如果Termux不能满足你的需求,可以使用proot-distro安装一个Linux来进一步模拟。如果觉得Termux好用的话,可以给它捐款

参考链接:
Termux wiki
Termux 高级终端安装使用配置教程

即时通信IRC服务Oragono

前一篇讲了XMPP通信,还有一种比它还早的协议IRC(Internet Relay Chat),是一种简单的文本聊天协议,支持一对一聊天和群聊(频道)。通常IRC聊天只需要昵称而不需要密码(也可以使用密码),这与XMML不同,使得它可以作为公开的聊天频道,比如游戏直播平台Twitch就使用类似的功能来实现聊天。IRC同样有许多服务端,比如InspIRCd和客户端软件,比如Irssi,也可以直接连接公开的IRC服务器,比如Freenode。事实上IRC协议非常简单,只要能对相应的命令作出响应就可以在各个客户端之间通信,因此有许多不同编程语言的服务端/客户端实现。Oragono便是IRC 服务器的Golang实现,支持IRCv3,支持SASL/LDAP认证,支持服务端保存消息历史记录。
Oragono运行很简单,克隆代码,编辑配置文件就可以了

➜  oragono24 ls
CHANGELOG.md     README           default.yaml     docs             languages        oragono          oragono.motd     traditional.yaml
➜  oragono24 cp default.yaml ircd.yaml
➜  oragono24 ./oragono mkcerts
2020/11/11 12:01:04 making self-signed certificates
2020/11/11 12:01:04  making cert for :6697 listener
2020/11/11 12:01:04   Certificate created at fullchain.pem : privkey.pem
➜  oragono24 ls
CHANGELOG.md     README           default.yaml     docs             fullchain.pem    ircd.yaml        languages        oragono          oragono.motd     privkey.pem      traditional.yaml
➜  oragono24 ./oragono run
2020-11-13T03:09:29.320Z : info  : server     : oragono-2.4.0 starting
2020-11-13T03:09:29.320Z : info  : server     : Using config file : ircd.yaml
2020-11-13T03:09:29.320Z : info  : server     : Using datastore : ircd.db
2020-11-13T03:09:29.344Z : info  : server     : Proxied IPs will be accepted from : localhost
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on [::1]:6667, tls=false, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on :6697, tls=true, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on :8097, tls=true, tlsproxy=false, tor=false, websocket=true.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on 127.0.0.1:6667, tls=false, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : server     : Server running
2020-11-13T03:09:38.386Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:38.439Z : info  : accounts   : client : * : logged into account : kiwi-n30
2020-11-13T03:09:38.442Z : info  : connect    : Client connected [kiwi-n30] [u:~u] [r:https://kiwiirc.com/]
2020-11-13T03:09:40.955Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:40.963Z : info  : connect    : Client connected [kiwi-n28] [u:~u] [r:https://kiwiirc.com/]
2020-11-13T03:09:56.198Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:56.209Z : info  : connect    : Client connected [kiwi-n29] [u:~u] [r:https://kiwiirc.com/]

编辑ircd.yml,设置管理密码,启用一下WebSocket,保存历史消息到MySQL。opers/admin管理密码在认证管理员的时候使用的,可以使用命令oragono genpasswd生成。ircd.db用来保存注册用户,使用的并不是SQLite,而是Golang实现的一个嵌入式的内存Key/Value存储BuntDB,可以持久化到硬盘。

# network configuration
network:
    # name of the network
    name: OragonoTest

# server configuration
server:
    # server name
    name: oragono.test

    # addresses to listen on
    listeners:
        # The standard plaintext port for IRC is 6667. Allowing plaintext over the
        # public Internet poses serious security and privacy issues. Accordingly,
        # we recommend using plaintext only on local (loopback) interfaces:
        "127.0.0.1:6667": # (loopback ipv4, localhost-only)
        "[::1]:6667":     # (loopback ipv6, localhost-only)
        # If you need to serve plaintext on public interfaces, comment out the above
        # two lines and uncomment the line below (which listens on all interfaces):
        # ":6667":
        # Alternately, if you have a TLS certificate issued by a recognized CA,
        # you can configure port 6667 as an STS-only listener that only serves
        # "redirects" to the TLS port, but doesn't allow chat. See the manual
        # for details.

        # The standard SSL/TLS port for IRC is 6697. This will listen on all interfaces:
        ":6697":
            tls:
                cert: fullchain.pem
                key: privkey.pem
                # 'proxy' should typically be false. It's only for Kubernetes-style load
                # balancing that does not terminate TLS, but sends an initial PROXY line
                # in plaintext.
                proxy: false

        # Example of a Unix domain socket for proxying:
        # "/tmp/oragono_sock":

        # Example of a Tor listener: any connection that comes in on this listener will
        # be considered a Tor connection. It is strongly recommended that this listener
        # *not* be on a public interface --- it should be on 127.0.0.0/8 or unix domain:
        # "/hidden_service_sockets/oragono_tor_sock":
        #     tor: true

        # Example of a WebSocket listener:
        ":8097":
            websocket: true
            tls:
                cert: fullchain.pem
                key: privkey.pem
# ircd operators
opers:
    # operator named 'admin'; log in with /OPER admin [password]
    admin:
        # which capabilities this oper has access to
        class: "server-admin"

        # custom whois line
        whois-line: is a server admin

        # custom hostname
        vhost: "staff"

        # normally, operator status is visible to unprivileged users in WHO and WHOIS
        # responses. this can be disabled with 'hidden'. ('hidden' also causes the
        # 'vhost' line above to be ignored.)
        hidden: false

        # modes are modes to auto-set upon opering-up. uncomment this to automatically
        # enable snomasks ("server notification masks" that alert you to server events;
        # see `/quote help snomasks` while opered-up for more information):
        #modes: +is acjknoqtuxv

        # operators can be authenticated either by password (with the /OPER command),
        # or by certificate fingerprint, or both. if a password hash is set, then a
        # password is required to oper up (e.g., /OPER dan mypassword). to generate
        # the hash, use `oragono genpasswd`.
        password: "$2a$04$I2Yhr7UF4p3iyEZcKTPJgukLA9mm00B1wQgicJvGZP/gf0u0tbQQy"
# datastore configuration
datastore:
    # path to the datastore
    path: ircd.db

    # if the database schema requires an upgrade, `autoupgrade` will attempt to
    # perform it automatically on startup. the database will be backed
    # up, and if the upgrade fails, the original database will be restored.
    autoupgrade: true

    # connection information for MySQL (currently only used for persistent history):
    mysql:
        enabled: true
        host: "127.0.0.1"
        port: 3306
        # if socket-path is set, it will be used instead of host:port
        #socket-path: "/var/run/mysqld/mysqld.sock"
        user: "root"
        password: "root"
        history-database: "oragono_history"
        timeout: 30s

IRC服务端设置好了,使用客户端连上去就可以聊天了。IRC协议类似HTTP协议,服务器地址类似irc://irc.freenode.net,但是也可以使用web来连接,前面配置了监听来自8097端口的websocket请求。这里使用web的IRC客户端Kiwi IRC,这是Vue实现的一个web客户端,各个组件之间通过触发/订阅Vue实例状态/事件变化进行通信,支持自定义组件/中间件,这也是Freenode使用的客户端。Kiwiirc主要聚焦连接和消息的处理,比如websocket连接/发送/监听,消息的格式化/通知/声音/状态。它能够连接websocket的IRC服务器,甚至还提供在线生成器,只需要添加websocket url就好了

{
    "windowTitle": "Kiwi IRC with Oragono Test",
    "startupScreen": "welcome",
    "kiwiServer": "http://oragono.test",
    "restricted": false,
    "theme": "Default",
    "themes": [
        { "name": "Default", "url": "static/themes/default" },
        { "name": "Dark", "url": "static/themes/dark" },
        { "name": "Coffee", "url": "static/themes/coffee" },
        { "name": "GrayFox", "url": "static/themes/grayfox" },
        { "name": "Nightswatch", "url": "static/themes/nightswatch" },
        { "name": "Osprey", "url": "static/themes/osprey" },
        { "name": "Radioactive", "url": "static/themes/radioactive" },
        { "name": "Sky", "url": "static/themes/sky" },
        { "name": "Elite", "url": "static/themes/elite" }
    ],
    "startupOptions" : {
        "websocket": "wss://127.0.0.1:8097",
        "channel": "#kiwiirc-default",
        "nick": "kiwi-n?"
    },
    "embedly": {
        "key": ""
    },
    "plugins": [
        { "name": "customise", "url": "static/plugins/customise.html" }
    ]
}

运行登录后的效果


其实IRC是基于TCP连接的文本交互的,也可以使用telnet来模拟

➜  ~ telnet 127.0.0.1 6667
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CAP LS 302
:oragono.test CAP * LS * :account-notify account-tag away-notify batch cap-notify chghost draft/channel-rename draft/chathistory draft/event-playback draft/languages=16,en,~bs,~de,~el,~en-AU,~es,~fi,~fr-FR,~it,~nl,~no,~pl,~pt-BR,~ro,~tr-TR,~zh-CN draft/multiline=max-bytes=4096,max-lines=100 draft/register=before-connect draft/relaymsg=/ draft/resume-0.5 echo-message extended-join invite-notify labeled-response message-tags multi-prefix oragono.io/nope sasl=PLAIN,EXTERNAL server-time setname
:oragono.test CAP * LS :userhost-in-names znc.in/playback znc.in/self-message
NICK kiwi-n26
USER kiwi-n26 0 * https://kiwiirc.com/
CAP REQ :account-notify account-tag away-notify batch cap-notify draft/chathistory extended-join invite-notify message-tags multi-prefix server-time userhost-in-names znc.in/self-message
@time=2020-11-13T01:55:02.201Z :oragono.test CAP * ACK :account-notify account-tag away-notify batch cap-notify draft/chathistory extended-join invite-notify message-tags multi-prefix server-time userhost-in-names znc.in/self-message
CAP END
@time=2020-11-13T01:55:07.714Z :oragono.test 001 kiwi-n26 :Welcome to the Internet Relay Network kiwi-n26
@time=2020-11-13T01:55:07.714Z :oragono.test 002 kiwi-n26 :Your host is oragono.test, running version oragono-2.4.0
@time=2020-11-13T01:55:07.714Z :oragono.test 003 kiwi-n26 :This server was created Fri, 13 Nov 2020 00:35:48 UTC
@time=2020-11-13T01:55:07.714Z :oragono.test 004 kiwi-n26 oragono.test oragono-2.4.0 BERTZios CEIMRUabehiklmnoqstuv Iabehkloqv
@time=2020-11-13T01:55:07.714Z :oragono.test 005 kiwi-n26 AWAYLEN=390 BOT=B CASEMAPPING=ascii CHANLIMIT=#:100 CHANMODES=Ibe,k,l,CEMRUimnstu CHANNELLEN=64 CHANTYPES=# ELIST=U EXCEPTS EXTBAN=,m INVEX KICKLEN=390 MAXLIST=beI:60 :are supported by this server
@time=2020-11-13T01:55:07.714Z :oragono.test 005 kiwi-n26 MAXTARGETS=4 MODES MONITOR=100 NETWORK=OragonoTest NICKLEN=32 PREFIX=(qaohv)~&@%+ STATUSMSG=~&@%+ TARGMAX=NAMES:1,LIST:1,KICK:1,WHOIS:1,USERHOST:10,PRIVMSG:4,TAGMSG:4,NOTICE:4,MONITOR:100 TOPICLEN=390 UTF8MAPPING=rfc8265 WHOX draft/CHATHISTORY=100 :are supported by this server
@time=2020-11-13T01:55:07.714Z :oragono.test 251 kiwi-n26 :There are 0 users and 4 invisible on 1 server(s)
@time=2020-11-13T01:55:07.714Z :oragono.test 252 kiwi-n26 0 :IRC Operators online
@time=2020-11-13T01:55:07.714Z :oragono.test 253 kiwi-n26 0 :unregistered connections
@time=2020-11-13T01:55:07.714Z :oragono.test 254 kiwi-n26 2 :channels formed
@time=2020-11-13T01:55:07.714Z :oragono.test 255 kiwi-n26 :I have 4 clients and 0 servers
@time=2020-11-13T01:55:07.714Z :oragono.test 265 kiwi-n26 4 4 :Current local users 4, max 4
@time=2020-11-13T01:55:07.714Z :oragono.test 266 kiwi-n26 4 4 :Current global users 4, max 4
@time=2020-11-13T01:55:07.714Z :oragono.test 375 kiwi-n26 :- oragono.test Message of the day -
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-       ▄▄▄   ▄▄▄·  ▄▄ •        ▐ ▄
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- ▪     ▀▄ █·▐█ ▀█ ▐█ ▀ ▪▪     •█▌▐█▪
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-  ▄█▀▄ ▐▀▀▄ ▄█▀▀█ ▄█ ▀█▄ ▄█▀▄▪▐█▐▐▌ ▄█▀▄
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- ▐█▌.▐▌▐█•█▌▐█ ▪▐▌▐█▄▪▐█▐█▌ ▐▌██▐█▌▐█▌.▐▌
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-  ▀█▄▀▪.▀  ▀ ▀  ▀ ·▀▀▀▀  ▀█▄▀ ▀▀ █▪ ▀█▄▀▪
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- This is the default Oragono MOTD.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- If motd-formatting is enabled in the config file, you can use the dollarsign character to
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- create special formatting such as bold, italics and color codes.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- For example, here are a few formatted lines (enable motd-formatting to see these in action):
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is bold text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is italics text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is 4red and 2blue text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is 4,12red text with a light blue background.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is a normal escaped dollarsign: $
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- And now a few fun colour charts!
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,0 00 0,1 01 0,2 02 0,3 03 1,4 04 0,5 05 0,6 06 1,7 07
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,8 08 1,9 09 0,10 10 1,11 11 0,12 12 1,13 13 1,14 14 1,15 15
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,16 16 0,17 17 0,18 18 0,19 19 0,20 20 0,21 21 0,22 22 0,23 23 0,24 24 0,25 25 0,26 26 0,27 27
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,28 28 0,29 29 0,30 30 0,31 31 0,32 32 0,33 33 0,34 34 0,35 35 0,36 36 0,37 37 0,38 38 0,39 39
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,40 40 0,41 41 0,42 42 0,43 43 0,44 44 0,45 45 0,46 46 0,47 47 0,48 48 0,49 49 0,50 50 0,51 51
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,52 52 0,53 53 1,54 54 1,55 55 1,56 56 1,57 57 1,58 58 0,59 59 0,60 60 0,61 61 0,62 62 0,63 63
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,64 64 1,65 65 1,66 66 1,67 67 1,68 68 1,69 69 1,70 70 1,71 71 0,72 72 0,73 73 0,74 74 0,75 75
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,76 76 1,77 77 1,78 78 1,79 79 1,80 80 1,81 81 1,82 82 1,83 83 1,84 84 1,85 85 1,86 86 1,87 87
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,88 88 0,89 89 0,90 90 0,91 91 0,92 92 0,93 93 0,94 94 0,95 95 1,96 96 1,97 97 1,98 98 99,99 99
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- For more information on using these, see MOTDFORMATTING.md
@time=2020-11-13T01:55:07.714Z :oragono.test 376 kiwi-n26 :End of MOTD command
@time=2020-11-13T01:55:07.715Z :oragono.test 221 kiwi-n26 +Zi
WHO kiwi-n26
@time=2020-11-13T01:55:15.649Z :oragono.test 352 kiwi-n26 * ~u gcjc79gmtbe42.irc oragono.test kiwi-n26 H :0 https://kiwiirc.com/
@time=2020-11-13T01:55:15.649Z :oragono.test 315 kiwi-n26 kiwi-n26!*@* :End of WHO list
JOIN #kiwiirc-default
@msgid=bidkvpja5njjtgchub8u65pxgn;time=2020-11-13T01:55:26.706Z :[email protected] JOIN #kiwiirc-default * https://kiwiirc.com/
@time=2020-11-13T01:55:26.706Z :oragono.test 353 kiwi-n26 = #kiwiirc-default :@[email protected] [email protected] [email protected] [email protected]
@time=2020-11-13T01:55:26.706Z :oragono.test 366 kiwi-n26 #kiwiirc-default :End of NAMES list
JOIN #ops
@msgid=hftqmpu4ieyjybx9bbgf4vpbia;time=2020-11-13T01:55:32.145Z :[email protected] JOIN #ops * https://kiwiirc.com/
@time=2020-11-13T01:55:32.145Z :oragono.test 353 kiwi-n26 = #ops :@[email protected] [email protected] [email protected] [email protected]
@time=2020-11-13T01:55:32.145Z :oragono.test 366 kiwi-n26 #ops :End of NAMES list
PRIVMSG #ops hello
PRIVMSG #ops :kiwi-n28 good morning
PRIVMSG kiwi-n28 =D
PING kiwitime-1605232555315
@time=2020-11-13T01:57:14.150Z :oragono.test PONG oragono.test kiwitime-1605232555315
CHATHISTORY BEFORE kiwi-n28 * 50
@time=2020-11-13T01:57:38.462Z :oragono.test BATCH +1 chathistory kiwi-n28
@msgid=rqdbvmytzppssc5ymp7ffhmkga;time=2020-11-13T01:56:05.705Z;batch=1 :[email protected] PRIVMSG kiwi-n28 :=D
@time=2020-11-13T01:57:38.462Z :oragono.test BATCH -1
@time=2020-11-13T01:59:08.466Z PING kiwi-n26
:[email protected] QUIT :Ping timeout: 2m30s
ERROR :Ping timeout: 2m30s
Connection closed by foreign host.

可以看到这些消息都带有一定的格式,客户端根据特地的格式解析就可以个性化显示来。带有符号@/:前缀的消息都是服务器推送下来的消息,反之都是客户端请求的消息。“CAP LS 302”即协商会话协议版本,随即用命令“NICK ”设置昵称(登录名),和“USER 0 * ”设置该昵称显示的用户名,最后以“CAP END”结束协商,开始会话。服务端推送过来的消息格式为时间戳+服务器+响应码+接受者昵称+消息内容,即“@time=2020-11-13T01:55:07.714Z :oragono.test 001 :Welcome to the Internet Relay Network ”,001代表欢迎信息,具体响应码可以参考这里。客户端发送消息则直接发送,比如使用“JOIN #channel”加入某个频道,服务器会向该频道里的每个人广播对JOIN应的消息“@msgid=hftqmpu4ieyjybx9bbgf4vpbia;time=2020-11-13T01:55:32.145Z :@gcjc79gmtbe42.irc JOIN #channel * ”并向操作者推送频道用户列表“@time=2020-11-13T01:55:32.145Z :oragono.test 353 = #channel :@[email protected] [email protected] [email protected]
”。Twitch则在消息的基础上加入了其他格式化控制,比如

@badge-info=;badges=premium/1;client-nonce=2f70a1efd6f84523d0f180cc44f76c61;color=#B22222;display-name=Khodeine;emotes=58765:9-19;flags=;id=50d8790f-c474-42a8-b873-2d5e8008bd52;mod=0;room-id=30220059;subscriber=0;tmi-sent-ts=1605279854692;turbo=0;user-id=49980352;user-type= :[email protected] PRIVMSG #esl_sc2 :the scvs NotLikeThis

会显示以一定的颜色/图标显示用户名/消息。空闲状态下IRC服务端与客户端需要维持心跳信息,比如“PING kiwitime-1605232555315”和“time=2020-11-13T01:57:14.150Z :oragono.test PONG oragono.test kiwitime-1605232555315”,否则连接会被关闭。一些常用的命令,可以看出IRC交互非常简单,IRC核心组件也不多。使用“@+dratf/typing=active TAGMSG ”显示输入状态,借助CTCP协议甚至可以点对点发送文件

#使用password注册当前nickname
/NS REGISTER <your password>
#注册一个频道
/CS REGISTER #channelname
/CS REGISTER #channe
#要去某人加入频道
/INVITE nickname #channel
#给予某个用户频道管理权限
/mode +o nickname
#认证成为管理员
/OPER admin <admin passowrd>
#在频道发送消息
/PRIVMSG #channe hello
#给频道某人发送消息
/PRIVMSG #channe :<nick> good morning
#给频道某人发送消息
/INVITE <nick> #channe
#使用密码登录
AUTHENTICATE PLAIN
AUTHENTICATE +
AUTHENTICATE a2l3aS1uMzAAa2l3aS1uMzAAMTAyMDMwNDA=

Thelounge是另外一个Vue实现的web的IRC客户端,同样支持websocket。它还用express实现了服务端,支持额外的密码/LDAP登录,即先认证通过才能连接IRC服务器,它本身也可以作为一个服务代理网关,为用户提供永远在线服务/消息存储-如果一个IRC用户不在线是不能够给它发送消息的,这一点也跟XMPP不一样。


Kiwiirc和Thelounge的设计实现良好,可以作为web chat实现的参考,比如在线客服/销售。比较两个web的IRC客户端发现,他们使用的websocket库并不一样。Kiwiirc使用的是sockjs-client,一个类似websocket的JavaScript网络连接库,支持不同浏览器,能够跨域名通信,在浏览器不支持websocket的情况下退化为polling(需要服务端配合)。而Thelounge使用的是socket.io。Socket.IO由Engine.IO发展而来,致力于为web提供实时通信能力。Socket.IO并不是WebSocket的一个实现,只是提供了类似的连接能力,需要服务端配合。两个连接库在浏览器支持websocket的情况下都可以直接与websocket服务端通信(不需要对应的库)。Kiwiirc还提供了irc-framework开发库,可以快速搭建一个IRC客户端,比如IRC Bot。至今仍然存在着许多IRC聊天机器人,为用户提供方便/服务器报警。Twitch一个用户加入频道时甚至能收到广告!

Websocket提供了与其他socket一样的能力,能够持续连接和双向通讯,甚至可以服用80/443端口(nginx支持101协议升级)。它不止可以使用在web上,也可以使用在非web上面,因为能够穿透防火墙,对树莓派的远程链接支持。传统的web应用都是一应一答,这节省的服务器/客户端的资源能够不需要持续请求的话。而对于有些应用而言,比如消息通知,希望是能够持续从服务器推送的,websocket增强了它的能力,在这之前都是long polling(所以才有那些包装库)。
Http协议在tcp基础上实现,请求头里面带有host、cookie,url里面带有路径等,这样后端能够识别、路由、分发请求。而tcp并没有这些,websocket必须自己实现类似的路由分发/调用,可以在连接建立初期,鉴别授权(可以在url上面带参数token/jwt/security key),对于后续消息的应答可以在url上面指定,也可以在事件上面指定。这样实现出来似乎与http差不多,仍然有event对应route,但却可以持续交互。
再观察IRC协议也不过是对各个命令的应答,其实也跟经典web MVC差不多-使用URL映射对应的controller/action。IRC每个会话都是一个队列,join #channel命令即订阅消息,part #channel命令则是取消订阅。再看Rabbitmq消息队列,实现了exchange,route,queue,channel,topic等,然后是持续的消费/推送,与前面所诉差不多,这样子看IRC也是个消息服务器。IRC实现简单,通过扩展支持的命令,也能够与内部/外部系统结合,快速搭建一个即时消息平台。
再看邮件服务,其实也是消息服务,也有一对一/群发消息/历史消息,甚至能够给不认识(不在线)的人发送邮件!而邮件服务至今流行,在许多平台,不论服务器还是手机都自带,不需要额外安装软件。Slack的产品宣传即对比了自己相对邮件服务的优势。其实手机默认带的电话和短信功能是基于硬件模块的在线功能。
相比Ejabberd/Openfire,IRC服务器的部署简单许多,甚至可以部署在树莓派上!

参考连接:
IRC Bot
基于 RabbitMQ 的实时消息推送

即时通信XMPP服务器ejabberd/Openfire

即时通信的需求,不仅在公共互联网上,比如微信、QQ,也在企业的内部网络,特别是与内部系统的整合,比如OA、客服系统。XMPP,也叫扩展消息与存在协议,是一种以XML为基础的开放式即时通信协议。XMPP不但能够用来做消息通信(单聊/群聊/订阅),也可以做语音通信。XMPP甚至支持多个服务器之间互相连接,互相通信,早期Gtalk就能够与其他XMPP服务器通信。采用XMPP来开发的好处是,已经有大量的开源服务器(ejabberd、Openfire)和客户端(Spark/Smack/converse.js/XMPPFramework)实现,能够快速搭建。
Ejabberd则是ProcessOne出品的,基于Erlang开发的XMPP/MQTT/SIP服务器,内置了WebSocket/文件上传支持,提供更加丰富的REST API。在CentOS64下面安装Ejabberd,会安装到/etc/init.d/ejabberd,直接service start ejabberd就好了,可以使用docker快速启用

[root@vagrant-centos64 tmp]# wget https://static.process-one.net/ejabberd/downloads/20.04/ejabberd-20.04-0.x86_64.rpm
--2020-10-29 08:26:16--  https://static.process-one.net/ejabberd/downloads/20.04/ejabberd-20.04-0.x86_64.rpm
Resolving static.process-one.net... 13.226.36.69, 13.226.36.105, 13.226.36.34, ...
Connecting to static.process-one.net|13.226.36.69|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18713774 (18M) [application/x-rpm]
Saving to: “ejabberd-20.04-0.x86_64.rpm”

100%[====================================================================================================================================================================================================================================>] 18,713,774  2.84M/s   in 9.1s    

2020-10-29 08:26:27 (1.96 MB/s) - “ejabberd-20.04-0.x86_64.rpm” saved [18713774/18713774]

[root@vagrant-centos64 tmp]# yum localinstall ejabberd-20.04-0.x86_64.rpm
Loaded plugins: fastestmirror
Setting up Local Package Process
Examining ejabberd-20.04-0.x86_64.rpm: ejabberd-20.04-0.x86_64
Marking ejabberd-20.04-0.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * epel: mirror.math.princeton.edu
 * extras: mirrors.cn99.com
 * updates: mirrors.cn99.com
 * webtatic: uk.repo.webtatic.com
phalcon_stable/signature                                                                                                                                                                                                                               |  819 B     00:00     
phalcon_stable/signature                                                                                                                                                                                                                               |  951 B     00:00 ... 
phalcon_stable-source/signature                                                                                                                                                                                                                        |  819 B     00:00     
phalcon_stable-source/signature                                                                                                                                                                                                                        |  951 B     00:00 ... 
Resolving Dependencies
--> Running transaction check
---> Package ejabberd.x86_64 0:20.04-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                       Arch                                                        Version                                                        Repository                                                                     Size
==============================================================================================================================================================================================================================================================================
Installing:
 ejabberd                                                      x86_64                                                      20.04-0                                                        /ejabberd-20.04-0.x86_64                                                       29 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install       1 Package(s)

Total size: 29 M
Installed size: 29 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : ejabberd-20.04-0.x86_64                                                                                                                                                                                                                                    1/1 
  Verifying  : ejabberd-20.04-0.x86_64                                                                                                                                                                                                                                    1/1 

Installed:
  ejabberd.x86_64 0:20.04-0                                                                                                                                                                                                                                                   

Complete!
[root@vagrant-centos64 tmp]# ls  /etc/init.d/ejabberd
/etc/init.d/ejabberd
[root@vagrant-centos64 tmp]# ls -la /opt/ejabberd
total 24
drwxr-xr-x  5 ejabberd ejabberd 4096 Oct 29 08:27 .
drwxr-xr-x. 8 root     root     4096 Oct 29 08:27 ..
drwxr-xr-x  2 ejabberd ejabberd 4096 Oct 29 08:27 conf
drwxr-xr-x  3 ejabberd ejabberd 4096 Oct 29 08:27 database
-r--------  1 ejabberd ejabberd   20 Oct 29 00:00 .erlang.cookie
drwxr-xr-x  2 ejabberd ejabberd 4096 Oct 29 08:27 logs
[root@vagrant-centos64 bin]# service ejabberd start
Starting ejabberd...
done.
[root@vagrant-centos64 sql]# ps aux | grep ejabberd
ejabberd 10097  0.2  7.9 1817344 48144 ?       Sl   08:51   0:04 /opt/ejabberd-20.04/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-20.04 -progname /opt/ejabberd-20.04/bin/erl -- -home /opt/ejabberd -- -sname ejabberd@localhost -smp enable -mnesia dir "/opt/ejabberd/database/ejabberd@localhost" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -noshell -noinput
ejabberd 10105  0.0  0.0   4060   496 ?        Ss   08:51   0:00 erl_child_setup 1024
ejabberd 10186  0.0  0.0   4052   572 ?        Ss   08:51   0:00 /opt/ejabberd-20.04/lib/os_mon-2.4.7/priv/bin/memsup
root     11215  0.0  0.1 103324   892 pts/4    S+   09:16   0:00 grep ejabberd
[root@vagrant-centos64 bin]# netstat -apn | grep beam
tcp        0      0 127.0.0.1:7777              0.0.0.0:*                   LISTEN      10097/beam.smp      
tcp        0      0 0.0.0.0:37617               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:15672               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:55672               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:43674               0.0.0.0:*                   LISTEN      10097/beam.smp      
tcp        0      0 127.0.0.1:44055             127.0.0.1:4369              ESTABLISHED 1599/beam           
tcp        0      0 127.0.0.1:35415             127.0.0.1:4369              ESTABLISHED 10097/beam.smp      
tcp        0      0 :::5280                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5443                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5222                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5672                     :::*                        LISTEN      1599/beam           
tcp        0      0 :::5269                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::1883                     :::*                        LISTEN      10097/beam.smp      
unix  3      [ ]         STREAM     CONNECTED     1637533 10097/beam.smp   

Ejabberd自带Erlang的数据库服务Mnesia,但有一些限制建议使用第三方的比如MySQL,这样也方便第三方交互(比如认证Token)。初始化的SQL脚本在/opt/ejabberd-20.04/lib/ejabberd-20.04/priv/sql,如果只是支持单个domain的话使用mysql.sql就可以了,多domain支持使用mysql.new.sql

[root@vagrant-centos64 sql]# ls
lite.new.sql  lite.sql  mssql.sql  mysql.new.sql  mysql.sql  pg.new.sql  pg.sql
[root@vagrant-centos64 sql]# pwd
/opt/ejabberd-20.04/lib/ejabberd-20.04/priv/sql

更改为使用MySQL存储和认证

[root@vagrant-centos64 bin]# cat /opt/ejabberd/conf/ejabberd.yml
###
###'           ejabberd configuration file
###
### The parameters used in this configuration file are explained at
###
###       https://docs.ejabberd.im/admin/configuration
###
### The configuration file is written in YAML.
### *******************************************************
### *******           !!! WARNING !!!               *******
### *******     YAML IS INDENTATION SENSITIVE       *******
### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******
### *******************************************************
### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.
###

hosts:
  - "vagrant-centos64"

loglevel: 4
log_rotate_size: 10485760
log_rotate_date: ""
log_rotate_count: 1
log_rate_limit: 100

certfiles:
  - "/opt/ejabberd/conf/server.pem"
##  - "/etc/letsencrypt/live/localhost/fullchain.pem"
##  - "/etc/letsencrypt/live/localhost/privkey.pem"

ca_file: "/opt/ejabberd/conf/cacert.pem"

listen:
  -
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
  -
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
  -
    port: 5443
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
      "/admin": ejabberd_web_admin
      "/api": mod_http_api
      "/bosh": mod_bosh
      "/captcha": ejabberd_captcha
      "/upload": mod_http_upload
      "/ws": ejabberd_http_ws
      "/oauth": ejabberd_oauth
  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    request_handlers:
      "/admin": ejabberd_web_admin
  -
    port: 1883
    ip: "::"
    module: mod_mqtt
    backlog: 1000

s2s_use_starttls: optional

acl:
  local:
    user_regexp: ""
  loopback:
    ip:
      - 127.0.0.0/8
      - ::1/128
      - ::FFFF:127.0.0.1/128
  admin:
    user:
      - "admin@vagrant-centos64"

access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: local
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback

api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
      - mod_http_api
    who: all
    what: "*"
  "admin access":
    who:
      access:
        allow:
          acl: loopback
          acl: admin
      oauth:
        scope: "ejabberd:admin"
        access:
          allow:
            acl: loopback
            acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who:
      ip: 127.0.0.1/8
    what:
      - status
      - connected_users_number

shaper:
  normal: 1000
  fast: 50000

shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast

max_fsm_queue: 10000

acme:
   contact: "mailto:admin@vagrant-centos64"
   ca_url: "https://acme-v01.api.letsencrypt.org"

modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}
  mod_http_upload:
    put_url: https://@HOST@:5443/upload
  mod_last: {}
  mod_mam:
    ## Mnesia is limited to 2GB, better to use an SQL backend
    ## For small servers SQLite is a good fit and is very easy
    ## to configure. Uncomment this when you have SQL configured:
    ## db_type: sql
    assume_mam_usage: true
    default: never
  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      allow_subscription: true  # enable MucSub
      mam: false
  mod_muc_admin: {}
  mod_offline:
    access_max_user_messages: max_user_offline_messages
  mod_ping: {}
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      ## Avoid buggy clients to make their bookmarks public
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  mod_register:
    ## Only accept registration requests from the "trusted"
    ## network (see access_rules section above).
    ## Think twice before enabling registration from any
    ## address. See the Jabber SPAM Manifesto for details:
    ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
    ip_access: trusted_network
  mod_roster:
    versioning: true
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_stream_mgmt:
    resend_on_timeout: if_offline
  mod_vcard: {}
  mod_vcard_xupdate: {}
  mod_version:
    show_os: false

auth_method: sql


sql_type: mysql
sql_server: "localhost"
sql_database: "ejabberd"
sql_username: "ejabberd"
sql_password: "ejabberd"

default_db: sql
## If you want to specify the port:
#sql_port: 3306
### Local Variables:
### mode: yaml
### End:
### vim: set filetype=yaml tabstop=8

试着注册一下管理员

[root@vagrant-centos64 bin]# ./ejabberdctl register admin1 localhost admin
Error: cannot_register

失败了,这是因为hostname的关系,在ejabberd.yml配置的host并不是localhost,改一下注册的domain就好了,具体可以参考这里这里

[root@vagrant-centos64 bin]# ./ejabberdctl status
The node ejabberd@localhost is started with status: started
[root@vagrant-centos64 logs]# hostname -s
vagrant-centos64
[root@vagrant-centos64 bin]# ./ejabberdctl register admin vagrant-centos64 admin
User admin@vagrant-centos64 successfully registered

访问http://127.0.0.1:5280/admin就可以进入到管理界面了。管理界面比较简单,只能管理用户/消息/聊天室,完整的xmpp协议服务器是支持的,客户端对应实现就好了。


使用conversejs快速搭建一个web客户端来验证一下,创建chat.html

<html>
    <head>
        <title>chat</title>
        <link rel="stylesheet" type="text/css" media="screen" href="https://cdn.conversejs.org/dist/converse.min.css">
        <script src="https://cdn.conversejs.org/dist/converse.min.js" charset="utf-8"></script>
    </head>
    <body>
        <script>
            converse.initialize({
                //bosh_service_url: 'http://chat.vagrant-centos64.com/bosh',
                websocket_url: 'wss:chat.vagrant-centos64.com/ws',
                //show_controlbox_by_default: true,
                view_mode: 'fullscreen'
            });
        </script>
    </body>
</html>

打开浏览器,访问对应的url登录就可以看到了聊天界面了。这里使用的域名是经过nginx代理转发过的了,配置可以参考API 网关 Kong,如果使用nginx-proxy-servier需要打开Websockets Support,以便进行协议升级101 Switching Protocols。



conversejs配置里面有两个url,一个是BOSH(Bidirectional-streams Over Synchronous HTTP)的,即HTTP长连接,以便通服务器即时交互,获取消息;另一个是WebSocket,基于TCP的,客户端/服务器双向通信。BOHS需要浏览器定时发起一个请求直致服务器返回消息,而WebSocket可以像其他TCP那样双向通信,灵活的多。如果服务端不支持Websocket协议升级或者连接失败conversejs会自动切换使用BOSH通信。如果使用https://www.websocket.org/echo.html来测试连接,需要遵循XMPP协议要求,先认证,否则连接会被关闭。conversejs的认证是在Websocket里面的做的,并不基于cookie之类的,所以一个浏览器打开多个聊天窗口登录不同的账户也是可以的



Ejabberd支持REST API来做交互比如查看在线用户、发送消息

➜  chat-example git:(master) ✗ curl -k https://192.168.33.14:5443/api/connected_users
["admin@vagrant-centos64/converse.js-16142875"]                                                                                                                                                                                                                              ➜  chat-example git:(master) ✗ curl -k https://192.168.33.14:5443/api/send_message -X POST -d '{"type":"headline","from":"test@vagrant-centos64","to":"admin@vagrant-centos64","subject":"Restart","body":"In 5 minutes"}'

这些API将大大增加ejabberd与第三方软件的交互。虽然ejabberd支持OAuth认证,但那是以ejabberd为账户中心的认证,方便其他系统调用ejabberd功能。通常即时通信只是内部系统的一部分,账户中心部署在其他地方,所以需要ejabberd支持外部的认证。前边已经配置ejabberd为数据库认证,还可以配置为使用LDAP认证。如果不满足,还可以配置为外部脚本认证或者使用第三方开发的HTTP认证。对于简单的内部交互,可以将认证服务的token刷新到ejabberd的数据库即可。
Openfire则是ignite realtime出品的Java实现的XMPP服务器,同时提供Java客户端Spark、Java开发库Smack,还提供Chrome扩展Pade。Openfire提供基于web的管理界面,支持LDAP登录及数据库存储。有一些功能Openfire并不直接支持,比如API,而是以扩展的形式支持,包括用户管理/分组管理/聊天室管理/消息广播/邮件监听/WebSocket/Meeting/Sip等等。

用户管理

配置用户分组,可以配置部门之类的,会出现在用户的个人分组里面(Spark)

在线会话管理,可以踢人


创建聊天室之前需要创建对应的service,默认的service叫conference。
Openfire的API并不支持OAuth/SSO,简单的1V1消息发送,聊天室消息订阅等等,需要自己基于Java扩展。
Ejabberd是采用Erlang开发的,一如消息队列服务器RabbitMQ,具有极高吞吐能力,提供REST API/XMl RPC,方便交互;Openfire则易于管理和扩展,采用哪个软件进行开发需要结合企业实际进行考量。
XMPP服务器主要提供消息聊天,对于语音服务可以使用SIP服务器实现,结合客户端sip.js,比如ctxSip

参考链接:
TCP UDP探索
SIP(会话发起协议)

API 网关 Kong

从前开发一个互联网服务程序,大概可以在一台机器上完成:数据库、应用都在一起。随着业务发展壮大,会把数据库独立出来,以便扩展拆分。然后再把一部分公用业务独立出来扩展,譬如文件存储、缓存等。接着业务也才拆分,比如会员、商品。微服务大行其道,各个团队维护着许多服务、API。这么多服务,前端业务逻辑该怎么接入呢?
如今单一的前端UI也可能是多个团队共同开发的结果。当你在本地开发一个小功能时,甚至会牵扯到多个前端UI/后端API。一个前端页面除了加载自己的资源外,还加载V2/V3的API(为什么会同时存在V1/V2/V3的API),甚至嵌入了另外一个页面(React看起来也不错),该怎么调试这种混合开发呢?cookie都传递不过去,好吧,使用JWT代替,再设置一下跨源资源共享(CORS)。。。但是那些旧的应用怎么办?
当我们启动一个Node.js应用时npm run start,默认监听3000端口。当然我们也可以让它监听80端口。但是当我们多开几个应用时,只好让它们都监听不同端口了,怎么样才能统一端口监听呢?当然Node.js可以通过诸如node-http-proxy来转发这些请求,但是代理转发跟应用业务无关吧?如果是其他编程语言呢?都重复这些代理配置/开发吗?Don’t Repeat Yourself。
Faas也日渐流行,比如AWS Lambda, 强调仅仅专注某个功能,根据事件驱动进行计算,那么每个服务也要开发一整套路由分发/认证/日志吗?
这些问题可以使用反向代理来解决,提供统一的服务入口,对前端/客户端隐藏背后的细节,最简单的当然是Nginx。Nginx监听来自某个端口(比如80)的请求,然后根据不同的来源/端口/域名/url分发给不同的后端服务器。这看起来跟云服务厂商各自开发的负载均衡器差不多,比如AWS ELB
在本地开发的时候,我们当然不会使用ELB来解决。直接使用Nginx当然也没啥问题,编辑一下配置文件,重启应用。但是如果有个UI就更好了,如果还有API就非常好了-这样就能方便的在线注册/更改路由而不需要重启服务器,在这个快速发展、弹性开发的年代更需要这个能力。
Kong是一个基于OpenResty的网关服务器,可以进行路由(转发/负载),插件(日志/认证/监控)管理,并提供RESTful API。而OpenResty 是一个基于Nginx与Lua的高性能Web平台,使用Lua来构建动态网关。听起来像是在Nginx上面编程,这跟在Apache上面使用PHP模块进行编程有什么区别?最大的区别在于,这里的编程对象是Nginx(或者公共模块),扩展Nginx能力,比如负载均衡/日志/认证/监控,而不是输出web页面/业务逻辑。这些东西抽出来以后,就不需要每个模块再重复开发了,比如认证/安全。
在本地可以使用docker来运行Kong服务,麻烦在于数据库迁移工作。通常部署Kong需要几个步骤,比如初始化数据库,迁移升级等等。网上的配置大都过时了,这里可以使用官方提供docker-compose.yml来做数据库工作,并且使用Konga作为管理UI

➜  kong ls
POSTGRES_PASSWORD         data                      docker-compose.yml
➜  kong ls data
postgresql
➜  kong cat POSTGRES_PASSWORD
kong
➜  kong cat docker-compose.yml
version: '3.7'

volumes:
  kong_data: {}

networks:
  kong-net:
    external: false

services:
  kong-migrations:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations bootstrap
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  kong-migrations-up:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations up && kong migrations finish
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  kong:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    user: "${KONG_USER:-kong}"
    depends_on:
      - db
    environment:
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: '0.0.0.0:8001'
      KONG_CASSANDRA_CONTACT_POINTS: db
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    ports:
      - "8000:8000/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "8443:8443/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  db:
    image: postgres:9.5
    environment:
      POSTGRES_DB: ${KONG_PG_DATABASE:-kong}
      POSTGRES_USER: ${KONG_PG_USER:-kong}
      POSTGRES_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "${KONG_PG_USER:-kong}"]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
    stdin_open: true
    tty: true
    networks:
      - kong-net
    volumes:
      - /Users/xxxx/docker/kong/data/postgresql:/var/lib/postgresql/data

  konga:
    image: pantsel/konga
    environment:
      TOKEN_SECRET: channing.token
      DB_ADAPTER: postgres
      DB_HOST: db
      DB_USER: ${KONG_PG_USER:-kong}
      DB_PASSWORD: kong
      DB_DATABASE: ${KONG_PG_DATABASE:-kong}
    ports:
     - 1337:1337
    networks:
     - kong-net

    depends_on:
      - db

secrets:
  kong_postgres_password:
    file: ./POSTGRES_PASSWORD

运行docker-composer up就可以看到

➜  kong docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                             PORTS                                                                                                NAMES
20ebe7885c0d        kong:latest                       "/docker-entrypoint.…"   2 hours ago         Up 11 seconds (healthy)            0.0.0.0:8000->8000/tcp, 127.0.0.1:8001->8001/tcp, 0.0.0.0:8443->8443/tcp, 127.0.0.1:8444->8444/tcp   kong_kong_1
4a7a39c863ae        pantsel/konga                     "/app/start.sh"          2 hours ago         Up 11 seconds                      0.0.0.0:1337->1337/tcp                                                                               kong_konga_1
aa732758fc51        postgres:9.5                      "docker-entrypoint.s…"   2 hours ago         Up 11 seconds (health: starting)   5432/tcp                                                                                             kong_db_1

访问http://127.0.0.1:1337/即可以进入Konga 管理界面了。8000/8443端口是需要监听转发的端口,8001/8444则是Kong RESTFUl管理API的端口。这里也可以把8000/8443改为80/443,这样访问的时候就可以直接使用域名/localhost而不必加端口了。
注册登录进去后首先要添加Kong RESTFUl管理API的地址,这里使用的是docker环境,IP是动态分配的,所以使用链接名就可以了

连接上Kong API后可以看到dashboard列出了支持的插件

Kong里面的管理对象是service,路由转发/插件都是围绕service展开的,添加一个service


在它上面添加一个路由,这里我们在服务只监听以/api开头的url,并转发到后端服务器去


注意这里在route里面的path里面也需要添加/api,否则转发的时候会出错(多拼api)。最简单就是service监听不指定path,在route里指定。一个service下面可以有多个转发路由,每个可以独立管理、设置超时等。
还可以为每个service绑定不同的插件进行处理,比如认证/安全/日志等等,这样就不用在不同的系统里面重复开发这些功能,使得各个团队更加专注也本业务开发


有效插件还可以进行流量限制、请求头/响应头修改等等。这些插件的功能需要基于consumer来开发管理,功能可以非常强大,详细可以参考文档
Kong还有一项功能Upstream配置,与Nginx的ngx_http_upstream_module差不多,可以作为负载均衡、流量分发控制使用。可以用命令行测试基于hostname的路由,

➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev1.example.com'
HTTP/1.1 404 Not Found
Date: Tue, 27 Oct 2020 08:41:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.1.4

{"message":"no Route matched with those values"}                                                                                                                                                                                                                             ➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev.example.com'

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Tue, 27 Oct 2020 08:42:21 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.6.40
Set-Cookie: PHPSESSID=0uc3aoc735j21sk3ni5s72vjc1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Access-Control-Allow-Headers: X-CSRF-Token,Authorization,X-Accept-Charset,X-Accept,Content-Type
X-Kong-Upstream-Latency: 8397
X-Kong-Proxy-Latency: 0
Via: kong/2.1.4

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...

作为一个API网关,Kong能做的很多。如果只是在本地简单的做反向代理,可以使用nginx-proxy-manager,这也是一个基于Nginx开发的网关,根据WEB UI动态生成Nginx配置文件,然后执行/usr/sbin/nginx -s reload生效。有些参数不能通过UI配置(比如超时设置),可以直接写Nginx配置,upstream则支持直接转发tcp请求,支持websockets代理转发,甚至集成了Let’s encrypt自动申请SSL证书(需要验证域名)


本地同样可以使用docker跑起来,默认监听80/443端口,81端口即管理界面

➜  nginx-proxy-manager ls -lah
total 16
drwxr-xr-x   6 xxxx domain users   192B Sep 29 10:24 .
drwxr-xr-x  15 xxxx domain users   480B Sep 29 09:15 ..
-rw-r--r--   1 xxxx domain users   2.3K Sep 29 09:42 config.json
drwxr-xr-x   8 xxxx domain users   256B Sep 29 09:31 data
-rw-r--r--   1 xxxx domain users   740B Sep 29 10:24 docker-compose.yml
drwxr-xr-x   3 xxxx domain users    96B Oct 27 16:38 letsencrypt
➜  nginx-proxy-manager cat docker-compose.yml
version: "3"
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: always
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
      # TCP Forward Example:
      - '8022:8022'
    volumes:
      # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: jc21/mariadb-aria
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql
➜  nginx-proxy-manager docker ps
CONTAINER ID        IMAGE                             COMMAND             CREATED             STATUS                PORTS                                                                    NAMES
3dd58e9cff1f        jc21/nginx-proxy-manager:latest   "/init"             4 weeks ago         Up 7 days (healthy)   0.0.0.0:80-81->80-81/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8022->8022/tcp   nginx-proxy-manager_app_1
a033151b28ed        jc21/mariadb-aria                 "/scripts/run.sh"   4 weeks ago         Up 7 days             3306/tcp                                                                 nginx-proxy-manager_db_1

nginx-proxy-manager会将配置写在data目录下面,可以直接编辑这些文件,Nginx reload之后便会生效,可以看到这些文件,具体的加载规则参考文档

➜  nginx-proxy-manager ls data/nginx
dead_host        default_host     default_www      dummycert.pem    dummykey.pem     proxy_host       redirection_host stream           temp
➜  nginx-proxy-manager cat data/nginx/proxy_host/1.conf
# ------------------------------------------------------------
# dev.example.com
# ------------------------------------------------------------
server {
  set $forward_scheme http;
  set $server         "192.168.33.14";
  set $port           80;

  listen 80;
listen [::]:80;

  server_name dev.example.com;

  access_log /data/logs/proxy_host-1.log proxy;

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

  location /Login {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://login.example.com:443;

  }

  location /api/v2 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.14:9070;

  }

  location /api/v3 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;

  }

  location /Chat {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;

  }

  location /Catalog {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location /static {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location /css {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location / {

    # Proxy!
    include conf.d/include/proxy.conf;
  }

  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}

➜  nginx-proxy-manager cat data/nginx/stream/1.conf
# ------------------------------------------------------------
# 8022 TCP: 1 UDP: 0
# ------------------------------------------------------------
server {
  listen 8022;
listen [::]:8022;

  proxy_pass 192.168.33.14:22;

  # Custom
  include /data/nginx/custom/server_stream[.]conf;
  include /data/nginx/custom/server_stream_tcp[.]conf;
}

在docker-compose.yml里面设置一下upstream监听转发的端口,就可以通过8022端口访问192.168.33.14的22端口了

➜  nginx-proxy-manager telnet 127.0.0.1 8022
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3
^C^C^C^C^C^C^CConnection closed by foreign host.

不论是Kong还是nginx-proxy-manager均有提供API,极大的增强服务网关的可编程性,为动态上线/弹性扩展/自动化运维提供了便利。

参考链接:
从IaaS到FaaS—— Serverless架构的前世今生
聊一聊微服务网关 Kong
KONG网关 — KongA管理UI使用
云原生架构下的 API 网关实践: Kong (二)
微服务 API 网关 -Kong 详解
Creating a web API with Lua using Nginx OpenResty
Nginx基于TCP/UDP端口的四层负载均衡(stream模块)配置梳理
Nginx支持TCP代理和负载均衡-stream模块
聊聊 API Gateway 和 Netflix Zuul
Envoy 是什么?