分类目录归档:服务器

即时通信IRC服务Oragono

前一篇讲了XMPP通信,还有一种比它还早的协议IRC(Internet Relay Chat),是一种简单的文本聊天协议,支持一对一聊天和群聊(频道)。通常IRC聊天只需要昵称而不需要密码(也可以使用密码),这与XMML不同,使得它可以作为公开的聊天频道,比如游戏直播平台Twitch就使用类似的功能来实现聊天。IRC同样有许多服务端,比如InspIRCd和客户端软件,比如Irssi,也可以直接连接公开的IRC服务器,比如Freenode。事实上IRC协议非常简单,只要能对相应的命令作出响应就可以在各个客户端之间通信,因此有许多不同编程语言的服务端/客户端实现。Oragono便是IRC 服务器的Golang实现,支持IRCv3,支持SASL/LDAP认证,支持服务端保存消息历史记录。
Oragono运行很简单,克隆代码,编辑配置文件就可以了

➜  oragono24 ls
CHANGELOG.md     README           default.yaml     docs             languages        oragono          oragono.motd     traditional.yaml
➜  oragono24 cp default.yaml ircd.yaml
➜  oragono24 ./oragono mkcerts
2020/11/11 12:01:04 making self-signed certificates
2020/11/11 12:01:04  making cert for :6697 listener
2020/11/11 12:01:04   Certificate created at fullchain.pem : privkey.pem
➜  oragono24 ls
CHANGELOG.md     README           default.yaml     docs             fullchain.pem    ircd.yaml        languages        oragono          oragono.motd     privkey.pem      traditional.yaml
➜  oragono24 ./oragono run
2020-11-13T03:09:29.320Z : info  : server     : oragono-2.4.0 starting
2020-11-13T03:09:29.320Z : info  : server     : Using config file : ircd.yaml
2020-11-13T03:09:29.320Z : info  : server     : Using datastore : ircd.db
2020-11-13T03:09:29.344Z : info  : server     : Proxied IPs will be accepted from : localhost
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on [::1]:6667, tls=false, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on :6697, tls=true, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on :8097, tls=true, tlsproxy=false, tor=false, websocket=true.
2020-11-13T03:09:29.345Z : info  : listeners  : now listening on 127.0.0.1:6667, tls=false, tlsproxy=false, tor=false, websocket=false.
2020-11-13T03:09:29.345Z : info  : server     : Server running
2020-11-13T03:09:38.386Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:38.439Z : info  : accounts   : client : * : logged into account : kiwi-n30
2020-11-13T03:09:38.442Z : info  : connect    : Client connected [kiwi-n30] [u:~u] [r:https://kiwiirc.com/]
2020-11-13T03:09:40.955Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:40.963Z : info  : connect    : Client connected [kiwi-n28] [u:~u] [r:https://kiwiirc.com/]
2020-11-13T03:09:56.198Z : info  : connect-ip : Client connecting: real IP 192.168.33.1, proxied IP <nil>
2020-11-13T03:09:56.209Z : info  : connect    : Client connected [kiwi-n29] [u:~u] [r:https://kiwiirc.com/]

编辑ircd.yml,设置管理密码,启用一下WebSocket,保存历史消息到MySQL。opers/admin管理密码在认证管理员的时候使用的,可以使用命令oragono genpasswd生成。ircd.db用来保存注册用户,使用的并不是SQLite,而是Golang实现的一个嵌入式的内存Key/Value存储BuntDB,可以持久化到硬盘。

# network configuration
network:
    # name of the network
    name: OragonoTest

# server configuration
server:
    # server name
    name: oragono.test

    # addresses to listen on
    listeners:
        # The standard plaintext port for IRC is 6667. Allowing plaintext over the
        # public Internet poses serious security and privacy issues. Accordingly,
        # we recommend using plaintext only on local (loopback) interfaces:
        "127.0.0.1:6667": # (loopback ipv4, localhost-only)
        "[::1]:6667":     # (loopback ipv6, localhost-only)
        # If you need to serve plaintext on public interfaces, comment out the above
        # two lines and uncomment the line below (which listens on all interfaces):
        # ":6667":
        # Alternately, if you have a TLS certificate issued by a recognized CA,
        # you can configure port 6667 as an STS-only listener that only serves
        # "redirects" to the TLS port, but doesn't allow chat. See the manual
        # for details.

        # The standard SSL/TLS port for IRC is 6697. This will listen on all interfaces:
        ":6697":
            tls:
                cert: fullchain.pem
                key: privkey.pem
                # 'proxy' should typically be false. It's only for Kubernetes-style load
                # balancing that does not terminate TLS, but sends an initial PROXY line
                # in plaintext.
                proxy: false

        # Example of a Unix domain socket for proxying:
        # "/tmp/oragono_sock":

        # Example of a Tor listener: any connection that comes in on this listener will
        # be considered a Tor connection. It is strongly recommended that this listener
        # *not* be on a public interface --- it should be on 127.0.0.0/8 or unix domain:
        # "/hidden_service_sockets/oragono_tor_sock":
        #     tor: true

        # Example of a WebSocket listener:
        ":8097":
            websocket: true
            tls:
                cert: fullchain.pem
                key: privkey.pem
# ircd operators
opers:
    # operator named 'admin'; log in with /OPER admin [password]
    admin:
        # which capabilities this oper has access to
        class: "server-admin"

        # custom whois line
        whois-line: is a server admin

        # custom hostname
        vhost: "staff"

        # normally, operator status is visible to unprivileged users in WHO and WHOIS
        # responses. this can be disabled with 'hidden'. ('hidden' also causes the
        # 'vhost' line above to be ignored.)
        hidden: false

        # modes are modes to auto-set upon opering-up. uncomment this to automatically
        # enable snomasks ("server notification masks" that alert you to server events;
        # see `/quote help snomasks` while opered-up for more information):
        #modes: +is acjknoqtuxv

        # operators can be authenticated either by password (with the /OPER command),
        # or by certificate fingerprint, or both. if a password hash is set, then a
        # password is required to oper up (e.g., /OPER dan mypassword). to generate
        # the hash, use `oragono genpasswd`.
        password: "$2a$04$I2Yhr7UF4p3iyEZcKTPJgukLA9mm00B1wQgicJvGZP/gf0u0tbQQy"
# datastore configuration
datastore:
    # path to the datastore
    path: ircd.db

    # if the database schema requires an upgrade, `autoupgrade` will attempt to
    # perform it automatically on startup. the database will be backed
    # up, and if the upgrade fails, the original database will be restored.
    autoupgrade: true

    # connection information for MySQL (currently only used for persistent history):
    mysql:
        enabled: true
        host: "127.0.0.1"
        port: 3306
        # if socket-path is set, it will be used instead of host:port
        #socket-path: "/var/run/mysqld/mysqld.sock"
        user: "root"
        password: "root"
        history-database: "oragono_history"
        timeout: 30s

IRC服务端设置好了,使用客户端连上去就可以聊天了。IRC协议类似HTTP协议,服务器地址类似irc://irc.freenode.net,但是也可以使用web来连接,前面配置了监听来自8097端口的websocket请求。这里使用web的IRC客户端Kiwi IRC,这是Vue实现的一个web客户端,各个组件之间通过触发/订阅Vue实例状态/事件变化进行通信,支持自定义组件/中间件,这也是Freenode使用的客户端。Kiwiirc主要聚焦连接和消息的处理,比如websocket连接/发送/监听,消息的格式化/通知/声音/状态。它能够连接websocket的IRC服务器,甚至还提供在线生成器,只需要添加websocket url就好了

{
    "windowTitle": "Kiwi IRC with Oragono Test",
    "startupScreen": "welcome",
    "kiwiServer": "http://oragono.test",
    "restricted": false,
    "theme": "Default",
    "themes": [
        { "name": "Default", "url": "static/themes/default" },
        { "name": "Dark", "url": "static/themes/dark" },
        { "name": "Coffee", "url": "static/themes/coffee" },
        { "name": "GrayFox", "url": "static/themes/grayfox" },
        { "name": "Nightswatch", "url": "static/themes/nightswatch" },
        { "name": "Osprey", "url": "static/themes/osprey" },
        { "name": "Radioactive", "url": "static/themes/radioactive" },
        { "name": "Sky", "url": "static/themes/sky" },
        { "name": "Elite", "url": "static/themes/elite" }
    ],
    "startupOptions" : {
        "websocket": "wss://127.0.0.1:8097",
        "channel": "#kiwiirc-default",
        "nick": "kiwi-n?"
    },
    "embedly": {
        "key": ""
    },
    "plugins": [
        { "name": "customise", "url": "static/plugins/customise.html" }
    ]
}

运行登录后的效果


其实IRC是基于TCP连接的文本交互的,也可以使用telnet来模拟

➜  ~ telnet 127.0.0.1 6667
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CAP LS 302
:oragono.test CAP * LS * :account-notify account-tag away-notify batch cap-notify chghost draft/channel-rename draft/chathistory draft/event-playback draft/languages=16,en,~bs,~de,~el,~en-AU,~es,~fi,~fr-FR,~it,~nl,~no,~pl,~pt-BR,~ro,~tr-TR,~zh-CN draft/multiline=max-bytes=4096,max-lines=100 draft/register=before-connect draft/relaymsg=/ draft/resume-0.5 echo-message extended-join invite-notify labeled-response message-tags multi-prefix oragono.io/nope sasl=PLAIN,EXTERNAL server-time setname
:oragono.test CAP * LS :userhost-in-names znc.in/playback znc.in/self-message
NICK kiwi-n26
USER kiwi-n26 0 * https://kiwiirc.com/
CAP REQ :account-notify account-tag away-notify batch cap-notify draft/chathistory extended-join invite-notify message-tags multi-prefix server-time userhost-in-names znc.in/self-message
@time=2020-11-13T01:55:02.201Z :oragono.test CAP * ACK :account-notify account-tag away-notify batch cap-notify draft/chathistory extended-join invite-notify message-tags multi-prefix server-time userhost-in-names znc.in/self-message
CAP END
@time=2020-11-13T01:55:07.714Z :oragono.test 001 kiwi-n26 :Welcome to the Internet Relay Network kiwi-n26
@time=2020-11-13T01:55:07.714Z :oragono.test 002 kiwi-n26 :Your host is oragono.test, running version oragono-2.4.0
@time=2020-11-13T01:55:07.714Z :oragono.test 003 kiwi-n26 :This server was created Fri, 13 Nov 2020 00:35:48 UTC
@time=2020-11-13T01:55:07.714Z :oragono.test 004 kiwi-n26 oragono.test oragono-2.4.0 BERTZios CEIMRUabehiklmnoqstuv Iabehkloqv
@time=2020-11-13T01:55:07.714Z :oragono.test 005 kiwi-n26 AWAYLEN=390 BOT=B CASEMAPPING=ascii CHANLIMIT=#:100 CHANMODES=Ibe,k,l,CEMRUimnstu CHANNELLEN=64 CHANTYPES=# ELIST=U EXCEPTS EXTBAN=,m INVEX KICKLEN=390 MAXLIST=beI:60 :are supported by this server
@time=2020-11-13T01:55:07.714Z :oragono.test 005 kiwi-n26 MAXTARGETS=4 MODES MONITOR=100 NETWORK=OragonoTest NICKLEN=32 PREFIX=(qaohv)~&@%+ STATUSMSG=~&@%+ TARGMAX=NAMES:1,LIST:1,KICK:1,WHOIS:1,USERHOST:10,PRIVMSG:4,TAGMSG:4,NOTICE:4,MONITOR:100 TOPICLEN=390 UTF8MAPPING=rfc8265 WHOX draft/CHATHISTORY=100 :are supported by this server
@time=2020-11-13T01:55:07.714Z :oragono.test 251 kiwi-n26 :There are 0 users and 4 invisible on 1 server(s)
@time=2020-11-13T01:55:07.714Z :oragono.test 252 kiwi-n26 0 :IRC Operators online
@time=2020-11-13T01:55:07.714Z :oragono.test 253 kiwi-n26 0 :unregistered connections
@time=2020-11-13T01:55:07.714Z :oragono.test 254 kiwi-n26 2 :channels formed
@time=2020-11-13T01:55:07.714Z :oragono.test 255 kiwi-n26 :I have 4 clients and 0 servers
@time=2020-11-13T01:55:07.714Z :oragono.test 265 kiwi-n26 4 4 :Current local users 4, max 4
@time=2020-11-13T01:55:07.714Z :oragono.test 266 kiwi-n26 4 4 :Current global users 4, max 4
@time=2020-11-13T01:55:07.714Z :oragono.test 375 kiwi-n26 :- oragono.test Message of the day -
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-       ▄▄▄   ▄▄▄·  ▄▄ •        ▐ ▄
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- ▪     ▀▄ █·▐█ ▀█ ▐█ ▀ ▪▪     •█▌▐█▪
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-  ▄█▀▄ ▐▀▀▄ ▄█▀▀█ ▄█ ▀█▄ ▄█▀▄▪▐█▐▐▌ ▄█▀▄
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- ▐█▌.▐▌▐█•█▌▐█ ▪▐▌▐█▄▪▐█▐█▌ ▐▌██▐█▌▐█▌.▐▌
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-  ▀█▄▀▪.▀  ▀ ▀  ▀ ·▀▀▀▀  ▀█▄▀ ▀▀ █▪ ▀█▄▀▪
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- This is the default Oragono MOTD.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- If motd-formatting is enabled in the config file, you can use the dollarsign character to
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- create special formatting such as bold, italics and color codes.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- For example, here are a few formatted lines (enable motd-formatting to see these in action):
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is bold text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is italics text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is 4red and 2blue text.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is 4,12red text with a light blue background.
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- - this is a normal escaped dollarsign: $
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- And now a few fun colour charts!
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,0 00 0,1 01 0,2 02 0,3 03 1,4 04 0,5 05 0,6 06 1,7 07
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,8 08 1,9 09 0,10 10 1,11 11 0,12 12 1,13 13 1,14 14 1,15 15
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,16 16 0,17 17 0,18 18 0,19 19 0,20 20 0,21 21 0,22 22 0,23 23 0,24 24 0,25 25 0,26 26 0,27 27
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,28 28 0,29 29 0,30 30 0,31 31 0,32 32 0,33 33 0,34 34 0,35 35 0,36 36 0,37 37 0,38 38 0,39 39
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,40 40 0,41 41 0,42 42 0,43 43 0,44 44 0,45 45 0,46 46 0,47 47 0,48 48 0,49 49 0,50 50 0,51 51
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,52 52 0,53 53 1,54 54 1,55 55 1,56 56 1,57 57 1,58 58 0,59 59 0,60 60 0,61 61 0,62 62 0,63 63
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,64 64 1,65 65 1,66 66 1,67 67 1,68 68 1,69 69 1,70 70 1,71 71 0,72 72 0,73 73 0,74 74 0,75 75
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 1,76 76 1,77 77 1,78 78 1,79 79 1,80 80 1,81 81 1,82 82 1,83 83 1,84 84 1,85 85 1,86 86 1,87 87
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- 0,88 88 0,89 89 0,90 90 0,91 91 0,92 92 0,93 93 0,94 94 0,95 95 1,96 96 1,97 97 1,98 98 99,99 99
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :-
@time=2020-11-13T01:55:07.714Z :oragono.test 372 kiwi-n26 :- For more information on using these, see MOTDFORMATTING.md
@time=2020-11-13T01:55:07.714Z :oragono.test 376 kiwi-n26 :End of MOTD command
@time=2020-11-13T01:55:07.715Z :oragono.test 221 kiwi-n26 +Zi
WHO kiwi-n26
@time=2020-11-13T01:55:15.649Z :oragono.test 352 kiwi-n26 * ~u gcjc79gmtbe42.irc oragono.test kiwi-n26 H :0 https://kiwiirc.com/
@time=2020-11-13T01:55:15.649Z :oragono.test 315 kiwi-n26 kiwi-n26!*@* :End of WHO list
JOIN #kiwiirc-default
@msgid=bidkvpja5njjtgchub8u65pxgn;time=2020-11-13T01:55:26.706Z :[email protected] JOIN #kiwiirc-default * https://kiwiirc.com/
@time=2020-11-13T01:55:26.706Z :oragono.test 353 kiwi-n26 = #kiwiirc-default :@[email protected] [email protected] [email protected] [email protected]
@time=2020-11-13T01:55:26.706Z :oragono.test 366 kiwi-n26 #kiwiirc-default :End of NAMES list
JOIN #ops
@msgid=hftqmpu4ieyjybx9bbgf4vpbia;time=2020-11-13T01:55:32.145Z :[email protected] JOIN #ops * https://kiwiirc.com/
@time=2020-11-13T01:55:32.145Z :oragono.test 353 kiwi-n26 = #ops :@[email protected] [email protected] [email protected] [email protected]
@time=2020-11-13T01:55:32.145Z :oragono.test 366 kiwi-n26 #ops :End of NAMES list
PRIVMSG #ops hello
PRIVMSG #ops :kiwi-n28 good morning
PRIVMSG kiwi-n28 =D
PING kiwitime-1605232555315
@time=2020-11-13T01:57:14.150Z :oragono.test PONG oragono.test kiwitime-1605232555315
CHATHISTORY BEFORE kiwi-n28 * 50
@time=2020-11-13T01:57:38.462Z :oragono.test BATCH +1 chathistory kiwi-n28
@msgid=rqdbvmytzppssc5ymp7ffhmkga;time=2020-11-13T01:56:05.705Z;batch=1 :[email protected] PRIVMSG kiwi-n28 :=D
@time=2020-11-13T01:57:38.462Z :oragono.test BATCH -1
@time=2020-11-13T01:59:08.466Z PING kiwi-n26
:[email protected] QUIT :Ping timeout: 2m30s
ERROR :Ping timeout: 2m30s
Connection closed by foreign host.

可以看到这些消息都带有一定的格式,客户端根据特地的格式解析就可以个性化显示来。带有符号@/:前缀的消息都是服务器推送下来的消息,反之都是客户端请求的消息。“CAP LS 302”即协商会话协议版本,随即用命令“NICK ”设置昵称(登录名),和“USER 0 * ”设置该昵称显示的用户名,最后以“CAP END”结束协商,开始会话。服务端推送过来的消息格式为时间戳+服务器+响应码+接受者昵称+消息内容,即“@time=2020-11-13T01:55:07.714Z :oragono.test 001 :Welcome to the Internet Relay Network ”,001代表欢迎信息,具体响应码可以参考这里。客户端发送消息则直接发送,比如使用“JOIN #channel”加入某个频道,服务器会向该频道里的每个人广播对JOIN应的消息“@msgid=hftqmpu4ieyjybx9bbgf4vpbia;time=2020-11-13T01:55:32.145Z :@gcjc79gmtbe42.irc JOIN #channel * ”并向操作者推送频道用户列表“@time=2020-11-13T01:55:32.145Z :oragono.test 353 = #channel :@[email protected] [email protected] [email protected]
”。Twitch则在消息的基础上加入了其他格式化控制,比如

@badge-info=;badges=premium/1;client-nonce=2f70a1efd6f84523d0f180cc44f76c61;color=#B22222;display-name=Khodeine;emotes=58765:9-19;flags=;id=50d8790f-c474-42a8-b873-2d5e8008bd52;mod=0;room-id=30220059;subscriber=0;tmi-sent-ts=1605279854692;turbo=0;user-id=49980352;user-type= :[email protected] PRIVMSG #esl_sc2 :the scvs NotLikeThis

会显示以一定的颜色/图标显示用户名/消息。空闲状态下IRC服务端与客户端需要维持心跳信息,比如“PING kiwitime-1605232555315”和“time=2020-11-13T01:57:14.150Z :oragono.test PONG oragono.test kiwitime-1605232555315”,否则连接会被关闭。一些常用的命令,可以看出IRC交互非常简单,IRC核心组件也不多。使用“@+dratf/typing=active TAGMSG ”显示输入状态,借助CTCP协议甚至可以点对点发送文件

#使用password注册当前nickname
/NS REGISTER <your password>
#注册一个频道
/CS REGISTER #channelname
/CS REGISTER #channe
#要去某人加入频道
/INVITE nickname #channel
#给予某个用户频道管理权限
/mode +o nickname
#认证成为管理员
/OPER admin <admin passowrd>
#在频道发送消息
/PRIVMSG #channe hello
#给频道某人发送消息
/PRIVMSG #channe :<nick> good morning
#给频道某人发送消息
/INVITE <nick> #channe
#使用密码登录
AUTHENTICATE PLAIN
AUTHENTICATE +
AUTHENTICATE a2l3aS1uMzAAa2l3aS1uMzAAMTAyMDMwNDA=

Thelounge是另外一个Vue实现的web的IRC客户端,同样支持websocket。它还用express实现了服务端,支持额外的密码/LDAP登录,即先认证通过才能连接IRC服务器,它本身也可以作为一个服务代理网关,为用户提供永远在线服务/消息存储-如果一个IRC用户不在线是不能够给它发送消息的,这一点也跟XMPP不一样。


Kiwiirc和Thelounge的设计实现良好,可以作为web chat实现的参考,比如在线客服/销售。比较两个web的IRC客户端发现,他们使用的websocket库并不一样。Kiwiirc使用的是sockjs-client,一个类似websocket的JavaScript网络连接库,支持不同浏览器,能够跨域名通信,在浏览器不支持websocket的情况下退化为polling(需要服务端配合)。而Thelounge使用的是socket.io。Socket.IO由Engine.IO发展而来,致力于为web提供实时通信能力。Socket.IO并不是WebSocket的一个实现,只是提供了类似的连接能力,需要服务端配合。两个连接库在浏览器支持websocket的情况下都可以直接与websocket服务端通信(不需要对应的库)。Kiwiirc还提供了irc-framework开发库,可以快速搭建一个IRC客户端,比如IRC Bot。至今仍然存在着许多IRC聊天机器人,为用户提供方便/服务器报警。Twitch一个用户加入频道时甚至能收到广告!

Websocket提供了与其他socket一样的能力,能够持续连接和双向通讯,甚至可以服用80/443端口(nginx支持101协议升级)。它不止可以使用在web上,也可以使用在非web上面,因为能够穿透防火墙,对树莓派的远程链接支持。传统的web应用都是一应一答,这节省的服务器/客户端的资源能够不需要持续请求的话。而对于有些应用而言,比如消息通知,希望是能够持续从服务器推送的,websocket增强了它的能力,在这之前都是long polling(所以才有那些包装库)。
Http协议在tcp基础上实现,请求头里面带有host、cookie,url里面带有路径等,这样后端能够识别、路由、分发请求。而tcp并没有这些,websocket必须自己实现类似的路由分发/调用,可以在连接建立初期,鉴别授权(可以在url上面带参数token/jwt/security key),对于后续消息的应答可以在url上面指定,也可以在事件上面指定。这样实现出来似乎与http差不多,仍然有event对应route,但却可以持续交互。
再观察IRC协议也不过是对各个命令的应答,其实也跟经典web MVC差不多-使用URL映射对应的controller/action。IRC每个会话都是一个队列,join #channel命令即订阅消息,part #channel命令则是取消订阅。再看Rabbitmq消息队列,实现了exchange,route,queue,channel,topic等,然后是持续的消费/推送,与前面所诉差不多,这样子看IRC也是个消息服务器。IRC实现简单,通过扩展支持的命令,也能够与内部/外部系统结合,快速搭建一个即时消息平台。
再看邮件服务,其实也是消息服务,也有一对一/群发消息/历史消息,甚至能够给不认识(不在线)的人发送邮件!而邮件服务至今流行,在许多平台,不论服务器还是手机都自带,不需要额外安装软件。Slack的产品宣传即对比了自己相对邮件服务的优势。其实手机默认带的电话和短信功能是基于硬件模块的在线功能。
相比Ejabberd/Openfire,IRC服务器的部署简单许多,甚至可以部署在树莓派上!

参考连接:
IRC Bot
基于 RabbitMQ 的实时消息推送

即时通信XMPP服务器ejabberd/Openfire

即时通信的需求,不仅在公共互联网上,比如微信、QQ,也在企业的内部网络,特别是与内部系统的整合,比如OA、客服系统。XMPP,也叫扩展消息与存在协议,是一种以XML为基础的开放式即时通信协议。XMPP不但能够用来做消息通信(单聊/群聊/订阅),也可以做语音通信。XMPP甚至支持多个服务器之间互相连接,互相通信,早期Gtalk就能够与其他XMPP服务器通信。采用XMPP来开发的好处是,已经有大量的开源服务器(ejabberd、Openfire)和客户端(Spark/Smack/converse.js/XMPPFramework)实现,能够快速搭建。
Ejabberd则是ProcessOne出品的,基于Erlang开发的XMPP/MQTT/SIP服务器,内置了WebSocket/文件上传支持,提供更加丰富的REST API。在CentOS64下面安装Ejabberd,会安装到/etc/init.d/ejabberd,直接service start ejabberd就好了,可以使用docker快速启用

[[email protected] tmp]# wget https://static.process-one.net/ejabberd/downloads/20.04/ejabberd-20.04-0.x86_64.rpm
--2020-10-29 08:26:16--  https://static.process-one.net/ejabberd/downloads/20.04/ejabberd-20.04-0.x86_64.rpm
Resolving static.process-one.net... 13.226.36.69, 13.226.36.105, 13.226.36.34, ...
Connecting to static.process-one.net|13.226.36.69|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18713774 (18M) [application/x-rpm]
Saving to: “ejabberd-20.04-0.x86_64.rpm”

100%[====================================================================================================================================================================================================================================>] 18,713,774  2.84M/s   in 9.1s    

2020-10-29 08:26:27 (1.96 MB/s) - “ejabberd-20.04-0.x86_64.rpm” saved [18713774/18713774]

[[email protected] tmp]# yum localinstall ejabberd-20.04-0.x86_64.rpm
Loaded plugins: fastestmirror
Setting up Local Package Process
Examining ejabberd-20.04-0.x86_64.rpm: ejabberd-20.04-0.x86_64
Marking ejabberd-20.04-0.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
 * base: mirrors.163.com
 * epel: mirror.math.princeton.edu
 * extras: mirrors.cn99.com
 * updates: mirrors.cn99.com
 * webtatic: uk.repo.webtatic.com
phalcon_stable/signature                                                                                                                                                                                                                               |  819 B     00:00     
phalcon_stable/signature                                                                                                                                                                                                                               |  951 B     00:00 ... 
phalcon_stable-source/signature                                                                                                                                                                                                                        |  819 B     00:00     
phalcon_stable-source/signature                                                                                                                                                                                                                        |  951 B     00:00 ... 
Resolving Dependencies
--> Running transaction check
---> Package ejabberd.x86_64 0:20.04-0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                       Arch                                                        Version                                                        Repository                                                                     Size
==============================================================================================================================================================================================================================================================================
Installing:
 ejabberd                                                      x86_64                                                      20.04-0                                                        /ejabberd-20.04-0.x86_64                                                       29 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install       1 Package(s)

Total size: 29 M
Installed size: 29 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : ejabberd-20.04-0.x86_64                                                                                                                                                                                                                                    1/1 
  Verifying  : ejabberd-20.04-0.x86_64                                                                                                                                                                                                                                    1/1 

Installed:
  ejabberd.x86_64 0:20.04-0                                                                                                                                                                                                                                                   

Complete!
[[email protected] tmp]# ls  /etc/init.d/ejabberd
/etc/init.d/ejabberd
[[email protected] tmp]# ls -la /opt/ejabberd
total 24
drwxr-xr-x  5 ejabberd ejabberd 4096 Oct 29 08:27 .
drwxr-xr-x. 8 root     root     4096 Oct 29 08:27 ..
drwxr-xr-x  2 ejabberd ejabberd 4096 Oct 29 08:27 conf
drwxr-xr-x  3 ejabberd ejabberd 4096 Oct 29 08:27 database
-r--------  1 ejabberd ejabberd   20 Oct 29 00:00 .erlang.cookie
drwxr-xr-x  2 ejabberd ejabberd 4096 Oct 29 08:27 logs
[[email protected] bin]# service ejabberd start
Starting ejabberd...
done.
[[email protected] sql]# ps aux | grep ejabberd
ejabberd 10097  0.2  7.9 1817344 48144 ?       Sl   08:51   0:04 /opt/ejabberd-20.04/bin/beam.smp -K true -P 250000 -- -root /opt/ejabberd-20.04 -progname /opt/ejabberd-20.04/bin/erl -- -home /opt/ejabberd -- -sname [email protected] -smp enable -mnesia dir "/opt/ejabberd/database/[email protected]" -ejabberd log_rate_limit 100 log_rotate_size 10485760 log_rotate_count 1 log_rotate_date "" -s ejabberd -noshell -noinput
ejabberd 10105  0.0  0.0   4060   496 ?        Ss   08:51   0:00 erl_child_setup 1024
ejabberd 10186  0.0  0.0   4052   572 ?        Ss   08:51   0:00 /opt/ejabberd-20.04/lib/os_mon-2.4.7/priv/bin/memsup
root     11215  0.0  0.1 103324   892 pts/4    S+   09:16   0:00 grep ejabberd
[[email protected] bin]# netstat -apn | grep beam
tcp        0      0 127.0.0.1:7777              0.0.0.0:*                   LISTEN      10097/beam.smp      
tcp        0      0 0.0.0.0:37617               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:15672               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:55672               0.0.0.0:*                   LISTEN      1599/beam           
tcp        0      0 0.0.0.0:43674               0.0.0.0:*                   LISTEN      10097/beam.smp      
tcp        0      0 127.0.0.1:44055             127.0.0.1:4369              ESTABLISHED 1599/beam           
tcp        0      0 127.0.0.1:35415             127.0.0.1:4369              ESTABLISHED 10097/beam.smp      
tcp        0      0 :::5280                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5443                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5222                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::5672                     :::*                        LISTEN      1599/beam           
tcp        0      0 :::5269                     :::*                        LISTEN      10097/beam.smp      
tcp        0      0 :::1883                     :::*                        LISTEN      10097/beam.smp      
unix  3      [ ]         STREAM     CONNECTED     1637533 10097/beam.smp   

Ejabberd自带Erlang的数据库服务Mnesia,但有一些限制建议使用第三方的比如MySQL,这样也方便第三方交互(比如认证Token)。初始化的SQL脚本在/opt/ejabberd-20.04/lib/ejabberd-20.04/priv/sql,如果只是支持单个domain的话使用mysql.sql就可以了,多domain支持使用mysql.new.sql

[[email protected] sql]# ls
lite.new.sql  lite.sql  mssql.sql  mysql.new.sql  mysql.sql  pg.new.sql  pg.sql
[[email protected] sql]# pwd
/opt/ejabberd-20.04/lib/ejabberd-20.04/priv/sql

更改为使用MySQL存储和认证

[[email protected] bin]# cat /opt/ejabberd/conf/ejabberd.yml
###
###'           ejabberd configuration file
###
### The parameters used in this configuration file are explained at
###
###       https://docs.ejabberd.im/admin/configuration
###
### The configuration file is written in YAML.
### *******************************************************
### *******           !!! WARNING !!!               *******
### *******     YAML IS INDENTATION SENSITIVE       *******
### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******
### *******************************************************
### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.
###

hosts:
  - "vagrant-centos64"

loglevel: 4
log_rotate_size: 10485760
log_rotate_date: ""
log_rotate_count: 1
log_rate_limit: 100

certfiles:
  - "/opt/ejabberd/conf/server.pem"
##  - "/etc/letsencrypt/live/localhost/fullchain.pem"
##  - "/etc/letsencrypt/live/localhost/privkey.pem"

ca_file: "/opt/ejabberd/conf/cacert.pem"

listen:
  -
    port: 5222
    ip: "::"
    module: ejabberd_c2s
    max_stanza_size: 262144
    shaper: c2s_shaper
    access: c2s
    starttls_required: true
  -
    port: 5269
    ip: "::"
    module: ejabberd_s2s_in
    max_stanza_size: 524288
  -
    port: 5443
    ip: "::"
    module: ejabberd_http
    tls: true
    request_handlers:
      "/admin": ejabberd_web_admin
      "/api": mod_http_api
      "/bosh": mod_bosh
      "/captcha": ejabberd_captcha
      "/upload": mod_http_upload
      "/ws": ejabberd_http_ws
      "/oauth": ejabberd_oauth
  -
    port: 5280
    ip: "::"
    module: ejabberd_http
    request_handlers:
      "/admin": ejabberd_web_admin
  -
    port: 1883
    ip: "::"
    module: mod_mqtt
    backlog: 1000

s2s_use_starttls: optional

acl:
  local:
    user_regexp: ""
  loopback:
    ip:
      - 127.0.0.0/8
      - ::1/128
      - ::FFFF:127.0.0.1/128
  admin:
    user:
      - "[email protected]"

access_rules:
  local:
    allow: local
  c2s:
    deny: blocked
    allow: all
  announce:
    allow: admin
  configure:
    allow: admin
  muc_create:
    allow: local
  pubsub_createnode:
    allow: local
  trusted_network:
    allow: loopback

api_permissions:
  "console commands":
    from:
      - ejabberd_ctl
      - mod_http_api
    who: all
    what: "*"
  "admin access":
    who:
      access:
        allow:
          acl: loopback
          acl: admin
      oauth:
        scope: "ejabberd:admin"
        access:
          allow:
            acl: loopback
            acl: admin
    what:
      - "*"
      - "!stop"
      - "!start"
  "public commands":
    who:
      ip: 127.0.0.1/8
    what:
      - status
      - connected_users_number

shaper:
  normal: 1000
  fast: 50000

shaper_rules:
  max_user_sessions: 10
  max_user_offline_messages:
    5000: admin
    100: all
  c2s_shaper:
    none: admin
    normal: all
  s2s_shaper: fast

max_fsm_queue: 10000

acme:
   contact: "mailto:[email protected]"
   ca_url: "https://acme-v01.api.letsencrypt.org"

modules:
  mod_adhoc: {}
  mod_admin_extra: {}
  mod_announce:
    access: announce
  mod_avatar: {}
  mod_blocking: {}
  mod_bosh: {}
  mod_caps: {}
  mod_carboncopy: {}
  mod_client_state: {}
  mod_configure: {}
  mod_disco: {}
  mod_fail2ban: {}
  mod_http_api: {}
  mod_http_upload:
    put_url: https://@[email protected]:5443/upload
  mod_last: {}
  mod_mam:
    ## Mnesia is limited to 2GB, better to use an SQL backend
    ## For small servers SQLite is a good fit and is very easy
    ## to configure. Uncomment this when you have SQL configured:
    ## db_type: sql
    assume_mam_usage: true
    default: never
  mod_mqtt: {}
  mod_muc:
    access:
      - allow
    access_admin:
      - allow: admin
    access_create: muc_create
    access_persistent: muc_create
    access_mam:
      - allow
    default_room_options:
      allow_subscription: true  # enable MucSub
      mam: false
  mod_muc_admin: {}
  mod_offline:
    access_max_user_messages: max_user_offline_messages
  mod_ping: {}
  mod_privacy: {}
  mod_private: {}
  mod_proxy65:
    access: local
    max_connections: 5
  mod_pubsub:
    access_createnode: pubsub_createnode
    plugins:
      - flat
      - pep
    force_node_config:
      ## Avoid buggy clients to make their bookmarks public
      storage:bookmarks:
        access_model: whitelist
  mod_push: {}
  mod_push_keepalive: {}
  mod_register:
    ## Only accept registration requests from the "trusted"
    ## network (see access_rules section above).
    ## Think twice before enabling registration from any
    ## address. See the Jabber SPAM Manifesto for details:
    ## https://github.com/ge0rg/jabber-spam-fighting-manifesto
    ip_access: trusted_network
  mod_roster:
    versioning: true
  mod_s2s_dialback: {}
  mod_shared_roster: {}
  mod_stream_mgmt:
    resend_on_timeout: if_offline
  mod_vcard: {}
  mod_vcard_xupdate: {}
  mod_version:
    show_os: false

auth_method: sql


sql_type: mysql
sql_server: "localhost"
sql_database: "ejabberd"
sql_username: "ejabberd"
sql_password: "ejabberd"

default_db: sql
## If you want to specify the port:
#sql_port: 3306
### Local Variables:
### mode: yaml
### End:
### vim: set filetype=yaml tabstop=8

试着注册一下管理员

[[email protected] bin]# ./ejabberdctl register admin1 localhost admin
Error: cannot_register

失败了,这是因为hostname的关系,在ejabberd.yml配置的host并不是localhost,改一下注册的domain就好了,具体可以参考这里这里

[[email protected] bin]# ./ejabberdctl status
The node [email protected] is started with status: started
[[email protected] logs]# hostname -s
vagrant-centos64
[[email protected] bin]# ./ejabberdctl register admin vagrant-centos64 admin
User [email protected] successfully registered

访问http://127.0.0.1:5280/admin就可以进入到管理界面了。管理界面比较简单,只能管理用户/消息/聊天室,完整的xmpp协议服务器是支持的,客户端对应实现就好了。


使用conversejs快速搭建一个web客户端来验证一下,创建chat.html

<html>
    <head>
        <title>chat</title>
        <link rel="stylesheet" type="text/css" media="screen" href="https://cdn.conversejs.org/dist/converse.min.css">
        <script src="https://cdn.conversejs.org/dist/converse.min.js" charset="utf-8"></script>
    </head>
    <body>
        <script>
            converse.initialize({
                //bosh_service_url: 'http://chat.vagrant-centos64.com/bosh',
                websocket_url: 'wss:chat.vagrant-centos64.com/ws',
                //show_controlbox_by_default: true,
                view_mode: 'fullscreen'
            });
        </script>
    </body>
</html>

打开浏览器,访问对应的url登录就可以看到了聊天界面了。这里使用的域名是经过nginx代理转发过的了,配置可以参考API 网关 Kong,如果使用nginx-proxy-servier需要打开Websockets Support,以便进行协议升级101 Switching Protocols。



conversejs配置里面有两个url,一个是BOSH(Bidirectional-streams Over Synchronous HTTP)的,即HTTP长连接,以便通服务器即时交互,获取消息;另一个是WebSocket,基于TCP的,客户端/服务器双向通信。BOHS需要浏览器定时发起一个请求直致服务器返回消息,而WebSocket可以像其他TCP那样双向通信,灵活的多。如果服务端不支持Websocket协议升级或者连接失败conversejs会自动切换使用BOSH通信。如果使用https://www.websocket.org/echo.html来测试连接,需要遵循XMPP协议要求,先认证,否则连接会被关闭。conversejs的认证是在Websocket里面的做的,并不基于cookie之类的,所以一个浏览器打开多个聊天窗口登录不同的账户也是可以的



Ejabberd支持REST API来做交互比如查看在线用户、发送消息

➜  chat-example git:(master) ✗ curl -k https://192.168.33.14:5443/api/connected_users
["[email protected]/converse.js-16142875"]                                                                                                                                                                                                                              ➜  chat-example git:(master) ✗ curl -k https://192.168.33.14:5443/api/send_message -X POST -d '{"type":"headline","from":"[email protected]","to":"[email protected]","subject":"Restart","body":"In 5 minutes"}'

这些API将大大增加ejabberd与第三方软件的交互。虽然ejabberd支持OAuth认证,但那是以ejabberd为账户中心的认证,方便其他系统调用ejabberd功能。通常即时通信只是内部系统的一部分,账户中心部署在其他地方,所以需要ejabberd支持外部的认证。前边已经配置ejabberd为数据库认证,还可以配置为使用LDAP认证。如果不满足,还可以配置为外部脚本认证或者使用第三方开发的HTTP认证。对于简单的内部交互,可以将认证服务的token刷新到ejabberd的数据库即可。
Openfire则是ignite realtime出品的Java实现的XMPP服务器,同时提供Java客户端Spark、Java开发库Smack,还提供Chrome扩展Pade。Openfire提供基于web的管理界面,支持LDAP登录及数据库存储。有一些功能Openfire并不直接支持,比如API,而是以扩展的形式支持,包括用户管理/分组管理/聊天室管理/消息广播/邮件监听/WebSocket/Meeting/Sip等等。

用户管理

配置用户分组,可以配置部门之类的,会出现在用户的个人分组里面(Spark)

在线会话管理,可以踢人


创建聊天室之前需要创建对应的service,默认的service叫conference。
Openfire的API并不支持OAuth/SSO,简单的1V1消息发送,聊天室消息订阅等等,需要自己基于Java扩展。
Ejabberd是采用Erlang开发的,一如消息队列服务器RabbitMQ,具有极高吞吐能力,提供REST API/XMl RPC,方便交互;Openfire则易于管理和扩展,采用哪个软件进行开发需要结合企业实际进行考量。
XMPP服务器主要提供消息聊天,对于语音服务可以使用SIP服务器实现,结合客户端sip.js,比如ctxSip

参考链接:
TCP UDP探索
SIP(会话发起协议)

API 网关 Kong

从前开发一个互联网服务程序,大概可以在一台机器上完成:数据库、应用都在一起。随着业务发展壮大,会把数据库独立出来,以便扩展拆分。然后再把一部分公用业务独立出来扩展,譬如文件存储、缓存等。接着业务也才拆分,比如会员、商品。微服务大行其道,各个团队维护着许多服务、API。这么多服务,前端业务逻辑该怎么接入呢?
如今单一的前端UI也可能是多个团队共同开发的结果。当你在本地开发一个小功能时,甚至会牵扯到多个前端UI/后端API。一个前端页面除了加载自己的资源外,还加载V2/V3的API(为什么会同时存在V1/V2/V3的API),甚至嵌入了另外一个页面(React看起来也不错),该怎么调试这种混合开发呢?cookie都传递不过去,好吧,使用JWT代替,再设置一下跨源资源共享(CORS)。。。但是那些旧的应用怎么办?
当我们启动一个Node.js应用时npm run start,默认监听3000端口。当然我们也可以让它监听80端口。但是当我们多开几个应用时,只好让它们都监听不同端口了,怎么样才能统一端口监听呢?当然Node.js可以通过诸如node-http-proxy来转发这些请求,但是代理转发跟应用业务无关吧?如果是其他编程语言呢?都重复这些代理配置/开发吗?Don’t Repeat Yourself。
Faas也日渐流行,比如AWS Lambda, 强调仅仅专注某个功能,根据事件驱动进行计算,那么每个服务也要开发一整套路由分发/认证/日志吗?
这些问题可以使用反向代理来解决,提供统一的服务入口,对前端/客户端隐藏背后的细节,最简单的当然是Nginx。Nginx监听来自某个端口(比如80)的请求,然后根据不同的来源/端口/域名/url分发给不同的后端服务器。这看起来跟云服务厂商各自开发的负载均衡器差不多,比如AWS ELB
在本地开发的时候,我们当然不会使用ELB来解决。直接使用Nginx当然也没啥问题,编辑一下配置文件,重启应用。但是如果有个UI就更好了,如果还有API就非常好了-这样就能方便的在线注册/更改路由而不需要重启服务器,在这个快速发展、弹性开发的年代更需要这个能力。
Kong是一个基于OpenResty的网关服务器,可以进行路由(转发/负载),插件(日志/认证/监控)管理,并提供RESTful API。而OpenResty 是一个基于Nginx与Lua的高性能Web平台,使用Lua来构建动态网关。听起来像是在Nginx上面编程,这跟在Apache上面使用PHP模块进行编程有什么区别?最大的区别在于,这里的编程对象是Nginx(或者公共模块),扩展Nginx能力,比如负载均衡/日志/认证/监控,而不是输出web页面/业务逻辑。这些东西抽出来以后,就不需要每个模块再重复开发了,比如认证/安全。
在本地可以使用docker来运行Kong服务,麻烦在于数据库迁移工作。通常部署Kong需要几个步骤,比如初始化数据库,迁移升级等等。网上的配置大都过时了,这里可以使用官方提供docker-compose.yml来做数据库工作,并且使用Konga作为管理UI

➜  kong ls
POSTGRES_PASSWORD         data                      docker-compose.yml
➜  kong ls data
postgresql
➜  kong cat POSTGRES_PASSWORD
kong
➜  kong cat docker-compose.yml
version: '3.7'

volumes:
  kong_data: {}

networks:
  kong-net:
    external: false

services:
  kong-migrations:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations bootstrap
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  kong-migrations-up:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    command: kong migrations up && kong migrations finish
    depends_on:
      - db
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  kong:
    image: "${KONG_DOCKER_TAG:-kong:latest}"
    user: "${KONG_USER:-kong}"
    depends_on:
      - db
    environment:
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: '0.0.0.0:8001'
      KONG_CASSANDRA_CONTACT_POINTS: db
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: ${KONG_PG_DATABASE:-kong}
      KONG_PG_HOST: db
      KONG_PG_USER: ${KONG_PG_USER:-kong}
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_PG_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    networks:
      - kong-net
    ports:
      - "8000:8000/tcp"
      - "127.0.0.1:8001:8001/tcp"
      - "8443:8443/tcp"
      - "127.0.0.1:8444:8444/tcp"
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure

  db:
    image: postgres:9.5
    environment:
      POSTGRES_DB: ${KONG_PG_DATABASE:-kong}
      POSTGRES_USER: ${KONG_PG_USER:-kong}
      POSTGRES_PASSWORD_FILE: /run/secrets/kong_postgres_password
    secrets:
      - kong_postgres_password
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "${KONG_PG_USER:-kong}"]
      interval: 30s
      timeout: 30s
      retries: 3
    restart: on-failure
    deploy:
      restart_policy:
        condition: on-failure
    stdin_open: true
    tty: true
    networks:
      - kong-net
    volumes:
      - /Users/xxxx/docker/kong/data/postgresql:/var/lib/postgresql/data

  konga:
    image: pantsel/konga
    environment:
      TOKEN_SECRET: channing.token
      DB_ADAPTER: postgres
      DB_HOST: db
      DB_USER: ${KONG_PG_USER:-kong}
      DB_PASSWORD: kong
      DB_DATABASE: ${KONG_PG_DATABASE:-kong}
    ports:
     - 1337:1337
    networks:
     - kong-net

    depends_on:
      - db

secrets:
  kong_postgres_password:
    file: ./POSTGRES_PASSWORD

运行docker-composer up就可以看到

➜  kong docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                             PORTS                                                                                                NAMES
20ebe7885c0d        kong:latest                       "/docker-entrypoint.…"   2 hours ago         Up 11 seconds (healthy)            0.0.0.0:8000->8000/tcp, 127.0.0.1:8001->8001/tcp, 0.0.0.0:8443->8443/tcp, 127.0.0.1:8444->8444/tcp   kong_kong_1
4a7a39c863ae        pantsel/konga                     "/app/start.sh"          2 hours ago         Up 11 seconds                      0.0.0.0:1337->1337/tcp                                                                               kong_konga_1
aa732758fc51        postgres:9.5                      "docker-entrypoint.s…"   2 hours ago         Up 11 seconds (health: starting)   5432/tcp                                                                                             kong_db_1

访问http://127.0.0.1:1337/即可以进入Konga 管理界面了。8000/8443端口是需要监听转发的端口,8001/8444则是Kong RESTFUl管理API的端口。这里也可以把8000/8443改为80/443,这样访问的时候就可以直接使用域名/localhost而不必加端口了。
注册登录进去后首先要添加Kong RESTFUl管理API的地址,这里使用的是docker环境,IP是动态分配的,所以使用链接名就可以了

连接上Kong API后可以看到dashboard列出了支持的插件

Kong里面的管理对象是service,路由转发/插件都是围绕service展开的,添加一个service


在它上面添加一个路由,这里我们在服务只监听以/api开头的url,并转发到后端服务器去


注意这里在route里面的path里面也需要添加/api,否则转发的时候会出错(多拼api)。最简单就是service监听不指定path,在route里指定。一个service下面可以有多个转发路由,每个可以独立管理、设置超时等。
还可以为每个service绑定不同的插件进行处理,比如认证/安全/日志等等,这样就不用在不同的系统里面重复开发这些功能,使得各个团队更加专注也本业务开发


有效插件还可以进行流量限制、请求头/响应头修改等等。这些插件的功能需要基于consumer来开发管理,功能可以非常强大,详细可以参考文档
Kong还有一项功能Upstream配置,与Nginx的ngx_http_upstream_module差不多,可以作为负载均衡、流量分发控制使用。可以用命令行测试基于hostname的路由,

➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev1.example.com'
HTTP/1.1 404 Not Found
Date: Tue, 27 Oct 2020 08:41:58 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 48
X-Kong-Response-Latency: 1
Server: kong/2.1.4

{"message":"no Route matched with those values"}                                                                                                                                                                                                                             ➜  kong curl -i -X GET \
  --url http://localhost:8000/ \
  --header 'Host: dev.example.com'

HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Tue, 27 Oct 2020 08:42:21 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.6.40
Set-Cookie: PHPSESSID=0uc3aoc735j21sk3ni5s72vjc1; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE
Access-Control-Allow-Headers: X-CSRF-Token,Authorization,X-Accept-Charset,X-Accept,Content-Type
X-Kong-Upstream-Latency: 8397
X-Kong-Proxy-Latency: 0
Via: kong/2.1.4

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
...

作为一个API网关,Kong能做的很多。如果只是在本地简单的做反向代理,可以使用nginx-proxy-manager,这也是一个基于Nginx开发的网关,根据WEB UI动态生成Nginx配置文件,然后执行/usr/sbin/nginx -s reload生效。有些参数不能通过UI配置(比如超时设置),可以直接写Nginx配置,upstream则支持直接转发tcp请求,支持websockets代理转发,甚至集成了Let’s encrypt自动申请SSL证书(需要验证域名)


本地同样可以使用docker跑起来,默认监听80/443端口,81端口即管理界面

➜  nginx-proxy-manager ls -lah
total 16
drwxr-xr-x   6 xxxx domain users   192B Sep 29 10:24 .
drwxr-xr-x  15 xxxx domain users   480B Sep 29 09:15 ..
-rw-r--r--   1 xxxx domain users   2.3K Sep 29 09:42 config.json
drwxr-xr-x   8 xxxx domain users   256B Sep 29 09:31 data
-rw-r--r--   1 xxxx domain users   740B Sep 29 10:24 docker-compose.yml
drwxr-xr-x   3 xxxx domain users    96B Oct 27 16:38 letsencrypt
➜  nginx-proxy-manager cat docker-compose.yml
version: "3"
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: always
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
      # TCP Forward Example:
      - '8022:8022'
    volumes:
      # Make sure this config.json file exists as per instructions above:
      - ./config.json:/app/config/production.json
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: jc21/mariadb-aria
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql
➜  nginx-proxy-manager docker ps
CONTAINER ID        IMAGE                             COMMAND             CREATED             STATUS                PORTS                                                                    NAMES
3dd58e9cff1f        jc21/nginx-proxy-manager:latest   "/init"             4 weeks ago         Up 7 days (healthy)   0.0.0.0:80-81->80-81/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8022->8022/tcp   nginx-proxy-manager_app_1
a033151b28ed        jc21/mariadb-aria                 "/scripts/run.sh"   4 weeks ago         Up 7 days             3306/tcp                                                                 nginx-proxy-manager_db_1

nginx-proxy-manager会将配置写在data目录下面,可以直接编辑这些文件,Nginx reload之后便会生效,可以看到这些文件,具体的加载规则参考文档

➜  nginx-proxy-manager ls data/nginx
dead_host        default_host     default_www      dummycert.pem    dummykey.pem     proxy_host       redirection_host stream           temp
➜  nginx-proxy-manager cat data/nginx/proxy_host/1.conf
# ------------------------------------------------------------
# dev.example.com
# ------------------------------------------------------------
server {
  set $forward_scheme http;
  set $server         "192.168.33.14";
  set $port           80;

  listen 80;
listen [::]:80;

  server_name dev.example.com;

  access_log /data/logs/proxy_host-1.log proxy;

proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

  location /Login {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://login.example.com:443;

  }

  location /api/v2 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.14:9070;

  }

  location /api/v3 {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;

  }

  location /Chat {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       https://dev.ops.example.com:443;

  }

  location /Catalog {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location /static {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location /css {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_pass       http://192.168.33.1:3000;

  }

  location / {

    # Proxy!
    include conf.d/include/proxy.conf;
  }

  # Custom
  include /data/nginx/custom/server_proxy[.]conf;
}

➜  nginx-proxy-manager cat data/nginx/stream/1.conf
# ------------------------------------------------------------
# 8022 TCP: 1 UDP: 0
# ------------------------------------------------------------
server {
  listen 8022;
listen [::]:8022;

  proxy_pass 192.168.33.14:22;

  # Custom
  include /data/nginx/custom/server_stream[.]conf;
  include /data/nginx/custom/server_stream_tcp[.]conf;
}

在docker-compose.yml里面设置一下upstream监听转发的端口,就可以通过8022端口访问192.168.33.14的22端口了

➜  nginx-proxy-manager telnet 127.0.0.1 8022
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.3
^C^C^C^C^C^C^CConnection closed by foreign host.

不论是Kong还是nginx-proxy-manager均有提供API,极大的增强服务网关的可编程性,为动态上线/弹性扩展/自动化运维提供了便利。

参考链接:
从IaaS到FaaS—— Serverless架构的前世今生
聊一聊微服务网关 Kong
KONG网关 — KongA管理UI使用
云原生架构下的 API 网关实践: Kong (二)
微服务 API 网关 -Kong 详解
Creating a web API with Lua using Nginx OpenResty
Nginx基于TCP/UDP端口的四层负载均衡(stream模块)配置梳理
Nginx支持TCP代理和负载均衡-stream模块
聊聊 API Gateway 和 Netflix Zuul
Envoy 是什么?

短网址生成

有时候我们想把一个URL分享给别人,如果URL参数太多,或者太长了,生成的二维码就不好扫描,不方便分享。于是便有了缩短网址的需求,比如微博的t.cn,微信的url.cn,Twitter的t.co。甚至有Bitly.com这样的第三方短网址服务商,提供url访问统计/营销。
怎么样将一个长URL映射成短的URL?如果只是单纯的字符串压缩,很难达到,没办法预计URL有多长。将URL MD5后看起来可行,但是有点长,我们不需要那么多位。最简单的就是在数据库里面存储,一个ID对应一个URL。事实上世界上的URL仍然没有超过2的64次方,因此我们采用自增数字ID就可以了,比如数据库自增量或分布式ID生成器。使用自增数字ID的好处是它可以作为数据库主键,通过它来查找对应的URL就很快来。但是随着ID的增长,它也会变得很长,我们还可以将它缩短:将一个10进制的数字转换为62进制的字符串,最多只要11个字符就可以了。


如果能预计需要生成的网址数量,还可以再缩短,譬如世界上的网址数量大概50亿,62的7次方远大于这个数,因此7个字符足够了。
当访问短网址时,需要301/302重定向到原本的长网址。如果是301永久重定向,则搜索引擎会直接显示重定向后的地址。
PHP函数base_convert可以在2和36进制之间转换,对于62进制就不行,这里分享一个Base62 的转换类,可以将10进制数字转换位62进制

<?php

declare(strict_types=1);

namespace Dig\Conversion;


class Base
{
    const BASE_MIN = 2;
    const BASE_MAX = 62;

    public function __construct(int $base, string $aplphabet)
    {
        if (($base < self::BASE_MIN) || ($base > self::BASE_MAX)) {
            throw new \Exception('base convert only require '.self::BASE_MIN.' <= base <= '.self::BASE_MAX);
        }
        if (empty($aplphabet)) {
            throw new \Exception('cannot have empty aplphabet');
        }
        if ($base > \strlen($aplphabet)) {
            throw new \Exception('base convert only require base <= aplphabet length ');
        }
        $this->base = $base;
        $this->alphabet = $aplphabet;
    }

    public function encode(int $number): string
    {
        $number = (string) $number;
        $base = (string) $this->base;
        $reminder = \bcmod($number, $base);
        $quotient = \bcdiv($number, $base);
        $result = $this->alphabet[$reminder];

        while ($quotient) {
            $reminder = \bcmod($quotient, $base);
            $quotient = \bcdiv($quotient, $base);
            $result = $this->alphabet[$reminder] . $result;
        }
        return $result;
    }
    public function decode(string $number): int
    {
        $base = (string) $this->base;
        $length = \strlen($number);
        $result = (string) \strpos($this->alphabet, $number[0]);

        for ($i = 1; $i < $length; $i++) {
            $result = \bcadd(\bcmul($base, $result), (string) \strpos($this->alphabet, $number[$i]));
        }
        return (int)$result;
    }
}

class Base62 extends Base
{
    public function __construct()
    {
        parent::__construct(62, '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ');
    }
}

这里使用a-z表示10-36,A-Z表示37-62。注意这里需要用到BCMath扩展,否则对于超过2的32次方的大数字数不准确。也可以使用GMP扩展的gmp_strval函数直接转换。注意gmp使用A-Z表示10-36,a-z表示37-62。
测试一下:

include __DIR__.'/../vendor/autoload.php';

use Dig\Conversion\Base62;


$base62 = new Base62();
$id = 56800235584;

for ($i = 1; $i < 100; $i++) {
    $encode = $base62->encode($id);
    $decode = $base62->decode($encode);
    echo $id.'->'.$encode.'->'.$decode.PHP_EOL;
    $id += $i;
}

这里的起始ID可以自定义或者在分布式ID生成器里面更改CUSTOM_EPOCH生成合适的值。
由于这个ID本身就是唯一,映射为固定长度的字符,也可以用来做唯一标识,比如匿名用户的ID,Ngrok三级域名等等。
这个类可以扩展成支持任意2-64进制的转换,更改映射的字符串就行了。
注意这里的Base62转换于PHP函数base64_encode/base64_decode没有任何关系,即使在这个类的基础上增加2个字符支持Base64编码也不一样。Base64是使用64个可打印字符对二进制数据的编码解码,它的编码表与这里定义的不一样。

参考链接:
如何设计一个短网址服务(TinyURL)?
converting a number base 10 to base 62 (a-zA-Z0-9)
Generating IDs like Youtube or Bit.ly using PHP
PHP Base-62 encoding
Shortening Strings (URLs) using Base 62 Encoding
Base 2, 8, 16, 62, N Conversion – PHP

基于Swoole和Zookeeper的分布式ID生成器

大部分业务都需要一个唯一标识ID,比如订单ID、消息ID,通常使用的ID就是数据库的自增ID,比如MySQL的AUTO_INCREMENT;有时候这个ID还需要在不同系统里面传递、保存,又要保证唯一性。单机MySQL在高并发请求下面又可能存在锁/性能问题,于是Flicker使用两台MySQL来生成ID,一台从0开始,一台从1开始,步长为2,这样两台生成的ID不会互相重复,这个方案也可以扩展成N台,自增步长为N即可。
作为一个分布式ID应当避免在不同节点同步ID信息,通常都是基于时间戳和机器信息来生成。比如MongoDB的ObjectId是提前生成的为12字节=4字节UNIX时间戳+3字节机器码+2字节进程ID+2字节计数序列。
如果不需要访问数据库即能生成ID,性能可以更高。比如UUID V1,基于时间戳和网卡,采用128位,可以生成范围非常广的ID,但是生成的16进制值的36位字符串不好排序。在MySQL里面可以通过调整机器码(MAC)和时间戳位置顺序,并采用binary来存储以提高性能
Twitter开源的Snowflake则生成64位数字ID,包括41时间戳,10位机器码/节点码,12位计数序列,另外1位保留。采用基于时间戳的数字ID的好处是这个ID可以当作主键,并且已经粗略按时间排好序,可以直接分页读取,省去在时间字段上建立索引。

分布式ID通常需要用到机器信息(节点ID或MAC),一个机器通常只运行一个服务进程,所以通常不采用Nginx/Apache + PHP。参考这里实现一个基于Swoole和Zookpeer的64位ID生成器。基于Swoole可以快速开发一个Web/Socket server,不同于Apache/Nginx,它的PHP进程启动后是常驻运行的,资源初始化后可以重复使用,使用Zookpeer来获取当前进程的节点ID,一旦PHP进程退出后便会销毁对应的节点ID。
首先是生成ID

<?php

declare(strict_types=1);

namespace Dig\Ticket;

use Dig\Ticket\Exception\IllegalTimeException;

class Number
{
    public const TOTAL_BIT = 64;
    public const EPOCH_BIT = 42;
    public const NODE_BIT = 10;
    public const SEQUENCE_BIT = 12;

    public const MAX_NODE_ID = 2 ** self::NODE_BIT - 1;
    public const MAX_SEQUENCE_NUMBER = 2 ** self::SEQUENCE_BIT - 1;
    public const CUSTOM_EPOCH = 1262332800000;

    private $lastTimestamp = 0;
    private $sequence = 0;

    private $nodeId = 0;

    public function __construct(int $nodeId)
    {
        $this->nodeId = $nodeId;
    }

    public function getNodeId(): int
    {
        return $this->nodeId;
    }

    public function getTimestamp(): int
    {
        return (int) (\microtime(true) * 1000) - self::CUSTOM_EPOCH;
    }

    public function generate(): int
    {
        $current = $this->getTimestamp();
        if ($current < $this->lastTimestamp) {
            throw new IllegalTimeException('current timestamp cannot less than before');
        }
        if ($current === $this->lastTimestamp) {
            $this->sequence = ($this->sequence + 1) & self::MAX_SEQUENCE_NUMBER;
            if (0 === $this->sequence) {
                $current = $this->_waitNextTimestamp($current);
            }
        } else {
            $this->sequence = 0;
        }
        $this->lastTimestamp = $current;
        $id = $current << (self::TOTAL_BIT - self::EPOCH_BIT);
        $id = $id | ($this->getNodeId() << (self::TOTAL_BIT - self::EPOCH_BIT - self::NODE_BIT));
        $id = $id | $this->sequence;

        return $id;
    }

    private function _waitNextTimestamp($current)
    {
        while ($current === $this->lastTimestamp) {
            $current = $this->getTimestamp();
        }

        return $current;
    }
}

这里只涉及到ID生成,包括时间戳,序列号获取,而节点ID由其他对象生成并传入。2的42次方减1等于4398046511103,大概就是2109年5月15日,可以使用(2^42-1)/(365*24*60*60*1000)≈139年,距离现在还有90年可以用,仍然是一个非常大的可使用范围。CUSTOM_EPOC是自定义的时间戳偏移量,以便选取合适的ID生成下限和上限。距离现在节点生成的接口定义

<?php

declare(strict_types=1);

namespace Dig\Ticket;

interface NodeInterface
{
    public const MAX_NODE_ID = 2 ** Number::NODE_BIT - 1;

    public function getId(): int;
}

这里定义了最大节点序号不能超过1023,可以依据自己的需求更改范围。节点ID的实现可以是基于网卡/进程ID/文件配置等等实现,但是不同机器或多进程之间需要不一样的ID或者需要锁保证上面的generate函数。

<?php

declare(strict_types=1);

namespace Dig\Zookeeper;

class Client extends \Zookeeper
{
    public function makePath(string $path, string $value = ''): bool
    {
        $arrPath = \explode('/', $path);
        if (!empty($arrPath)) {
            $arrPath = \array_filter($arrPath);
            $subpath = '';
            $flag = true;
            foreach ($arrPath as $p) {
                $subpath .= '/'.$p;
                if (!$this->exists($subpath)) {
                    if (!$this->makeNode($subpath, $value)) {
                        $flag = false;
                        break;
                    }
                }
            }

            return $flag;
        }

        return false;
    }

    public function makeNode(string $path, string $value, array $acls = [], int $flag = 0): bool
    {
        if (empty($acls)) {
            $acls = [
                [
                    'perms' => \Zookeeper::PERM_ALL,
                    'scheme' => 'world',
                    'id' => 'anyone',
                ],
            ];
        }
        if ($this->create($path, $value, $acls, $flag)) {
            return true;
        }

        return false;
    }

    public function deletePath(string $path): bool
    {
        $children = $this->getChildren($path);
        if (!empty($children)) {
            foreach ($children as $child) {
                $subpath = $path.'/'.$child;
                $this->deletePath($subpath);
            }
        }

        return $this->delete($path);
    }
}

这里使用Zookeeper实现

<?php

declare(strict_types=1);

namespace Dig\Ticket\Node;

use Dig\Ticket\Exception\UnavailableNodeIdException;
use Dig\Ticket\NodeInterface;
use Dig\Zookeeper\Client;

class Zookeeper implements NodeInterface
{
    private $zk;
    private $dsn;
    private $pool;
    private $basePath = '/dig/ticket';
    private $acls = [
        [
            'perms' => \Zookeeper::PERM_ALL,
            'scheme' => 'world',
            'id' => 'anyone',
        ],
    ];
    private $id;

    public function __construct(string $dsn, string $path = '/sim/ticket')
    {
        $this->dsn = $dsn;
        $this->pool = new \SplQueue();
        if (!empty($path)) {
            $this->basePath = $path;
        }
    }

    public function getZookeeper(): Client
    {
        if (null === $this->zk) {
            $this->zk = new Client($this->dsn);
        }

        return $this->zk;
    }

    public function getId(): int
    {
        if (null === $this->id) {
            if (!$this->getZookeeper()->exists($this->basePath)) {
                $this->getZookeeper()->makePath($this->basePath);
            }
            $i = 1;
            $length = \mb_strlen((string) self::MAX_NODE_ID);
            $nodeId = \sprintf('%0'.$length.'d', $i);
            $children = $this->getZookeeper()->getChildren($this->basePath);
            $children = empty($children) ? [] : $children;
            for (; $i <= self::MAX_NODE_ID; ++$i) {
                $nodeId = \sprintf('%0'.$length.'d', $i);
                if (!\in_array($nodeId, $children)) {
                    $path = $this->basePath.'/'.$nodeId;
                    if ($this->getZookeeper()->exists($path)) {
                        //throw new UnavailableNodeIdException('node already exist: '.$path);
                        continue;
                    }
                    try {
                        $this->getZookeeper()->makeNode($path, $nodeId, $this->acls, \Zookeeper::EPHEMERAL);
                        break;
                    } catch (\ZookeeperException $e) {
                        //throw new UnavailableNodeIdException('cannot create node in zookeeper: '.$e->getMessage());
                        continue;
                    }
                }
            }
            if (self::MAX_NODE_ID === $i) {
                throw new UnavailableNodeIdException('cannot create node in zookeeper: reach max node limit '.self::MAX_NODE_ID);
            }
            $this->id = $i;
        }

        return $this->id;
    }
}

这里遍历查询1-1023之间的节点是否都已在Zookeeper上注册,如果没有则注册,Zookeeper会保证只有一个客户端注册成功。注册的节点类型位Zookeeper::EPHEMERAL,在客户端退出时,该节点会被自动删除,方便其他机器/进程申请。在这篇文章里面我们也使用Zookeeper::EPHEMERAL配合Zookeeper::EPHEMERAL,生成序列号,用来确定进程的master/slave。
初始化并运行Swoole Web server,需要传入Zookeeper的连接字符串,可以使用docker快速部署

<?php
include __DIR__.'/../vendor/autoload.php';
use Dig\Ticket\Number;
use Dig\Ticket\Node\Zookeeper as ZookeeperNode;


/** 
 * swoole - zookeeper tick dispatch issue: https://github.com/php-zookeeper/php-zookeeper
*/
$host = getenv("ZOOKEEPER_CONNECTION");
$host = empty($host) ? "192.168.33.1:2181" : $host;
$node = new ZookeeperNode($host);

$http = new \Swoole\Http\Server("0.0.0.0", 9501);

$http->on("start", function ($server) {
    echo "Swoole http server is started at http://0.0.0.0:9501\n";
});

$http->on("WorkerStart", function ($server, $workerId) use($node) {
    // https://wiki.swoole.com/wiki/page/325.html
    // https://wiki.swoole.com/wiki/page/852.html
    // https://wiki.swoole.com/wiki/page/865.html
    // use lazy initial zk here, so that each worker can hold its own zk resource
    // if we only run swoole http server in 1 worker process (1 CPU), then no need to consider this
    $id = $node->getId();
    $server->nodeId = $id;
    $server->number = new Number($server->nodeId);
});

$http->on("request", function ($request, $response) use ($http) {
    $data = $http->number->generate();
    $response->end($data);
});

$http->start();

访问本机的9501端口即可以得到ID了,完整代码在这里。Swoole默认运行与CPU核数量相同的worker进程数,注意这里需要WorkerStart里初始化获取Node节点ID,如果只是运行一个Swoole worker进程,也可以在外面获取节点ID。可以将Swool\Htpp\Server替换成React\Http\Server或者Amp\Http\Server,它们在单个进程里面loop,每个进程分别持有自己的节点序号,可以保证生成的ID不冲突,性能方面Swoole > Amp > ReactPHP。
可以采用Swoole\Server + thrift/gRPC改造这些代码,提供RPC服务。
注意ID的生成是随时间递增的,依赖于时间戳,如果出现了时间回拨,将会抛出异常。一般解决方案包括:

  • 等待重试
  • 使用Int64原子自增量代替时间戳,跳过时间戳判断
  • 使用预留的节点ID
  • 关闭时钟同步
  • 使用备选自增量方案

生成的ID并不是严格递增的,只是千分一秒递增,对于微博、Twiter的Timeline够用;但也有好处,比如别人不能通过ID相减了解美团的订单量。

参考链接:
如何设计一个分布式ID生成器(Distributed ID Generator),并保证ID按时间粗略有序?
生成全局唯一ID的3个思路,来自一个资深架构师的总结
Distributed unique id generation
Unique ID generation in distributed systems
Optimised UUIDs in mysql
Storing UUID Values in MySQL Tables
Mysql 8.0: UUID support
How to store a 128 bit number in a single column in MySQL?
Generating unique IDs in a distributed environment at high scale
Leaf——美团点评分布式ID生成系统
分布式ID增强篇–优化时钟回拨问题
Ticket Servers: Distributed Unique Primary Keys on the Cheap
Sharding & IDs at Instagram