AMQP协议
AMQP，即Advanced Message Queuing Protocol，一个提供统一消息服务的应用层标准高级消息队列协议，是应用层协议的一个开放标准，为面向消息的中间件设计。基于此协议的客户端与消息中间件可传递消息，并不受客户端/中间件不同产品，不同的开发语言等条件的限制。Erlang中的实现有RabbitMQ等。
docker 集群部署
zookeeper
首先部署zookeeper集群（参考zookepper集群安装配置）然后创建日志目录log1 log2 log3 用于挂载 然后创建三个配置文件配置 server.properties 每台机器 borker.id
配置文件 三个服务的broker.id各不同 需要区分开
############################# Server Basics #############################
broker.id=1
#broker.id=2
#broker.id=3
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
############################# Socket Server Settings #############################
...
...
############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/kfk/logs
...
...
启动docker kafka容器 默认的entrypoint执行时会覆盖配置文件 虽然可通过环境变量方式配置 但这边采用挂载配置文件的方式 通过指定命令的方式执行 在最后添加/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
docker run -d --rm --name kfk1 --hostname="kfk1" -e JMX_PORT="19987" -e KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" -v /home/docker/kafka/config/kfk1/:/opt/kafka/config/ -v /home/docker/kafka/logs/log1:/kfk/logs -p 19091:19091 -p 19987:19987 --network inet apache/kafka:3.8.0 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties\
&&\
docker run -d --rm --name kfk2 --hostname="kfk2" -e JMX_PORT="19988" -e KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" -v /home/docker/kafka/config/kfk2/:/opt/kafka/config/ -v /home/docker/kafka/logs/log2:/kfk/logs -p 19092:19092 -p 19988:19988 --network inet apache/kafka:3.8.0 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties\
&&\
docker run -d --rm --name kfk3 --hostname="kfk3" -e JMX_PORT="19989" -e KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" -v /home/docker/kafka/config/kfk3/:/opt/kafka/config/ -v /home/docker/kafka/logs/log3:/kfk/logs -p 19093:19093 -p 19989:19989 --network inet apache/kafka:3.8.0 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
- EFAK
准备解压好的efak文件 通过dockerfile打包成自制容器
FROM openjdk:11-jdk
# 更换国内镜像源并安装ke.sh执行必要的工具
RUN echo "deb https://mirrors.ustc.edu.cn/debian/ bookworm main contrib non-free non-free-firmware \
deb-src https://mirrors.ustc.edu.cn/debian/ bookworm main contrib non-free non-free-firmware \
deb https://mirrors.ustc.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware \
deb-src https://mirrors.ustc.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware \
deb https://mirrors.ustc.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware \
deb-src https://mirrors.ustc.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware \
deb https://mirrors.ustc.edu.cn/debian-security/ bookworm-security main contrib non-free non-free-firmware \
deb-src https://mirrors.ustc.edu.cn/debian-security/ bookworm-security main contrib non-free non-free-firmware" > /etc/apt/sources.list && \
apt-get update && apt-get install -y procps gawk coreutils
# 复制efak文件 entrypoint 文件到容器内
COPY ./efak-web-3.0.1 ./entrypoint.sh /opt/efak/
WORKDIR /opt/efak/
#ke必要的环境变量
ENV KE_HOME=/opt/efak/
ENTRYPOINT ["/opt/efak/entrypoint.sh" ]
#entrypoint 执行ke.sh 并通过tail阻塞
#!/bin/bash
nohup /opt/efak/bin/ke.sh start ;\
tail -f /opt/efak/logs/log.log
执行打包命令
docker image build -f ./Dockerfile -t efak:1.0.0 .
efak的配置文件
#system-config.properties
######################################
# multi zookeeper & kafka cluster list
# Settings prefixed with 'kafka.eagle.' will be deprecated, use 'efak.' instead
######################################
# efak.zk.cluster.alias=cluster1,cluster2
efak.zk.cluster.alias=cluster1
cluster1.zk.list=zk1:2181,zk2:2181,zk3:2181
...
...
...
######################################
# kafka sqlite jdbc driver address 使用sqlite记得创建/hadoop/kafka-eagle/db/ 目录
######################################
efak.driver=org.sqlite.JDBC
efak.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
efak.username=root
efak.password=www.kafka-eagle.org
######################################
# kafka mysql jdbc driver address
######################################
# efak.driver=com.mysql.cj.jdbc.Driver
# efak.url=jdbc:mysql://127.0.0.1:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
# efak.username=root
# efak.password=123456
启动容器
docker run --rm --name efak --hostname="efak" -v /home/docker/efak/system-config.properties:/opt/efak/conf/system-config.properties -v /hadoop/kafka-eagle/db/:/hadoop/kafka-eagle/db/ -p 8048:8048 --network inet efak:1.0.0
全部启动后 docker ps
zhangzhitong@zhangzhitong-virtual-machine:/home/docker/efak$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
397e8a1986f7 efak:1.0.0 "/opt/efak/entrypoin…" 21 minutes ago Up 21 minutes 0.0.0.0:8048->8048/tcp, :::8048->8048/tcp efak
4344959a4507 apache/kafka:3.8.0 "/__cacert_entrypoin…" 8 hours ago Up 8 hours 0.0.0.0:19093->19093/tcp, :::19093->19093/tcp, 9092/tcp, 0.0.0.0:19989->19989/tcp, :::19989->19989/tcp kfk3
952e64d71704 apache/kafka:3.8.0 "/__cacert_entrypoin…" 8 hours ago Up 8 hours 0.0.0.0:19092->19092/tcp, :::19092->19092/tcp, 9092/tcp, 0.0.0.0:19988->19988/tcp, :::19988->19988/tcp kfk2
94de141e0da4 apache/kafka:3.8.0 "/__cacert_entrypoin…" 8 hours ago Up 8 hours 0.0.0.0:19091->19091/tcp, :::19091->19091/tcp, 9092/tcp, 0.0.0.0:19987->19987/tcp, :::19987->19987/tcp kfk1
6b53bd089bec zookeeper:3.9.2 "/docker-entrypoint.…" 10 hours ago Up 10 hours 8080/tcp, 0.0.0.0:12183->2181/tcp, :::12183->2181/tcp, 0.0.0.0:32772->2888/tcp, :::32772->2888/tcp, 0.0.0.0:32773->3888/tcp, :::32773->3888/tcp zk3
32526de8b56f zookeeper:3.9.2 "/docker-entrypoint.…" 10 hours ago Up 10 hours 8080/tcp, 0.0.0.0:12182->2181/tcp, :::12182->2181/tcp, 0.0.0.0:32770->2888/tcp, :::32770->2888/tcp, 0.0.0.0:32771->3888/tcp, :::32771->3888/tcp zk2
c945f184a078 zookeeper:3.9.2 "/docker-entrypoint.…" 10 hours ago Up 10 hours 8080/tcp, 0.0.0.0:12181->2181/tcp, :::12181->2181/tcp, 0.0.0.0:32768->2888/tcp, :::32768->2888/tcp, 0.0.0.0:32769->3888/tcp, :::32769->3888/tcp zk1
Kraft
docker kafka 通过inspect命令查看镜像启动执行了一个/__cacert_entrypoint.sh /etc/kafka/docker/run -> run -> configureDefaults->configure->launch 依次通过cat 命令查看，最后发现执行的是/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties 那么集群配置则需要对/opt/kafka/config/server.properties该配置文件进行挂载修改即可