etcd基本概念
etcd是一个高可用的分布式键值存储系统，是CoreOS（现在隶属于Red Hat）公司开发的一个开源项目。它提供了一个简单的接口来存储和检索键值对数据，并使用Raft协议实现了分布式一致性。etcd广泛应用于Docker、Kubernetes等分布式系统中，用于存储配置信息、服务发现、领导者选举等方面。
etcd的基本知识
Etcd的定义：Etcd是一个分布式的、高可用的键值存储系统，具有快速响应、支持高并发等特点。
Etcd的优点：高可用、数据一致性、快速响应、支持分布式事务。
Etcd的应用场景：存储集群中各种配置信息、元数据以及服务发现等。
Etcd的数据结构：类似于键值对的数据结构，支持多种数据类型，如字符串、整数、布尔、数组等。
Etcd的API：提供丰富的API接口，通过HTTP协议进行访问和操作。
Etcd的数据同步：Etcd使用Raft协议来保证数据的一致性和高可用性，在多个节点之间同步数据。
Etcd的集群部署：Etcd可以部署为单节点或多节点集群，多节点集群可以提高系统的可用性和性能。
Etcd的安全性：Etcd提供了诸如SSL/TLS加密、授权、访问控制等安全机制，保障了数据的安全性。
Etcd的监控：Etcd提供了各种监控指标和工具，方便管理员对集群进行监控和管理。
Etcd的常用工具：包括etcdctl命令行工具、etcd browser浏览器、etcd dashboard仪表板、etcdwatch等。
etcd优势
etcd是一种分布式键值存储系统，具有以下特点和优势：
强一致性：etcd使用Raft协议实现强一致性，确保每个节点上的数据始终保持一致。
高可用性：etcd可以通过多副本和自动故障转移机制来实现高可用性，即使某个节点故障，也可以保证服务不中断。
高性能：etcd采用预写日志（WAL）技术，可以快速写入数据，同时还支持快速读取和查询。
分布式：etcd可以在多台服务器上运行，数据可以在多个节点之间分布，提高了系统的可伸缩性和性能。
安全性：etcd支持TLS加密和认证，确保数据传输的安全性。
简单易用：etcd提供简单易用的HTTP/JSON API，可以方便地进行数据读写和查询。
开源免费：etcd是一个完全开源的项目，具有Apache 2.0许可证。
etcd单机部署
1. 主机环境介绍
测试环境，主机采用虚拟机。
主机名
ip
系统版本
CPU
内存
apisix
192.168.199.101
Rocky Linux release 8.9
2核2线程
4G
2. 软件版本
etcd-v3.5.13
下载地址：https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
3. 上传服务器解压
tar xf etcd-v3.5.13-linux-amd64.tar.gz
mkdir -p /opt/etcd/bin
cp -a etcd-v3.5.13-linux-amd64/etcd* /opt/etcd/bin/
4. 创建配置文件及数据目录
cd /opt/etcd/
mkdir -p data conf
5. 创建配置文件
cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME="apisix"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379,http://127.0.0.1:2379"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://0.0.0.0:2380,http://127.0.0.1:2380"
ETCD_ENABLE_V2="true"
EOF
6. 注册成一个服务
cat << 'EOF' > /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
User=root
Type=notify
EnvironmentFile=/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd
ExecStart=/opt/etcd/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
7. 启动服务
systemctl daemon-reload
systemctl enable --now etcd
netstat -ntplu | egrep etcd
tcp6 0 0 :::2379 :::* LISTEN 1228/etcd
tcp6 0 0 :::2380 :::* LISTEN 1228/etcd
8. 设置环境变量
echo 'export PATH=$PATH:/opt/etcd/bin' > /etc/profile.d/etcd.sh
source /etc/profile.d/etcd.sh
9. 查看服务状态
etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 17ac6c9b1b6e7f0c | 3.5.13 | 20 kB | true | false | 2 | 4 | 4 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
etcdctl endpoint health --write-out=table
+----------------+--------+------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+----------------+--------+------------+-------+
| 127.0.0.1:2379 | true | 3.571276ms | |
+----------------+--------+------------+-------+
etcdctl member list --write-out=table
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+
| 17ac6c9b1b6e7f0c | started | apisix | http://0.0.0.0:2380,http://127.0.0.1:2380 | http://0.0.0.0:2379,http://127.0.0.1:2379 | false |
+------------------+---------+--------+-------------------------------------------+-------------------------------------------+------------+
etcd集群部署
1. 主机环境介绍
主机名
ip
系统版本
CPU
内存
etcd-node01
192.168.199.101
Rocky Linux release 8.9
2核2线程
4G
etcd-node02
192.168.199.102
Rocky Linux release 8.9
2核2线程
4G
etcd-node03
192.168.199.103
Rocky Linux release 8.9
2核2线程
4G
2. 修改主机名及申明hosts
三个节点分别执行
#每台主机分别执行
hostnamectl set-hostname etcd-node01
hostnamectl set-hostname etcd-node02
hostnamectl set-hostname etcd-node03
#每台主机执行
cat << 'EOF' > /etc/hosts
192.168.199.101 etcd-node01
192.168.199.102 etcd-node02
192.168.199.103 etcd-node03
EOF
3. 校对时间
三个节点统一执行
yum install ntpdate -y
ntpdate -s ntp1.aliyun.com
4. 下载etcd二进制程序
下载地址：https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
5. 解压及创建配置目录
三个节点统一执行
tar xf etcd-v3.5.13-linux-amd64.tar.gz
mkdir -pv /opt/etcd/{bin,data,conf}
cp -a etcd-v3.5.13-linux-amd64/etcd* /opt/etcd/bin/
6. 编写配置文件
注意：每个节点的配置文件是不同的，这里一定要注意，否则启动时无法组建集群。
etcd-node01 - etcd.conf
cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node01
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.199.101:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.199.101:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.199.101:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.199.101:2380,etcd-node02=http://192.168.199.102:2380,etcd-node03=http://192.168.199.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.199.101:2379"
EOF
注意：
ETCD_NAME=etcd-node02 #注意：这里采用主机名，每个节点保持唯一性
ETCD_DATA_DIR="/opt/etcd/data" #数据存储目录
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.199.101:2380,etcd-node02=http://192.168.199.102:2380,etcd-node03=http://192.168.199.103:2380" #三个不同的节点
etcd-node02 - etcd.conf
cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node02
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.199.102:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.199.102:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.199.102:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.199.101:2380,etcd-node02=http://192.168.199.102:2380,etcd-node03=http://192.168.199.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.199.102:2379"
EOF
etcd-node03 - etcd.conf
cat << 'EOF' > /opt/etcd/conf/etcd.conf
ETCD_NAME=etcd-node03
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="http://192.168.199.103:2380,http://127.0.0.1:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.199.103:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.199.103:2380"
ETCD_INITIAL_CLUSTER="etcd-node01=http://192.168.199.101:2380,etcd-node02=http://192.168.199.102:2380,etcd-node03=http://192.168.199.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.199.103:2379"
EOF
7. 注册服务
注意：每个节点统一操作
cat << 'EOF' > /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
User=root
Type=notify
EnvironmentFile=/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd
ExecStart=/opt/etcd/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
8. 启动服务
注意：三个节点统一操作
systemctl enable --now etcd
9. 配置环境变量
注意：三个节点统一操作
echo 'PATH=$PATH:/opt/etcd/bin' > /etc/profile.d/etcd.sh
source /etc/profile.d/etcd.sh
10. 查看集群信息
etcdctl --endpoints=192.168.199.101:2379,192.168.199.102:2379,192.168.199.103:2379 endpoint status --write-out=table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.199.101:2379 | 8dc9f79502ff52fe | 3.5.13 | 20 kB | false | false | 2 | 13 | 13 | |
| 192.168.199.102:2379 | 8fdf9dd965ff87d5 | 3.5.13 | 20 kB | true | false | 2 | 13 | 13 | |
| 192.168.199.103:2379 | fc380e2e4f60630e | 3.5.13 | 20 kB | false | false | 2 | 13 | 13 | |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
11. 测试故障
将上面的 leader 节点关机，然后查看 leader 节点是否会在其他节点上启动。
etcdctl --endpoints=192.168.199.101:2379,192.168.199.102:2379,192.168.199.103:2379 endpoint status --write-out=table
{"level":"warn","ts":"2024-05-08T10:27:04.259984+0800","logger":"etcd-client","caller":"v3@v3.5.13/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000363c0/192.168.199.101:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint 192.168.199.102:2379 (context deadline exceeded)
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.199.101:2379 | 8dc9f79502ff52fe | 3.5.13 | 20 kB | true | false | 3 | 14 | 14 | |
| 192.168.199.103:2379 | fc380e2e4f60630e | 3.5.13 | 20 kB | false | false | 3 | 14 | 14 | |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
leader节点已经漂移到 192.168.199.101 上面了。
12. 故障恢复
启动被关闭的节点，查看 leader 会不会漂移，一般来说，这种情况下，leader不要漂移是最好的，实际操作如下
etcdctl --endpoints=192.168.199.101:2379,192.168.199.102:2379,192.168.199.103:2379 endpoint status --write-out=table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.199.101:2379 | 8dc9f79502ff52fe | 3.5.13 | 20 kB | true | false | 3 | 15 | 15 | |
| 192.168.199.102:2379 | 8fdf9dd965ff87d5 | 3.5.13 | 20 kB | false | false | 3 | 15 | 15 | |
| 192.168.199.103:2379 | fc380e2e4f60630e | 3.5.13 | 20 kB | false | false | 3 | 15 | 15 | |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
由上面可得，当故障恢复后，leader节点没有发生漂移。
13. 数据写入测试
在 etcd-node01 写入数据
root@etcd-node01(192.168.199.101)~>etcdctl put greeting "hello etcd"
OK
在 etcd-node03 读取数据
root@etcd-node03(192.168.199.103)~>etcdctl get greeting
greeting
hello etcd
14. 查看etcd命令
etcdctl member list -w table
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+
| 8dc9f79502ff52fe | started | etcd-node01 | http://192.168.199.101:2380 | http://192.168.199.101:2379 | false |
| 8fdf9dd965ff87d5 | started | etcd-node02 | http://192.168.199.102:2380 | http://192.168.199.102:2379 | false |
| fc380e2e4f60630e | started | etcd-node03 | http://192.168.199.103:2380 | http://192.168.199.103:2379 | false |
+------------------+---------+-------------+-----------------------------+-----------------------------+------------+
etcdctl --endpoints=etcd-node01:2379,etcd-node02:2379,etcd-node03:2379 endpoint status --write-out=table
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| etcd-node01:2379 | 8dc9f79502ff52fe | 3.5.13 | 20 kB | true | false | 3 | 16 | 16 | |
| etcd-node02:2379 | 8fdf9dd965ff87d5 | 3.5.13 | 20 kB | false | false | 3 | 16 | 16 | |
| etcd-node03:2379 | fc380e2e4f60630e | 3.5.13 | 20 kB | false | false | 3 | 16 | 16 | |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
etcdctl --endpoints=etcd-node01:2379,etcd-node02:2379,etcd-node03:2379 endpoint health --write-out=table
+------------------+--------+-------------+-------+
| ENDPOINT | HEALTH | TOOK | ERROR |
+------------------+--------+-------------+-------+
| etcd-node01:2379 | true | 6.397123ms | |
| etcd-node02:2379 | true | 8.103458ms | |
| etcd-node03:2379 | true | 10.570859ms | |
+------------------+--------+-------------+-------+
参考链接
https://juejin.cn/post/7218564871830323256
https://blog.csdn.net/qq_37837432/article/details/121755927
--- EOF ---