CentOS部署基于flannel网络的docker

Flannel实质上是一种“覆盖网络(overlay network)”，也就是将TCP数据包装在另一种网络包里面进行路由转发和通信，目前已经支持UDP、VxLAN、AWS VPC和GCE路由等数据转发方式。

安装docker

yum -y install yum-utils
yum-config-manager     --add-repo     https://mirrors.ustc.edu.cn/docker-ce/linux/centos/docker-ce.repo     
yum -y install docker-ce
systemctl start docker
#docker --version
Docker version 20.10.1, build 831ebea

已安装docker环境，开启路由转发，关闭沙盒，关闭防火墙

[root@localhost ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# vim /etc/selinux/config 
SELINUX=disabled
[root@localhost ~]# setenforce 0
[root@localhost ~]# getenforce 
Permissive

设置防火墙默认策略

[root@localhost ~]# iptables -P FORWARD ACCEPT

安装Flannel和etcd

[root@localhost ~]# yum -y install flannel etcd

配置etcd识别docker集群
这里以一台docker为例

[root@localhost ~]# cp -p /etc/etcd/etcd.conf /etc/etcd/etcd.conf.bak
[root@localhost ~]# vim /etc/etcd/etcd.conf
# etcd存放数据目录，为了方便改成了和etcd_name一样的名字
ETCD_DATA_DIR="/var/lib/etcd/etcd1"
# 用于与其他节点通信,写本机ip
ETCD_LISTEN_PEER_URLS="http://192.168.1.11:2380"
# 客户端会连接到这里和 etcd 交互，本机ip和回环ip
ETCD_LISTEN_CLIENT_URLS="http://192.168.1.11:2379,http://127.0.0.1:2379"
# 节点名称，每台主机都不一样
ETCD_NAME="etcd1"
# 该节点同伴监听地址，这个值会告诉集群中其他节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.11:2380"
# 对外公告的该节点客户端监听地址，这个值会告诉集群中其他节点
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.11:2379"
# 集群中的主机信息，多个docker主机，就name=http://ip:2380，逗号隔开
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.11:2380"
# 集群token，建议修改一个，同一个集群中的token一致
ETCD_INITIAL_CLUSTER_TOKEN="etcd-test"
# 新建集群的时候，这个值为new假如已经存在的集群，这个值为 existing
ETCD_INITIAL_CLUSTER_STATE="new"

etcd服务识别配置文件中的值

[root@localhost ~]# vim /usr/lib/systemd/system/etcd.service 
# 在第十三行（ExecStart开头）末尾的前一个引号里面，添加如下
# 添加时不换行，空格隔开即可
--listen-peer-urls=\"${
   ETCD_LISTEN_PEER_URLS}\" 
--advertise-client-urls=\"${
   ETCD_ADVERTISE_CLIENT_URLS}\" 
--initial-cluster-token=\"${
   ETCD_INITIAL_CLUSTER_TOKEN}\" 
--initial-cluster=\"${
   ETCD_INITIAL_CLUSTER}\" 
--initial-cluster-state=\"${
   ETCD_INITIAL_CLUSTER_STATE}\"

启动etcd服务
整个集群启动该服务时，会有其中一台阻塞，需要另一台启动之后，阻塞的一端才会重新启动，如果报错，或者启动不了，请仔细检查改过的两个配置文件

systemctl daemon-reload
systemctl start etcd

查看集群健康状态，集群中有多台会显示多台

[root@localhost ~]# etcdctl cluster-health
member 12b11316a20f4e7 is healthy: got healthy result from http://192.168.1.11:2379
cluster is healthy

查看集群leader

上述说到的启动时阻塞的情况就是因为leader没启动，集群中其他etcd节点不会启动

[root@localhost ~]# etcdctl member list
12b11316a20f4e7: name=etcd1 peerURLs=http://192.168.1.11:2380 clientURLs=http://192.168.1.11:2379 isLeader=true

创建分配网段的json文件

[root@localhost ~]# vim /root/etcd.json
{
   
  "NetWork":"10.10.0.0/16",
  "SubnetLen":24,
  "Backend":{
   
    "Type":"vxlan"
  }
}

将文件导入etcd的endpoint中

[root@localhost ~]# etcdctl --endpoint=http://192.168.1.11:2379 \
set /usr/local/bin/network/config < /root/etcd.json
{
   
  "NetWork":"10.10.0.0/16",
  "SubnetLen":24,
  "Backend":{
   
    "Type":"vxlan"
  }
}

查看导入文件

[root@localhost ~]# etcdctl get /usr/local/bin/network/config
{
   
  "NetWork":"10.10.0.0/16",
  "SubnetLen":24,
  "Backend":{
   
    "Type":"vxlan"
  }
}

更改flannel配置文件

[root@localhost ~]# vim /etc/sysconfig/flanneld
# 集群中每台都写自己的ip
FLANNEL_ETCD_ENDPOINTS="http://192.168.1.11:2379"
FLANNEL_ETCD_PREFIX="/usr/local/bin/network"

启动flannel服务

[root@localhost ~]# systemctl start flanneld
[root@localhost ~]# systemctl enable flanneld

查看flannel分配的ip

[root@localhost ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.10.41.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::f48f:51ff:fe8c:7500  prefixlen 64  scopeid 0x20<link>
        ether f6:8f:51:8c:75:00  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 33 overruns 0  carrier 0  collisions 0

整合docker与flannel网络通信
查看mtu值和网关

[root@localhost ~]# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.10.0.0/16
FLANNEL_SUBNET=10.10.41.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

修改docker启动项

[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
# 第十四行末尾添加
--bip=10.10.41.1/24 --mtu=1450

重启docker

[root@localhost ~]# systemctl daemon-reload 
[root@localhost ~]# systemctl restart docker

查看docker0网卡，已经作为了flannel网络的网关存在

[root@localhost ~]# ifconfig docker0
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.10.41.1  netmask 255.255.255.0  broadcast 10.10.41.255
        ether 02:42:31:92:1c:1e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0