使用CNI网络插件(calico)实现docker容器跨主机互联

博客 分享
0 308
张三
张三 2023-06-09 15:10:24
悬赏：0 积分 收藏

使用CNI网络插件(calico)实现docker容器跨主机互联

目录

一.系统环境

本文主要基于Docker 20.10.12和Linux操作系统CentOS7.4。

服务器版本 calico版本 docker软件版本 Kubernetes(k8s)集群版本 CPU架构
CentOS Linux release 7.4.1708 (Core) v2.6.12 Docker version 20.10.12 v1.21.9 x86_64

etcd集群架构：etcd1为leader，etcd2为follower，etcd3为follower。

服务器 操作系统版本 CPU架构 进程 功能描述
etcd1/192.168.110.133 CentOS Linux release 7.4.1708 (Core) x86_64 etcd leader
etcd2/192.168.110.131 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower
etcd3/192.168.110.132 CentOS Linux release 7.4.1708 (Core) x86_64 etcd follower

Kubernetes集群架构：k8scloude1作为master节点，k8scloude2，k8scloude3作为worker节点。

服务器 操作系统版本 CPU架构 进程 功能描述
k8scloude1/192.168.110.130 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico k8s master节点
k8scloude2/192.168.110.129 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kubelet，kube-proxy，calico k8s worker节点
k8scloude3/192.168.110.128 CentOS Linux release 7.4.1708 (Core) x86_64 docker，kubelet，kube-proxy，calico k8s worker节点

二.前言

在Kubernetes集群中，容器之间的通信是非常重要的一部分。为了实现容器之间的跨主机互联，需要使用CNI网络插件。本文将介绍CNI网络插件的概念和常见的几种插件对比，并详细讲解如何使用Calico实现Docker容器跨主机互联。

calico的信息保存在etcd里，所以需要一套etcd集群，关于etcd集群的安装部署，可以查看博客《Kubernetes后台数据库etcd：安装部署etcd集群，数据备份与恢复》。

查看Kubernetes(k8s)环境里的calico的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html。

三.CNI网络插件简介

CNI(Containernetworking Interface)网络插件是一个由Linux基金会维护的开源项目，它可以为容器提供网络连接。在Kubernetes中，可以通过CNI网络插件来为Pod提供网络连接。

目前市面上主流的CNI网络插件有以下几种：

  • Flannel：使用VXLAN技术实现网络隔离和扁平化IP；
  • Calico：采用BGP协议实现高效的容器网络互连；
  • Weave Net：使用虚拟机间通信(VXLAN)技术，在容器之间创建多层网络；
  • Canal：结合Flannel和Calico两种CNI网络插件的优点，实现网络隔离和BGP路由。

四.常见的几种CNI网络插件对比

下面我们来对比这几种CNI网络插件。

CNI网络插件 优点 缺点 是否支持网络策略
Flannel 部署简单，性能优秀 网络层延迟高 否
Calico 性能最好，支持容器内BGP协议，支持网络策略 配置复杂 是
Weave Net 功能强大，跨平台支持 性能低下，容易出现网络死锁 是
Canal 结合了Flannel和Calico两种插件的优点，支持多种网络模式，可以满足不同的需求 部署和配置较为繁琐 是

综上所述，每种CNI网络插件都有其独特的优势和局限性，需要根据实际情况进行选择。

五.Calico网络之间是如何通信的

Calico是一种基于IP路由技术的CNI网络插件，它利用BGP协议来实现高效的容器网络互连。在Calico中，每个容器都被赋予了一个唯一的IP地址，这些IP地址在网络层面上是可达的，并且是通过数据包路由直接到达目标容器的。

Calico使用路由表来管理容器网络，每个主机上都会存在一个Calico Agent，它会监听Kubernetes API服务器，从而了解集群中所有容器的IP地址和状态。当某个容器需要向其他容器发起请求时，Calico会根据路由表信息进行查找，找到合适的路径，并将数据包转发给目标容器。

六.配置calico让物理机A上的docker容器c1可以访问物理机B上的docker容器c2

现在要解决的问题是：让物理机A上的docker容器c1可以访问物理机B上的docker容器c2！

方法一：物理机A上的容器c1想和物理机B上的容器c2通信，可以通过容器c1在物理机上映射一个端口，容器c2在物理机上映射一个端口，访问物理机的端口达到访问容器的目的，但是这样过于麻烦，有没有更好的方法呢？

方法二：可以通过网络插件来实现这个需求，这里使用calico网络插件。

6.1 安装部署etcd集群

因为calico的信息保存在etcd里，所以需要一套etcd集群。

查看etcd集群的健康状态。

[root@etcd1 ~]# etcdctl cluster-health
member 341a3c460c1c993a is healthy: got healthy result from http://192.168.110.131:2379
member 4679fe0fcb37326d is healthy: got healthy result from http://192.168.110.132:2379
member ab23bcc86cf3190b is healthy: got healthy result from http://192.168.110.133:2379
cluster is healthy

查看etcd集群的成员，可以看到etcd133是Leader。

[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
4679fe0fcb37326d: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

etcd现在什么数据也没有

[root@etcd1 ~]# etcdctl ls /

6.2 安装部署docker

三个节点安装docker，用来启动容器。

[root@etcd1 ~]# yum -y install docker-ce

[root@etcd2 ~]# yum -y install docker-ce

[root@etcd3 ~]# yum -y install docker-ce

修改docker的启动参数，设置docker使用etcd来存储数据，可以看到docker的启动脚本在/usr/lib/systemd/system/docker.service。

[root@etcd1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://docs.docker.com
[root@etcd1 ~]# etcdctl member list
341a3c460c1c993a: name=etcd131 peerURLs=http://192.168.110.131:2380 clientURLs=http://192.168.110.131:2379,http://localhost:2379 isLeader=false
4679fe0fcb37326d: name=etcd132 peerURLs=http://192.168.110.132:2380 clientURLs=http://192.168.110.132:2379,http://localhost:2379 isLeader=false
ab23bcc86cf3190b: name=etcd133 peerURLs=http://192.168.110.133:2380 clientURLs=http://192.168.110.133:2379,http://localhost:2379 isLeader=true

添加启动参数：--cluster-store=etcd://192.168.110.133:2379。

[root@etcd1 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd1 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock

重新加载配置，启动docker。

[root@etcd1 ~]# systemctl daemon-reload ;systemctl restart docker

可以看到参数添加成功：/usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock。这样设置之后，etcd就可以存储docker的后端数据了。

[root@etcd1 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:50 CST; 39s ago
     Docs: https://docs.docker.com
 Main PID: 1390 (dockerd)
   Memory: 30.8M
   CGroup: /system.slice/docker.service
           └─1390 /usr/bin/dockerd --cluster-store=etcd://192.168.110.133:2379 -H fd:// --containerd=/run/containerd/containerd.sock

其他两个节点进行相同操作，但是etcd的IP要修改为本机的地址。

[root@etcd2 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd2 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.131:2379 -H fd:// --containerd=/run/containerd/containerd.sock

[root@etcd2 ~]# systemctl daemon-reload ;systemctl restart docker

[root@etcd2 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:57 CST; 41s ago
     Docs: https://docs.docker.com
 Main PID: 1348 (dockerd)
   Memory: 32.4M
   CGroup: /system.slice/docker.service
           └─1348 /usr/bin/dockerd --cluster-store=etcd://192.168.110.131:2379 -H fd:// --containerd=/run/containerd/containerd.sock


[root@etcd3 ~]# vim /usr/lib/systemd/system/docker.service

[root@etcd3 ~]# grep ExecStart /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd --cluster-store=etcd://192.168.110.132:2379 -H fd:// --containerd=/run/containerd/containerd.sock

[root@etcd3 ~]# systemctl daemon-reload ;systemctl restart docker

[root@etcd3 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since 三 2022-02-16 15:39:59 CST; 41s ago
     Docs: https://docs.docker.com
 Main PID: 1355 (dockerd)
   Memory: 34.7M
   CGroup: /system.slice/docker.service
           └─1355 /usr/bin/dockerd --cluster-store=etcd://192.168.110.132:2379 -H fd:// --containerd=/run/containerd/containerd.sock

6.3 配置calico

创建calico目录，并创建配置文件，三个节点都需要。

[root@etcd1 ~]# mkdir /etc/calico

[root@etcd1 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.133:2379"
> EOF

#calico的配置文件已经配置好了
[root@etcd1 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.133:2379"

[root@etcd2 ~]# mkdir /etc/calico

[root@etcd2 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.131:2379"
> EOF

[root@etcd2 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.131:2379"

[root@etcd3 ~]# mkdir /etc/calico

[root@etcd3 ~]# cat > /etc/calico/calicoctl.cfg <<EOF
> apiVersion: v1
> kind: calicoApiConfig
> metadata:
> spec:
>   datastoreType: "etcdv2"
>   etcdEndpoints: "http://192.168.110.132:2379"
> EOF

[root@etcd3 ~]# cat /etc/calico/calicoctl.cfg
apiVersion: v1
kind: calicoApiConfig
metadata:
spec:
  datastoreType: "etcdv2"
  etcdEndpoints: "http://192.168.110.132:2379"

创建目录放置calico镜像和工具。

[root@etcd1 ~]# mkdir etcd-calico

[root@etcd1 ~]# cd etcd-calico/

calicoctl是calico命令行工具，calico-node-v2.tar是calico-node镜像包。

[root@etcd1 etcd-calico]# ls
calicoctl  calico-node-v2.tar

其他两个节点也需要这两个文件

[root@etcd1 etcd-calico]# scp ./* etcd2:/root/etcd-calico/
root@etcd2's password: 
calicoctl                                                                                                                                                                       100%   31MB  98.1MB/s   00:00    
calico-node-v2.tar                                                                                                                                                              100%  269MB  29.9MB/s   00:09    

[root@etcd1 etcd-calico]# scp ./* etcd3:/root/etcd-calico/
root@etcd3's password: 
calicoctl                                                                                                                                                                       100%   31MB  96.3MB/s   00:00    
calico-node-v2.tar                                                                                                                                                              100%  269MB  67.3MB/s   00:04

给calicoctl赋予可执行权限

[root@etcd1 etcd-calico]# chmod +x calicoctl 

[root@etcd1 etcd-calico]# mv calicoctl /bin/

加载镜像

[root@etcd1 etcd-calico]# docker load -i calico-node-v2.tar
df64d3292fd6: Loading layer [==================================================>]  4.672MB/4.672MB
d6f0e85be2d0: Loading layer [==================================================>]  8.676MB/8.676MB
c9818c503193: Loading layer [==================================================>]  250.9kB/250.9kB
1f748fca5871: Loading layer [==================================================>]  4.666MB/4.666MB
714c5990d9e8: Loading layer [==================================================>]  263.9MB/263.9MB
Loaded image: quay.io/calico/node:v2.6.12

另外两个节点也是相同的操作。

[root@etcd2 ~]# mkdir etcd-calico

[root@etcd2 ~]# cd etcd-calico/

[root@etcd2 etcd-calico]# pwd
/root/etcd-calico

[root@etcd2 etcd-calico]# ls
calicoctl  calico-node-v2.tar

[root@etcd2 etcd-calico]# chmod +x calicoctl
[root@etcd2 etcd-calico]# mv calicoctl /bin/

[root@etcd2 etcd-calico]# docker load -i calico-node-v2.tar

[root@etcd3 ~]# mkdir etcd-calico

[root@etcd3 ~]# cd etcd-calico/

[root@etcd3 etcd-calico]# ls
calicoctl  calico-node-v2.tar

[root@etcd3 etcd-calico]# chmod +x calicoctl

[root@etcd3 etcd-calico]# mv calicoctl /bin/

[root@etcd3 etcd-calico]# docker load -i calico-node-v2.tar

三个节点上都启动Calico node

[root@etcd1 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg
Running command to load modules: modprobe -a xt_set ip6_tables
......
Running the following command to start calico-node:

docker run --net=host --privileged --name=calico-node -d --restart=always -e NODENAME=etcd1 -e CALICO_NETWORKING_BACKEND=bird -e CALICO_LIBNETWORK_ENABLED=true -e ETCD_ENDPOINTS=http://192.168.110.133:2379 -v /var/log/calico:/var/log/calico -v /var/run/calico:/var/run/calico -v /lib/modules:/lib/modules -v /run:/run -v /run/docker/plugins:/run/docker/plugins -v /var/run/docker.sock:/var/run/docker.sock quay.io/calico/node:v2.6.12

Image may take a short time to download if it is not available locally.
Container started, checking progress logs.

2022-02-16 08:00:06.363 [INFO][9] startup.go 173: Early log level set to info
......
2022-02-16 08:00:06.536 [INFO][14] client.go 202: Loading config from environment
Starting libnetwork service
Calico node started successfully

每个节点都建立了一个calico-node容器

[root@etcd1 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED          STATUS          PORTS     NAMES
ac7d48a378b6   quay.io/calico/node:v2.6.12   "start_runit"   57 seconds ago   Up 56 seconds             calico-node

另外两个节点也启动Calico node

[root@etcd2 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg

[root@etcd2 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED              STATUS              PORTS     NAMES
bc99f286802f   quay.io/calico/node:v2.6.12   "start_runit"   About a minute ago   Up About a minute             calico-node

[root@etcd3 etcd-calico]# calicoctl node run --node-image=quay.io/calico/node:v2.6.12 -c /etc/calico/calicoctl.cfg

[root@etcd3 etcd-calico]# docker ps
CONTAINER ID   IMAGE                         COMMAND         CREATED              STATUS              PORTS     NAMES
07ba9ccdcd4d   quay.io/calico/node:v2.6.12   "start_runit"   About a minute ago   Up About a minute             calico-node

因为我们是使用etcd保存数据的，可以看到对方的主机的信息。

[root@etcd1 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.131 | node-to-node mesh | up    | 08:00:13 | Established |
| 192.168.110.132 | node-to-node mesh | up    | 08:00:14 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

[root@etcd2 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.133 | node-to-node mesh | up    | 08:00:13 | Established |
| 192.168.110.132 | node-to-node mesh | up    | 08:00:14 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.


[root@etcd3 etcd-calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+-----------------+-------------------+-------+----------+-------------+
|  PEER ADDRESS   |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+-----------------+-------------------+-------+----------+-------------+
| 192.168.110.133 | node-to-node mesh | up    | 08:00:15 | Established |
| 192.168.110.131 | node-to-node mesh | up    | 08:00:15 | Established |
+-----------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

查看docker网络类型

[root@etcd1 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
2db83772936d   bridge    bridge    local
3c0a5a224b09   host      host      local
422becf3aa3b   none      null      local

创建calico类型的网络，--driver calico 指定使用 calico 的 libnetwork CNM driver。 --ipam-driver calico-ipam 指定使用 calico 的 IPAM driver 管理 IP。 calico 为 global 网络，etcd 会将 calnet1 同步到所有主机。

[root@etcd1 etcd-calico]# docker network create --driver calico --ipam-driver calico-ipam calnet1
735f15b514db3a7310a7f3ef0734a6cd6b966753dc8cf0f7847305e0ba9fe51f

calico 为 global 网络，etcd 会将 calnet1 同步到所有主机。

[root@etcd1 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
2db83772936d   bridge    bridge    local
735f15b514db   calnet1   calico    global
3c0a5a224b09   host      host      local
422becf3aa3b   none      null      local

[root@etcd2 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
df0044c9f6f6   bridge    bridge    local
735f15b514db   calnet1   calico    global
03b08fa135f8   host      host      local
c19501b7ea7b   none      null      local

[root@etcd3 etcd-calico]# docker network list
NETWORK ID     NAME      DRIVER    SCOPE
331a6b638487   bridge    bridge    local
735f15b514db   calnet1   calico    global
08f90f4840c1   host      host      local
0d2160ce7298   none      null      local

6.4 使用Calico实现Docker容器跨主机互联

三个节点拉取busybox镜像用来创建容器

[root@etcd1 etcd-calico]# docker pull busybox

[root@etcd2 etcd-calico]# docker pull busybox

[root@etcd3 etcd-calico]# docker pull busybox

三个节点上都创建一个容器，指定网络类型为calnet1

[root@etcd1 etcd-calico]# docker run --name c1 --net calnet1 -itd busybox
73359e36becf9859e073ebce9370b83ac36754f40356e53b82a1e2a8cd7b0066

[root@etcd2 etcd-calico]# docker run --name c2 --net calnet1 -itd busybox
28d27f3effb0ea15e6f5e6cca9e8982c68d24f459978098967842242478b6d8b

[root@etcd3 etcd-calico]# docker run --name c3 --net calnet1 -itd busybox
995241af841f2da4f69c7c3cfa2ce0766de49e7b43ec327f5dc8d57ff7838b62

进入容器c1，查看网卡信息

[root@etcd1 etcd-calico]# docker exec -it c1 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.36.192/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit

每在主机上创建一个容器，则会在物理机上创建一张虚拟网卡出来，注意：cali5aa980fa781@if4里的if4是容器里网卡的4，cali0@if5里的5是物理机网卡的5。从这里可以看到容器里的虚拟网卡 cali0 和物理机的 cali5aa980fa781 是 veth pair 关系。

[root@etcd1 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:1e:33:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.110.133/24 brd 192.168.110.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe1e:333e/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:8b:19:bc:63 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: cali5aa980fa781@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 9a:3d:aa:d2:bc:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::983d:aaff:fed2:bca2/64 scope link 
       valid_lft forever preferred_lft forever

另外两个节点也是类似的

[root@etcd2 etcd-calico]# docker exec -it c2 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    ......
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.57.64/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit
[root@etcd2 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
   ......
       valid_lft forever preferred_lft forever
5: cali2e3a79a8486@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether ce:2a:7a:5f:4e:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::cc2a:7aff:fe5f:4e83/64 scope link 
       valid_lft forever preferred_lft forever

[root@etcd3 etcd-calico]# docker exec -it c3 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
   ......
4: cali0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.175.64/32 scope global cali0
       valid_lft forever preferred_lft forever
/ # exit
[root@etcd3 etcd-calico]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
   ......
5: califd96a41066a@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 2e:ca:96:03:96:83 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::2cca:96ff:fe03:9683/64 scope link 
       valid_lft forever preferred_lft forever

使用route -n查看路由信息：

192.168.57.64 192.168.110.131 255.255.255.192 UG 0 0 0 ens32，表示在容器里ping 192.168.57.64这个地址，都会转发到192.168.110.131这台机器 ；

192.168.57.64 0.0.0.0 255.255.255.255 UH 0 0 0 cali2e3a79a8486，表示目的地址是192.168.57.64的数据包，转发到cali2e3a79a8486这张网卡 。

cali2e3a79a8486和容器里的网卡cali0是veth pair 关系，所以就可以从容器c1访问到容器c2，其他以此类推，calico相当于建立了一个隧道，可以在物理机A的c1容器访问物理机B的c2容器。

[root@etcd1 etcd-calico]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.110.2   0.0.0.0         UG    0      0        0 ens32
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 ens32
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.36.192  0.0.0.0         255.255.255.255 UH    0      0        0 cali5aa980fa781
192.168.36.192  0.0.0.0         255.255.255.192 U     0      0        0 *
192.168.57.64   192.168.110.131 255.255.255.192 UG    0      0        0 ens32
192.168.110.0   0.0.0.0         255.255.255.0   U     0      0        0 ens32
192.168.175.64  192.168.110.132 255.255.255.192 UG    0      0        0 ens32

在容器c1里，ping c2容器可以ping通。

[root@etcd1 etcd-calico]# docker exec -it c1 sh

/ # ping 192.168.57.64
PING 192.168.57.64 (192.168.57.64): 56 data bytes
64 bytes from 192.168.57.64: seq=0 ttl=62 time=0.578 ms
64 bytes from 192.168.57.64: seq=1 ttl=62 time=0.641 ms
64 bytes from 192.168.57.64: seq=2 ttl=62 time=0.543 ms
^C
--- 192.168.57.64 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.543/0.587/0.641 ms

/ # exit

在物理机上ping不通容器c2。

[root@etcd1 etcd-calico]# ping 192.168.57.64
PING 192.168.57.64 (192.168.57.64) 56(84) bytes of data.
^C
--- 192.168.57.64 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms

看下路由的关系:不管目的地是哪里都走 cali0。

[root@etcd1 etcd-calico]# docker exec c1 ip route
default via 169.254.1.1 dev cali0 
169.254.1.1 dev cali0 scope link

看下etcd1 的路由，目的地址到 192.168.36.192 的数据包都从 cali5aa980fa781（etcd1 新产生的虚拟网卡）走，目的地址到 192.168.57.64/26 网段的数据包都从 ens32 发到 192.168.110.131 上去，每台主机都知道不同的容器在哪台主机上，所以会动态的设置路由。

[root@etcd1 etcd-calico]# ip route
default via 192.168.110.2 dev ens32 
169.254.0.0/16 dev ens32 scope link metric 1002 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
192.168.36.192 dev cali5aa980fa781 scope link 
blackhole 192.168.36.192/26 proto bird 
192.168.57.64/26 via 192.168.110.131 dev ens32 proto bird 
192.168.110.0/24 dev ens32 proto kernel scope link src 192.168.110.133 
192.168.175.64/26 via 192.168.110.132 dev ens32 proto bird

七.Kubernetes(k8s)环境里的calico

在k8s环境里，每个节点上都有calico-node,calico数据存在etcd里。

[root@k8scloude1 ~]# kubectl get pod -o wide -n kube-system 
NAME                                       READY   STATUS    RESTARTS   AGE   IP                NODE         NOMINATED NODE   READINESS GATES
calico-kube-controllers-6b9fbfff44-4jzkj   1/1     Running   55         38d   10.244.251.210    k8scloude3   <none>           <none>
calico-node-bdlgm                          1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
calico-node-hx8bk                          1/1     Running   27         38d   192.168.110.128   k8scloude3   <none>           <none>
calico-node-nsbfs                          1/1     Running   27         38d   192.168.110.129   k8scloude2   <none>           <none>
coredns-545d6fc579-7wm95                   1/1     Running   27         38d   10.244.158.121    k8scloude1   <none>           <none>
coredns-545d6fc579-87q8j                   1/1     Running   27         38d   10.244.158.122    k8scloude1   <none>           <none>
etcd-k8scloude1                            1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-apiserver-k8scloude1                  1/1     Running   18         27d   192.168.110.130   k8scloude1   <none>           <none>
kube-controller-manager-k8scloude1         1/1     Running   29         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-proxy-599xh                           1/1     Running   27         38d   192.168.110.128   k8scloude3   <none>           <none>
kube-proxy-lpj8z                           1/1     Running   27         38d   192.168.110.129   k8scloude2   <none>           <none>
kube-proxy-zxlk9                           1/1     Running   27         38d   192.168.110.130   k8scloude1   <none>           <none>
kube-scheduler-k8scloude1                  1/1     Running   29         38d   192.168.110.130   k8scloude1   <none>           <none>
metrics-server-bcfb98c76-n4fnb             1/1     Running   26         30d   10.244.251.196    k8scloude3   <none>           <none>

八.总结

本文介绍了CNI网络插件的概念和常见的几种插件对比，详细讲解了如何使用Calico实现Docker容器跨主机互联。通过使用Calico，我们可以轻松地在Kubernetes集群中实现高效的容器网络互连，提升应用程序的可靠性和可扩展性。

posted @ 2023-06-09 14:05  人生的哲理  阅读(13)  评论(1)  编辑  收藏  举报
上一篇：Quartz+SpringBoot实现定时任务（多任务，多执行时间）代码模板（直接CV即可）
下一篇：ChatGPT的原理与前端领域实践
回帖
    张三

    张三 （王者 段位）

    921 积分 (2)粉丝 (41)源码

     

    温馨提示

    亦奇源码

    最新会员