跳到主要内容

KubeEdge meets Cilium !!!

· 阅读需 10 分钟
Tomoya Fujita

This blog introduces how to enable Cilium Container Network Interface with KubeEdge.

Why Cilium for KubeEdge​

Cilium is the one of the most advanced and efficient container network interface plugin for Kubernetes, that provides network connectivity and security for containerized applications in Kubernetes clusters. It leverages eBPF (extended Berkeley Packet Filter) technology to implement networking and security policies at the Linux kernel level, allowing for high-performance data plane operations and fine-grained security controls.

And KubeEdge extends the cluster orchestration capability down to edge environments to provide unified cluster management and sophisticated edge specific features.

Enabling Cilium with KubeEdge allows us to take advantage of both benefits even for edge computing environments. We can deploy the application containers where EdgeCore running and bind Cilium to connect with workloads in the cloud infrastructure. This is because Cilium can also enable WireGuard VPN with transparent encryption of traffic between Cilium-managed endpoints.

Further more, we can also rely on Cilium Tetragon Security Observability and Runtime Enforcement to confine security risk and vulnerability in edge environment.

How to enable Cilium with KubeEdge​

The following procedures to set up a simple cluster system with Kubernetes and KubeEdge with Cilium. Since this is a new approach and still beta phase, the following manual operations are required to do so.

After all the operations, we can develop the following cluster configuration with KubeEdge with Cilium.

overview

Prerequisites​

  • KubeEdge Release v1.16 or later required.

    To enable Cilium with KubeEdge, we must use KubeEdge Release v1.16 or later. This is because that cilium-agent needs to issue InClusterConfig APIs to Kubernetes API server to configure cilium-agent. This should be no problem with Kubernetes nodes, but with KubeEdge those API requests and responses need to be bypassed via KubeEdge MetaManager. You can see KubeEdge EdgeCore supports Cilium CNI for more details.

  • Compatible Kubernetes version with KubeEdge Release v1.16.

    You can find compatible and supported Kubernetes version here.

  • It Requires super user rights (or root rights) to run commands.

Kubernetes Master Setup​

Refer to KubeEdge Setup Prerequisites, and set up the Kubernetes API server as following.

### Check node status
> kubectl get nodes -o wide
NAME           STATUS   ROLES           AGE   VERSION    INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
tomoyafujita   Ready    control-plane   25s   v1.26.15   AA.BBB.CCC.DD   <none>        Ubuntu 20.04.6 LTS   5.15.0-102-generic   containerd://1.6.32

### Taint this node so that CloudCore can be deployed on the control-plane
> kubectl taint node tomoyafujita node-role.kubernetes.io/control-plane:NoSchedule-
node/tomoyafujita untainted
> kubectl get nodes -o json | jq '.items[].spec.taints'
null

Cilium Install and Setup​

Refer to Cilium Quick Installation, install and set up cilium deployments in the cluster.

> cilium version
cilium-cli: v0.16.9 compiled with go1.22.3 on linux/amd64
cilium image (default): v1.15.5
cilium image (stable): v1.15.5
cilium image (running): unknown. Unable to obtain cilium version. Reason: release: not found

and install Cilium with enabling wireguard VPN in the cluster,

> cilium install --set encryption.enabled=true --set encryption.type=wireguard --set encryption.wireguard.persistentKeepalive=60
...

> cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

Deployment             cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet              cilium             Desired: 1, Ready: 1/1, Available: 1/1
Containers:            cilium             Running: 1
                       cilium-operator    Running: 1
Cluster Pods:          1/2 managed by Cilium
Helm chart version:
Image versions         cilium             quay.io/cilium/cilium:v1.15.5@sha256:4ce1666a73815101ec9a4d360af6c5b7f1193ab00d89b7124f8505dee147ca40: 1
                       cilium-operator    quay.io/cilium/operator-generic:v1.15.5@sha256:f5d3d19754074ca052be6aac5d1ffb1de1eb5f2d947222b5f10f6d97ad4383e8: 1

Add nodeAffinity for Cilium DaemonSet to make sure these pods are only created on cloud nodes. These Cilium pods are generic Cilium DaemonSet, so supposed to be running on cloud nodes but where EdgeCore running.

### Edit Cilium DaemonSet with the following patch
>  kubectl edit ds -n kube-system cilium
diff --git a/cilium-kubelet.yaml b/cilium-kubelet.yaml
index 21881e1..9946be9 100644
--- a/cilium-kubelet.yaml
+++ b/cilium-kubelet.yaml
@@ -29,6 +29,12 @@ spec:
         k8s-app: cilium
     spec:
       affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                - key: node-role.kubernetes.io/edge
+                  operator: DoesNotExist
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:

After editing, Cilium pods will be restarted.

KubeEdge CloudCore Setup​

1st of all, we need to install Keadm with the official procedure Installing KubeEdge with Keadm.

In this blog, we use Keadm v1.16.1 as following.

### Install v1.16.1 keadm command
> wget https://github.com/kubeedge/kubeedge/releases/download/v1.16.1/keadm-v1.16.1-linux-amd64.tar.gz
> tar -zxvf keadm-v1.16.1-linux-amd64.tar.gz
> cp keadm-v1.16.1-linux-amd64/keadm/keadm /usr/local/bin

> keadm version
version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"bd7b42acbfbe3a453c7bb75a6bb8f1e8b3db7415", GitTreeState:"clean", BuildDate:"2024-03-27T02:57:08Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

and then, start the CloudCore with v1.16.1.

> keadm init --advertise-address="AA.BBB.CCC.DD" --profile version=v1.16.1 --kube-config=/root/.kube/config
Kubernetes version verification passed, KubeEdge installation will start...
CLOUDCORE started
=========CHART DETAILS=======
NAME: cloudcore
LAST DEPLOYED: Tue Jun  4 17:19:15 2024
NAMESPACE: kubeedge
STATUS: deployed
REVISION: 1

After CloudCore is started, we also need to enable DynamicControllers.

### edit ConfigMap of CloudCore to enable dynamicController
> kubectl edit cm -n kubeedge cloudcore
> kubectl delete pod -n kubeedge --selector=kubeedge=cloudcore

### Check ConfigMap
> kubectl get cm -n kubeedge cloudcore -o yaml | grep "dynamicController" -A 1
      dynamicController:
        enable: true

To manage and handle the APIs from MetaManager that are originally come from Cilium running in the edge nodes, we need to give access permission to CloudCore by editing clusterRole and clusterRolebinding.

clusterRole:

### Edit and apply the following patch
> kubectl edit clusterrole cilium
diff --git a/cilium-clusterrole.yaml b/cilium-clusterrole.yaml
index 736e35c..fd5512e 100644
--- a/cilium-clusterrole.yaml
+++ b/cilium-clusterrole.yaml
@@ -66,6 +66,7 @@ rules:
   verbs:
   - list
   - watch
+  - get
 - apiGroups:
   - cilium.io
   resources:

clusterRolebinding:

### Edit and apply the following patch
> kubectl edit clusterrolebinding cilium
diff --git a/cilium-clusterrolebinding.yaml b/cilium-clusterrolebinding.yaml
index 9676737..ac956de 100644
--- a/cilium-clusterrolebinding.yaml
+++ b/cilium-clusterrolebinding.yaml
@@ -12,3 +12,9 @@ subjects:
 - kind: ServiceAccount
   name: cilium
   namespace: kube-system
+- kind: ServiceAccount
+  name: cloudcore
+  namespace: kubeedge
+- kind: ServiceAccount
+  name: cloudcore
+  namespace: default

Finally, we get the token after CloudCore has restarted.

> keadm gettoken
<TOKEN_HASH>

KubeEdge EdgeCore Setup​

With the token provided above, we can start the EdgeCore to join the cluster system.

> keadm join --cloudcore-ipport=AA.BBB.CCC.DD:10000 --kubeedge-version=v1.16.1 --cgroupdriver=systemd --token <TOKEN_HASH>
...<snip>
I0604 21:36:31.040859 2118064 join_others.go:265] KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -xe
I0604 21:36:41.050154 2118064 join.go:94] 9. Install Complete!

> systemctl status edgecore
● edgecore.service
     Loaded: loaded (/etc/systemd/system/edgecore.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-06-04 21:36:31 PDT; 40s ago
   Main PID: 2118341 (edgecore)
      Tasks: 24 (limit: 18670)
     Memory: 31.8M
        CPU: 849ms
     CGroup: /system.slice/edgecore.service
             └─2118341 /usr/local/bin/edgecore

After EdgeCore is started, we need to enable ServiceBus and MetaServer to edit edgecore.yaml.

### Edit and apply the following patch
> vi /etc/kubeedge/config/edgecore.yaml

### Restart edgecore systemd-service
> systemctl restart edgecore
diff --git a/edgecore.yaml b/edgecore.yaml
index 8d17418..5391776 100644
--- a/edgecore.yaml
+++ b/edgecore.yaml
@@ -62,6 +62,8 @@ modules:
       cgroupDriver: systemd
       cgroupsPerQOS: true
       clusterDomain: cluster.local
+      clusterDNS:
+      - 10.96.0.10
       configMapAndSecretChangeDetectionStrategy: Get
       containerLogMaxFiles: 5
       containerLogMaxSize: 10Mi
@@ -151,7 +151,7 @@ modules:
     enable: true
     metaServer:
       apiAudiences: null
-      enable: false
+      enable: true
       server: 127.0.0.1:10550
       serviceAccountIssuers:
       - https://kubernetes.default.svc.cluster.local
@@ -161,7 +161,7 @@ modules:
       tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
     remoteQueryTimeout: 60
   serviceBus:
-    enable: false
+    enable: true
     port: 9060
     server: 127.0.0.1
     timeout: 60

And then, we need to create another DaemonSet of cilium-agent only for EdgeCore nodes. We need to deploy cilium-agent pods to the nodes where KubeEdge EdgeCore runs and labeled with kubernetes.io/edge=. Besides that, cilium-agent needs to query the APIs to MetaServer instead of Kubernetes API-server that is required to keep the edge autonomy provided by KubeEdge.

### Dump original Cilium DaemonSet configuration
> kubectl get ds -n kube-system cilium -o yaml > cilium-edgecore.yaml

### Edit and apply the following patch
> vi cilium-edgecore.yaml

### Deploy cilium-agent aligns with edgecore
> kubectl apply -f cilium-edgecore.yaml
diff --git a/cilium-edgecore.yaml b/cilium-edgecore.yaml
index bff0f0b..3d941d1 100644
--- a/cilium-edgecore.yaml
+++ b/cilium-edgecore.yaml
@@ -8,7 +8,7 @@ metadata:
     app.kubernetes.io/name: cilium-agent
     app.kubernetes.io/part-of: cilium
     k8s-app: cilium
-  name: cilium
+  name: cilium-kubeedge
   namespace: kube-system
 spec:
   revisionHistoryLimit: 10
@@ -29,6 +29,12 @@ spec:
         k8s-app: cilium
     spec:
       affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+              - matchExpressions:
+                - key: node-role.kubernetes.io/edge
+                  operator: Exists
         podAntiAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
           - labelSelector:
@@ -39,6 +45,8 @@ spec:
       containers:
       - args:
         - --config-dir=/tmp/cilium/config-map
+        - --k8s-api-server=127.0.0.1:10550
+        - --auto-create-cilium-node-resource=true
         - --debug
         command:
         - cilium-agent
@@ -178,7 +186,9 @@ spec:
       dnsPolicy: ClusterFirst
       hostNetwork: true
       initContainers:
-      - command:
+      - args:
+        - --k8s-api-server=127.0.0.1:10550
+        command:
         - cilium
         - build-config
         env:

We can see below that cilium-pq45v (normal cilium-agent pod) running on cloud node, and cilium-kubeedge-kkb7z (edgecore specific DaemonSet) is running with edgecore.

> kubectl get pods -A -o wide
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES
kube-system   cilium-kubeedge-kkb7z                  1/1     Running   0          32s     43.135.146.155   edgemaster     <none>           <none>
kube-system   cilium-operator-fdf6bc9f4-445p6        1/1     Running   0          3h40m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   cilium-pq45v                           1/1     Running   0          3h32m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   coredns-787d4945fb-2bbdf               1/1     Running   0          8h      10.0.0.104       tomoyafujita   <none>           <none>
kube-system   coredns-787d4945fb-nmd2p               1/1     Running   0          8h      10.0.0.130       tomoyafujita   <none>           <none>
kube-system   etcd-tomoyafujita                      1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-apiserver-tomoyafujita            1/1     Running   1          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-controller-manager-tomoyafujita   1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-proxy-qmxqp                       1/1     Running   0          19m     43.135.146.155   edgemaster     <none>           <none>
kube-system   kube-proxy-v2ht7                       1/1     Running   0          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kube-system   kube-scheduler-tomoyafujita            1/1     Running   1          8h      AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kubeedge      cloudcore-df8544847-6mlw2              1/1     Running   0          4h23m   AA.BBB.CCC.DD    tomoyafujita   <none>           <none>
kubeedge      edge-eclipse-mosquitto-9cw6r           1/1     Running   0          19m     43.135.146.155   edgemaster     <none>           <none>

Check Cilium Connectivity from Pods​

Now Cilium is ready to be used for application pods and containers to provide network connectivity. We can use busybox DaemonSet as following to try ping the network connectivity via Cilium.

> cat busybox.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - image: busybox
        command: ["sleep", "3600"]
        imagePullPolicy: IfNotPresent
        name: busybox

> kubectl apply -f busybox.yaml
daemonset.apps/busybox created

> kubectl get pods -o wide
NAME            READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
busybox-mn98w   1/1     Running   0          84s   10.0.0.58    tomoyafujita   <none>           <none>
busybox-z2mbw   1/1     Running   0          84s   10.0.1.121   edgemaster     <none>           <none>

> kubectl exec --stdin --tty busybox-mn98w -- /bin/sh
/ #
/ # ping 10.0.1.121
PING 10.0.1.121 (10.0.1.121): 56 data bytes
64 bytes from 10.0.1.121: seq=0 ttl=63 time=1.326 ms
64 bytes from 10.0.1.121: seq=1 ttl=63 time=1.620 ms
64 bytes from 10.0.1.121: seq=2 ttl=63 time=1.341 ms
64 bytes from 10.0.1.121: seq=3 ttl=63 time=1.685 ms
^C
--- 10.0.1.121 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 1.326/1.493/1.685 ms
/ # exit
> kubectl exec --stdin --tty busybox-z2mbw -- /bin/sh
/ #
/ # ping 10.0.0.58
PING 10.0.0.58 (10.0.0.58): 56 data bytes
64 bytes from 10.0.0.58: seq=0 ttl=63 time=0.728 ms
64 bytes from 10.0.0.58: seq=1 ttl=63 time=1.178 ms
64 bytes from 10.0.0.58: seq=2 ttl=63 time=0.635 ms
64 bytes from 10.0.0.58: seq=3 ttl=63 time=1.152 ms
^C
--- 10.0.0.58 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.635/0.923/1.178 ms

Finally we can confirm the cross-communication via Cilium between busybox containers are just fine!!!

If you want to know more technical insights and development, see and subscribe KubeEdge EdgeCore supports Cilium CNI for more details.