[draft] 우분투에서 쿠버네티스 클러스터를 구성하는 방법

우분투에서 쿠버네티스 클러스터를 구성하는 방법

각 노드에는 kubeadm, kubelet, kubectl을 설치하고, control plane 노드에서는 Kubernetes API 서버 및 다른 구성 요소를 초기화합니다.

모든 노드(control plane, knode1, knode2, knode3)

• swap 메모리 비활성화

sudo swapoff -a

sudo sed -i '/\s*swap\s*/ s/^/#/' /etc/fstab

sudo swapon --show

• 커널 모듈 설정(overlay 모듈, br_netfilter 모듈)

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay

sudo modprobe br_netfilter

sudo lsmod | egrep '^overlay|^br_netfilter'

$ sudo lsmod | egrep '^overlay|^br_netfilter'
br_netfilter           32768  0
overlay               151552  0

• sysctl 파라미터 설정

sudo sed -i 's/^#\(net\.ipv4\.ip_forward\s*=\s*1\)/net.ipv4.ip_forward = 1/' /etc/sysctl.conf

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

sudo sysctl -a | egrep 'net.ipv4.ip_forward|net.bridge.bridge-nf-call-iptables|net.bridge.bridge-nf-call-ip6tables'

$ sudo sysctl -a | egrep 'net.ipv4.ip_forward|net.bridge.bridge-nf-call-iptables|net.bridge.bridge-nf-call-ip6tables'
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

컨테이너 런타임(ContainerD) 설치

sudo apt-get update

필요한 패키지 설치

sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common

GPG 키 추가

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -

리포지토리 추가

echo "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

containerd 설치

sudo apt-get update

sudo apt-get install -y containerd.io

containerd 버전 확인

sudo containerd --version

containerd 설정 편집

sudo containerd config default > /etc/containerd/config.toml

sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

또는

sudo vim /etc/containerd/config.toml

• SystemdCgroup = false -> SystemdCgroup = true

config.toml_SystemdCgroup

containerd 서비스 시작 및 부팅 시 활성화

sudo systemctl --now enable containerd.service

sudo systemctl restart containerd.service

sudo systemctl status containerd.service

containerd socket 확인

ls -l /var/run/containerd/containerd.sock

728x90

kubeadm, kubelet 및 kubectl 설치

sudo apt-get update

sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

 

apt-cache policy kubeadm | egrep "Installed|Candidate"

apt-cache policy kubectl | egrep "Installed|Candidate"

apt-cache policy kubectl | egrep "Installed|Candidate"

kubeadm, kubelet 및 kubectl 설치

sudo apt-get install -y kubelet=<VERSION> kubeadm=<VERSION> kubectl=<VERSION>

• control-plane

sudo apt-get install -y kubelet kubeadm kubectl

• node

sudo apt-get install -y kubelet kubeadm

sudo apt-mark hold kubelet kubeadm kubectl

kubelet 서비스 시작 및 부팅 시 활성화

sudo systemctl daemon-reload

sudo systemctl --now enable kubelet.service

sudo systemctl restart kubelet.service

sudo systemctl status kubelet.service

kubeadm, kubelet 및 kubectl 버전 확인

kubeadm version

kubelet --version

kubectl version --client

control plane 노드

1. Kubernetes 클러스터 초기화

<control-plane-node-ip>를 control plane 노드의 IP 주소로 바꿔주어야 합니다.

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=<control-plane-node-ip>

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=$(hostname -I | awk '{print $1}')

sudo kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.56.51 \
--cri-socket=unix:///var/run/containerd/containerd.sock \
2>&1 | tee /root/kubeadm_init_output.txt

---

$ sudo kubeadm init \
> --pod-network-cidr=10.244.0.0/16 \
> --apiserver-advertise-address=192.168.56.51 \
> --cri-socket=unix:///var/run/containerd/containerd.sock \
> 2>&1 | tee /root/kubeadm_init_output.txt
I0216 16:10:07.841837   16039 version.go:256] remote version is much newer: v1.29.2; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.7
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0216 16:10:09.015902   16039 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.51]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [control-plane localhost] and IPs [192.168.56.51 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [control-plane localhost] and IPs [192.168.56.51 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.003624 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node control-plane as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node control-plane as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: t2qibi.49a4d4we27crsnns
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.51:6443 --token t2qibi.49a4d4we27crsnns \
        --discovery-token-ca-cert-hash sha256:d98a8bb1b2f0eb5b92970d26ecdb0d1bf2d7abce0280dd49518022174a5ea009

---

2. 클러스터 설정

kubeadm init 명령이 완료되면, 출력에 나오는 명령어를 따라서 control plane 노드에서 실행하여 Kubernetes 클러스터를 설정합니다.

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.51:6443 --token t2qibi.49a4d4we27crsnns \
        --discovery-token-ca-cert-hash sha256:d98a8bb1b2f0eb5b92970d26ecdb0d1bf2d7abce0280dd49518022174a5ea009

루트(non-root)가 아닌 사용자

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

root 사용자

export KUBECONFIG=/etc/kubernetes/admin.conf

 

sudo kubectl get nodes

get_pods

파드 네트워크 애드온 설치(CNI 추가)

sudo kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

$ sudo kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
poddisruptionbudget.policy/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
serviceaccount/calico-cni-plugin created
configmap/calico-config created
...
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created

sudo kubectl get pods --all-namespaces

get_nodes2

sudo kubectl get nodes -o wide

$ sudo kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
control-plane   Ready    control-plane   6m6s   v1.28.2   192.168.56.51   <none>        Ubuntu 22.04.2 LTS   5.15.0-94-generic   containerd://1.6.28

knode1, knode2, knode3 노드

apiserver 서비스 포트 확인(TCP 6443)

nc -zv 192.168.56.51 6443

$ nc -zv 192.168.56.51 6443
Connection to 192.168.56.51 6443 port [tcp/*] succeeded!

1. 클러스터 조인

• control plane 노드에서 초기화된 Kubernetes 클러스터에 다른 노드를 조인시킵니다.

sudo kubeadm join <control-plane-node-ip>:<control-plane-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>

kubeadm join 192.168.56.51:6443 --token 70fmb7.reuhknci6hyayw17 \
        --discovery-token-ca-cert-hash sha256:fafd6bf64e773a68f316dc3c19ddaa925757728b77836db59aeb98b4c732f705

---

$ kubeadm join 192.168.56.51:6443 --token t2qibi.49a4d4we27crsnns \
>         --discovery-token-ca-cert-hash sha256:d98a8bb1b2f0eb5b92970d26ecdb0d1bf2d7abce0280dd49518022174a5ea009
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

---

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

2. 클러스터 상태 확인

• control plane 노드에서 다음 명령어를 사용하여 모든 노드가 올바르게 조인되었는지 확인합니다.

sudo kubectl get nodes

get_nodes3

 

참고URL

- docker docs : Install Docker Engine on Ubuntu

- kubernetes document : kubeadm, kubelet 및 kubectl 설치

- kubernetes document : Installing kubeadm