[draft] 쿠버네티스 클러스터를 구현하는 방법

쿠버네티스 클러스터(Kubernetes Cluster)를 구현하는 방법

쿠버네티스 클러스터를 멀티 마스터 환경에서 고가용성(HA)을 구현하는 것은 클러스터의 Control Plane을 여러 마스터 노드로 분산하여 단일 장애 지점을 없애는 중요한 작업입니다. 이를 통해 클러스터의 신뢰성과 가용성을 높일 수 있습니다.

테스트 환경

Hostname	IP Address	패키지	비고
k8s-lb1	192.168.0.130	haproxy
k8s-master1	192.168.0.131	kubelet kubeadm kubectl
k8s-master2	192.168.0.132	kubelet kubeadm kubectl
k8s-master3	192.168.0.111	kubelet kubeadm kubectl

1. 로드 밸런서 HAProxy 설정

로드 밸런서 : 클러스터 API 서버에 대한 요청을 분산시키기 위해 로드 밸런서를 설정합니다. 모든 마스터 노드의 API 서버 IP를 로드 밸런서에 등록합니다.

---

sudo add-apt-repository -y ppa:vbernat/haproxy-3.0

sudo apt-get install haproxy=3.0.\*

sudo systemctl --now enable haproxy

sudo vim /etc/haproxy/haproxy.cfg

# HAProxy configuration file
global
    log 127.0.0.1   local0
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

# Kubernetes API Server Frontend
frontend kubernetes
    bind 0.0.0.0:6443
    mode tcp
    option tcplog
    default_backend kubernetes-master-nodes

# Backend for Kubernetes Master Nodes
backend kubernetes-master-nodes
    mode tcp
    option tcp-check
    balance roundrobin
    server master1 192.168.0.131:6443 check fall 3 rise 2
    server master2 192.168.0.132:6443 check fall 3 rise 2
    server master3 192.168.0.111:6443 check fall 3 rise 2

# Statistics interface
listen stats
    bind 192.168.0.130:8888
    mode http
    stats enable
    stats uri /
    stats refresh 10s
    stats realm HAProxy\ Statistics
    stats auth admin:admin

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

HAProxy 재시작

sudo systemctl restart haproxy

sudo systemctl status haproxy --no-pager -l

---

2. CRI와 Kubernetes 패키지 설치

모든 마스터 노드에서 CRI와 Kubernetes 패키지를 설치합니다.

---

Swap 비활성화

swapoff -a

방화벽 설정

ufw allow 6443/tcp  # API 서버 포트
ufw allow 2379:2380/tcp  # etcd 포트
ufw allow 10250/tcp  # Kubelet 포트
ufw allow 10251/tcp  # kube-scheduler 포트
ufw allow 10252/tcp  # kube-controller-manager 포트
ufw allow 30000:32767/tcp  # NodePort 포트

NTP 설정

sudo apt-get install -y ntp
sudo systemctl --now enable ntp
ntpq -p

컨테이너 런타임(containerd) 설정

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg2 software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key --keyring /etc/apt/trusted.gpg.d/docker.gpg add -
echo "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y containerd.io
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml
sudo systemctl enable containerd
sudo systemctl restart containerd

Kubernetes 패키지 설치 및 환경 설정

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
KUBERNETES_VERSION="v1.30"
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

---

3. Kubernetes 클러스터 초기화

클러스터 초기화

• 첫 번째 마스터 노드에서 클러스터를 초기화합니다.

sudo kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--control-plane-endpoint "Load Balancer IP Address:6443" \
--upload-certs

sudo kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--control-plane-endpoint 192.168.0.130:6443 \
--upload-certs | tee $HOME/kubeadm_init_output.log

• <LOAD_BALANCER_DNS> : API 서버에 접근할 수 있는 로드 밸런서의 DNS 또는 IP
• <PORT> : 기본 포트는 6443입니다.

...
You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.0.130:6443 --token tsn8fn.7igyens2natcr4bh \
        --discovery-token-ca-cert-hash sha256:0934c0528df98ced02e3b13fe0b93c8439b7a2f92a3306dc523754ddab485205 \
        --control-plane --certificate-key 52e3cd6afc4e9409110cb9cbb1ea3795c1e48a21b4bff59ecdcde6dd29a62ec1

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.130:6443 --token tsn8fn.7igyens2natcr4bh \
        --discovery-token-ca-cert-hash sha256:0934c0528df98ced02e3b13fe0b93c8439b7a2f92a3306dc523754ddab485205

kubeconfig 설정

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

클러스터 상태 확인

kubectl get nodes

NAME          STATUS     ROLES           AGE   VERSION
k8s-master1   NotReady   control-plane   41m   v1.30.3

4. 다른 마스터 노드 연결

다른 마스터를 클러스터에 추가합니다.

  kubeadm join 192.168.0.130:6443 \
  --token tsn8fn.7igyens2natcr4bh \
  --discovery-token-ca-cert-hash sha256:0934c0528df98ced02e3b13fe0b93c8439b7a2f92a3306dc523754ddab485205 \
  --control-plane --certificate-key 52e3cd6afc4e9409110cb9cbb1ea3795c1e48a21b4bff59ecdcde6dd29a62ec1

• <TOKEN> : 첫 번째 마스터에서 생성한 토큰
• <HASH> : 인증서 해시
• <CERTIFICATE_KEY> : 인증서 키

...
This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

kubeconfig 설정

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

클러스터 상태 확인

kubectl get nodes

NAME          STATUS     ROLES           AGE     VERSION
k8s-master1   NotReady   control-plane   47m     v1.30.3
k8s-master2   NotReady   control-plane   3m9s    v1.30.3
k8s-master3   NotReady   control-plane   2m25s   v1.30.3

5. 네트워크 플러그인 설치

Calico 네트워크 플러그인을 설치합니다.

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

클러스터 확인

kubectl get nodes

NAME          STATUS   ROLES           AGE   VERSION
k8s-master1   Ready    control-plane   58m   v1.30.3
k8s-master2   Ready    control-plane   14m   v1.30.3
k8s-master3   Ready    control-plane   13m   v1.30.3

클러스터 파드 확인

kubectl get pods --all-namespaces

pods

 

참고URL

- Kubernetes Documentation : Creating Highly Available Clusters with kubeadm

- Kubernetes Documentation : Kubernetes Components

- Kubernetes Documentation : Options for Highly Available Topology

- Kubernetes Components : 포트와 프로토콜