마스터

1) 초기 세팅

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
BOOTPROTO=none
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.108.3
PREFIX=24
GATEWAY=192.168.108.2
EOF

cat << EOF > /etc/resolv.conf
search labs.local
nameserver 192.168.108.2
EOF

cat << EOF > /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.108.3	control
192.168.108.4	master
192.168.108.5	slave1
192.168.108.6	slave2


hostnamectl set-hostname control
yum -y update
yum -y install yum-utils git net-tools bind-utils iptables-services bash-completion kexec-tools wget vim
vi /etc/selinux/config ## Selinux 종료 disable
vi /etc/fstab ## 맨 밑 Swap 메모리 주석처리
systemctl disable --now firewalld
reboot

2) 쿠버네티스 설치

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF


sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF


sudo sysctl --system

sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine


sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin

sudo systemctl daemon-reload
sudo systemctl enable containerd --now


cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

'''
위가 안되면 아래로 설치
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
'''




sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
rm -rf /etc/containerd/config.toml
systemctl restart containerd.service

3) [마스터만] - flannet

kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml


source <(kubectl completion bash) 
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "alias k=kubectl" >> ~/.bashrc
echo "complete -F __start_kubectl k" >> ~/.bashrc
source ~/.bashrc

systemctl status kubelet

# 안되면
source <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrcsource <(kubectl completion bash) # set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo "source <(kubectl completion bash)" >> ~/.bashrc

슬레이브

1) 초기 세팅

cat << EOF > /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet
BOOTPROTO=none
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.108.4
PREFIX=24
GATEWAY=192.168.108.2
EOF

cat << EOF > /etc/resolv.conf
search labs.local
nameserver 192.168.108.2
EOF

cat << EOF > /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.108.3	control
192.168.108.4	master
192.168.108.5	slave1
192.168.108.6	slave2

hostnamectl set-hostname node
yum -y update
yum -y install yum-utils git net-tools bind-utils iptables-services bash-completion kexec-tools wget vim
vi /etc/selinux/config ## Selinux 종료
vi /etc/fstab ## 맨 밑 Swap 메모리 주석처리
systemctl disable --now firewalld
reboot

2) 설치

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sudo sysctl --system

sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine


sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce docker-ce-cli containerd.io docker-compose-plugin

sudo systemctl daemon-reload
sudo systemctl enable containerd --now



cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

'''
위가 안되면 아래로 설치
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF
'''




sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet
rm -rf /etc/containerd/config.toml
systemctl restart containerd.service

3) Master 노드에서 생성된 토큰을 넘겨준다

주의~~~~~~~  ## kubeadm init 이후 kubeadm join [Master IP]:[Port]  --token .... 등과 같은 Config가 뜨는데 Worker Node에 붙여넣으면 된다

kubeadm join 192.168.108.3:6443 --token nb3xzc.x55vah76d1 \
--discovery-token-ca-cert-hash sha256:6bf6ac0e19e62f77a69fc0c21fda2507b5bca5b4fc1a

~~~~~~~~~~~


systemctl status kubelet

control plane

1) 마스터에 있는 해당 내용을 control plane으로 던져줘야함

cat /etc/kubernetes/admin.conf # 마스터에 있으며 control에 내용 복사
sudo cat /etc/kubernetes/admin.conf > $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf

2) kubectl 설치

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client --output=yaml

3) 동작확인

```````````````
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin
alias kubectl='kubectl --kubeconfig="$HOME/admin.conf"'
alias k=kubectl

export PATH

````````````````````

1. 워커 노드 2개 사용 시 둘 중 하나만 될 땐 워커노드 쪽도 kubeadm reset을 해줘야한다.

2. (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1ba776a55cc5ab1158d8c3cb1a6d6cbedb5d10ec8aaca666084cb2ca40458d03": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.244.2.1/24 에러 해결해줘


ChatGPT
이 오류는 Kubernetes에서 사용하는 CNI(Container Network Interface) 플러그인인 Flannel이 네트워크 설정을 실패하여 발생한 것으로 보입니다. cni0라는 이름의 인터페이스가 이미 다른 IP 주소(10.244.2.1/24)를 가지고 있어서 문제가 발생한 것으로 보입니다.

다음은 이 문제를 해결하기 위한 몇 가지 단계입니다:

cni0 인터페이스를 확인하고 해당 IP 주소를 가진 프로세스를 종료합니다. 다음 명령을 실행하세요:명령을 실행한 후에 cni0 인터페이스에 대한 정보가 표시됩니다. 그리고 다음 명령을 사용하여 해당 인터페이스를 종료합니다:
perlCopy code
sudo ip link delete cni0
bashCopy code
ip link show cni0
Flannel CNI 플러그인을 다시 시작합니다. 다음 명령을 실행하세요:
Copy code
sudo systemctl restart kubelet
다시 Pod를 생성하고 네트워크 설정이 올바르게 수행되는지 확인합니다. 예를 들어, 다음 명령을 사용하여 Nginx Pod를 생성해 보세요:Pod가 성공적으로 생성되었는지 확인하고, kubectl get pods 명령을 사용하여 Pod의 상태를 확인하세요.
arduinoCopy code
kubectl run nginx --image=nginx
위의 단계를 따라 시도해 보시고, 문제가 지속되는 경우 추가적인 로그 또는 오류 메시지를 제공해 주시면 더 자세한 도움을 드릴 수 있습니다.