centos7 通过kubeadm 安装 k8s v1.28 集群

全文预览

01.准备工作： 
     配置yum源，repo
     安装常用工具，同步时间
     关闭防火墙，关闭selinux，关闭swap
     加载ipvs模块，优化内核

02.在所有机器上安装docker ,配置containerd 中pause的镜像地址

03.在所有机器上安装kubeadm,kubelet,kubectl

04.部署集群master节点

05.让工作节点加入

06.安装CNI网络插件

0.系统要求

1.Linux 内核操作系统

2.各机器最低配置为2CPu ,2GB

3.集群机器之间网络互通

4.每个节点的主机名，mac地址，product_uuid 得唯一

5.部署时需要能连外网

查看IP和MAC命令

ip link

ifconfig -a

查看product_uuid

sudo cat /sys/class/dmi/id/product_uuid # 查看product_uuid

sudo cat /sys/class/dmi/id/product_uuid

1.准备工作

配置yum源，repo

# yum源
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo

# docker repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# kubernetes repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 缓存
yum clean all && yum makecache

安装常用包，同步时间

yum -y install tree vim wget bash-completion bash-completion-extras lrzsz net-tools sysstat iotop iftop htop unzip nc nmap telnet bc  psmisc httpd-tools ntpdate

# 时区修改,如果/etc/localtime有软连接,不是Shanghai,可以直接删除,在软链接
ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ntpdate ntp2.aliyun.com            # 同步阿里云服务器上的时间.
/sbin/hwclock --systohc            # 写入到bios系统

关闭防火墙，将selinux关闭，关闭swap

#关闭防火墙
systemctl stop firewalld  && systemctl disable firewalld

#关闭selinux
setenforce 0
sed -i  's/enforcing/disabled/' /etc/selinux/config

# 临时关闭swap。如果不关闭，kubelet会启动失败
sudo swapoff -a
# 永久防止开机自动挂载swap
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

加载ipvs模块,优化内核

cat > /etc/sysconfig/modules/ipvs.modules  << EOF
#!/bin/bash
modprobe br_netfilter
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack_ipv4
EOF

chmod 755  /etc/sysconfig/modules/ipvs.modules
bash  /etc/sysconfig/modules/ipvs.modules

#验证ipvs模块
lsmod  |grep ip_vs


# 内核文件 
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.max_map_count=262144
EOF

# 生效并验证内核优化
sysctl -p /etc/sysctl.d/k8s.conf

2.安装docker以及配置pause镜像地址

在所有机器执行

2.1 安装docker

# yum 安装
yum install docker-ce

# 配置 daemon.json
cat <<EOF > /etc/docker/daemon.json
{
    "registry-mirrors": [
        "https://registry.hub.docker.com",
        "http://hub-mirror.c.163.com",
        "https://docker.mirrors.ustc.edu.cn",
        "https://registry.docker-cn.com"
    ]
} 
EOF


# 启动docker并设置开机自启
systemctl enable docker   && systemctl start docker

2.2 配置containerd中pause镜像地址

## 如果没有/etc/containerd/config.toml文件，将默认配置导出到/etc/containerd/config.toml。
containerd config default > /etc/containerd/config.toml

## 修改配置文件/etc/containerd/config.toml， 更改sandbox_image配置
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

## PS: 如果生成的/etc/containerd/config.toml中没有如上配置项，可以运行如下命令导出当前所有配置项后再修改文件/etc/containerd/config.toml
# containerd config dump > /etc/containerd/config.toml

## 重启containerd
systemctl restart containerd

3.安装kubeadm,kubelet,kubectl

kubeadm：启动k8s集群的工具

kubelet: 该组件在集群中的所有机器上运行，并执行启动pod和容器之类的任务。

kubectl: 与集群通信的工具。可以只在master节点上安装。

# 安装kubeadm、kubectl、kubelet
yum install -y kubectl kubeadm kubelet

# 启动kubelet服务
systemctl enable kubelet && systemctl start kubelet

# 查看kubeadm版本
kubeadm version

4.部署集群master节点

在master 节点执行

4.1 更改配置文件

kubeadm init命令用于初始化master节点。kubeadm init 的参数可以通过命令行或yaml文件进行配置。本文介绍如何通过yaml文件进行配置。可以通过kubeadm config print init-defaults命令得到一份默认配置，然后对其进行修改。

 kubeadm config print init-defaults > kubeadm.yaml
对kubeadm.yaml进行编辑，修改内容如下：

修改advertiseAddress为master IP地址
imageRepository修改为registry.aliyuncs.com/google_containers，防止镜像拉不下来
建议将networking.podSubnet修改为10.244.0.0/16， 和后续安装的flannel CNI 插件的默认配置保持一致。

---
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.246.133    ## change the IP of apiserver.
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: node
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers   ## change imageRepository to aliyun.
kind: ClusterConfiguration
kubernetesVersion: 1.28.0
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16  ## add this line to config POD network. Same with CNI config.
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
如上配置文件等价于命令行：

kubeadm init \
  --apiserver-advertise-address=192.168.200.133 \
  --image-repository registry.aliyuncs.com/google_containers \
  --kubernetes-version v1.28.0 \
  --service-cidr=10.96.0.0/12 \
  --pod-network-cidr=10.244.0.0/16 
–apiserver-advertise-address 集群通告地址
–image-repository 由于默认拉取镜像地址http://k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址
–kubernetes-version K8s版本，与上面安装的一致
–service-cidr 集群内部虚拟网络，Pod统一访问入口
–pod-network-cidr Pod网络，与下面部署的CNI网络组件yaml中保持一致

.2 提前pull镜像（可选

## 验证配置文件格式是否正确
[shirley@master k8s_install]$ kubeadm config validate --config kubeadm.yaml
ok

## 查看需要下载哪些镜像。需要关注下载的repository地址是否正确。
## 如上在kubeadm.yaml文件配置了imageRepository：registry.aliyuncs.com/google_containers，因此images会从aliyun的repo下载。
[shirley@master k8s_install]$ kubeadm config images list --config kubeadm.yaml
registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.0
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.0
registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.0
registry.aliyuncs.com/google_containers/kube-proxy:v1.28.0
registry.aliyuncs.com/google_containers/pause:3.9
registry.aliyuncs.com/google_containers/etcd:3.5.9-0
registry.aliyuncs.com/google_containers/coredns:v1.10.1

 
## pull镜像。pull镜像有点慢，第一个镜像pull成功后才有日志输出。命令运行后发现没有日志不要着急，多等一会。
[shirley@master k8s_install]$ sudo kubeadm config images pull --config kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.0
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.0
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.9-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1

4.3 kubeadm init初始化

运行kubeadm init --config kubeadm.yaml， 当看到Your Kubernetes control-plane has initialized successfully!时表示安装成功。

[root@master k8s_install]# kubeadm init --config kubeadm.yaml
... ...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.246.133:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:9f7d37c768e658119242dfb49675eaeb3cbdbb7d191526bfa197dd92373b40ab



PS：最好将最后一行log需要记下来，worker节点安装会用到。

根据提示，退出root账户后运行命令

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.4 验证

验证master节点是否部署成功。

## kubectl 运行正常. STATUS=NotReady是因为CNI插件还没装。
[shirley@master k8s_install]$ kubectl get node
NAME   STATUS     ROLES           AGE   VERSION
node   NotReady   control-plane   20m   v1.28.2

## 通过crictl命令可以查看到运行的container
[shirley@master k8s_install]$ sudo crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD
955b0c87ad621       ea1030da44aa1       26 minutes ago      Running             kube-proxy                0                   78efbee65dfac       kube-proxy-kp9kw
f69d8c3246904       73deb9a3f7025       26 minutes ago      Running             etcd                      0                   77180bc7ff0a8       etcd-node
3efba65f263d3       f6f496300a2ae       26 minutes ago      Running             kube-scheduler            0                   f89fb4bb60e2e       kube-scheduler-node
5dfb28390f30b       4be79c38a4bab       26 minutes ago      Running             kube-controller-manager   0                   b716cb4652e1c       kube-controller-manager-node
b8cfce31fa842       bb5e0dde9054c       26 minutes ago      Running             kube-apiserver            0                   006db1ce43cfe       kube-apiserver-node

5.让工作节点加入

5.1 运行kubeadm join将工作节点加入集群

在master节点kubeadm init安装完成后，会有如下类似log


Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.246.133:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:9f7d37c768e658119242dfb49675eaeb3cbdbb7d191526bfa197dd92373b40ab
将上述命令粘贴到工作节点，将工作节点添加到集群

[root@slave ~]# kubeadm join 192.168.246.133:6443 --token abcdef.0123456789abcdef \
>         --discovery-token-ca-cert-hash sha256:9f7d37c768e658119242dfb49675eaeb3cbdbb7d191526bfa197dd92373b40ab
[preflight] Running pre-flight checks
        [WARNING Hostname]: hostname "slave.k8s" could not be reached
        [WARNING Hostname]: hostname "slave.k8s": lookup slave.k8s on 192.168.246.2:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

5.2 验证

到master节点运行kubelet get node,可以看到加进来的node

[shirley@master k8s_install]$ kubectl get nodes
NAME        STATUS     ROLES           AGE    VERSION
node        NotReady   control-plane   121m   v1.28.2
slave.k8s   NotReady   <none>          28s    v1.28.2
当前STATUS都是NotReady，这是因为还没有安装网络插件CNI

5.3 忘记token怎么办

如果master节点安装时，没有记录下token，或token超时(默认24小时)，可以运行如下命令重新生成

[shirley@master k8s_install]$ kubeadm token create --print-join-command
kubeadm join 192.168.246.133:6443 --token by8q23.65btteq9iud7ypso --discovery-token-ca-cert-

6.安装Flannel网络插件

6.1 下载kube-flannel.yml

## 1. 下载kube-flannel.yml
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kube-flannel.yaml文件中，需要注意，net-conf.json里的network要和kubeadm.yaml里配置的networking.podSubnet相同

... ...
  net-conf.json: |
    {
      ### 这里的network和kubeadm.yaml里配置的networking.podSubnet相同 
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
... ...

6.2 容器部署flannel

[root@k8s-master-01 ~]#kubectl apply -f kube-flannel.yml 
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

6.3 验证 运行kubectl -n kube-system get pod -o wide

每个节点上都运行了kube-proxy
[root@k8s-master-01 ~]# kubectl -n kube-system get pod -o wide
NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE        NOMINATED NODE   READINESS GATES
coredns-66f779496c-flbpd       1/1     Running   0          24h   10.244.0.2       node        <none>           <none>
coredns-66f779496c-g4b74       1/1     Running   0          24h   10.244.0.3       node        <none>           <none>
etcd-node                      1/1     Running   0          24h   192.168.188.61   node        <none>           <none>
kube-apiserver-node            1/1     Running   0          24h   192.168.188.61   node        <none>           <none>
kube-controller-manager-node   1/1     Running   0          24h   192.168.188.61   node        <none>           <none>
kube-proxy-8rglh               1/1     Running   0          23h   192.168.188.65   k8s-node1   <none>           <none>
kube-proxy-jjvh9               1/1     Running   0          24h   192.168.188.61   node        <none>           <none>
kube-scheduler-node            1/1     Running   0          24h   192.168.188.61   node        <none>           <none>


# node状态显示为Ready
[root@k8s-master-01 ~]#  kubectl get node
NAME        STATUS   ROLES           AGE   VERSION
k8s-node1   Ready    <none>          24h   v1.28.2
node        Ready    control-plane   24h   v1.28.2

其他说明

另外，在初始安装的Master节点上也启动了kubelet和kube-proxy，在默认情况下并不参与工作负载的调度。如果希望Master节点也作为Node角色，则可以运行下面的命令（删除Master节点的：node-role.kubernetes.io/control-plane:NoSchedule），让Master节点也成为一个Node：

kubectl taint nodes –all node-role.kubernetes.io/control-plane-

参考链接：

https://zhuanlan.zhihu.com/p/660899097

https://zhuanlan.zhihu.com/p/439991250

0.系统要求#

1.准备工作#

2.安装docker以及配置pause镜像地址#

3.安装kubeadm,kubelet,kubectl#

4.部署集群master节点#

5.让工作节点加入#

6.安装Flannel网络插件#