#夏日挑战赛# kubernetes集群搭建 原创 精华

本文正在参加星光计划3.0–夏日挑战赛

简介

学习k8s前建议先学习docker容器及k8s基础知识,有助于理解,使用k8s环境有很多方式,可以用minikube快速试用,也可以通过在线环境进行学习https://www.katacoda.com/courses/kubernetes
本文主要介绍模拟生产环境多节点的k8s集群搭建过程

1. 准备3台虚拟机节点

本示例节点都选用centos7操作系统环境

具体安装方式参考文章
http://www.hushowly.com/articles/1683

1.centos7-1(主节点)

分配的IP为：192.168.56.201

2.centos7-２(工作节点)

分配的IP为：192.168.56.202

3.centos7-３(工作节点)

分配的IP为：192.168.56.203

2. docker安装

集群搭建好后，需要为每个节点装docker环境，docker安装有好几种方案，此处采用最简单的官方的sh安装方案

https://docs.docker.com/engine/install/centos/

2.1 安装docker

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

2.2 启动docker服务

su root
sudo systemctl start docker

3. 安装三架马车

注：集群各节点都需要安装此工具

• kubeadm
  Kubeadm用于快速部署Kubernetes集群工具，提供kubeadm init和kubeadm join
• kubelet
  运行在cluster所有节点上,负责启动POD和容器
• kubectl
  kubenetes命令行工具，通过kubectl远程或本机方式操作部署和管理应用，查看各种资源，创建，删除和更新组件

说明：以下都是基于centos安装和配置，如果是其它操作系统有此许差异，具体可参考官网

官网安装说明　https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

3.1 安装源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3.2 Set SELinux in permissive mode

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

3.3 安装

# 安装
sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

#设置为开机自动激活单元并现在立刻启动
sudo systemctl enable --now kubelet

4 初始化集群

4.1 初始化master节点

也可以通过配置文件初始化，可以方便制定更多的参数

kubeadm init 
--apiserver-advertise-address=192.168.56.201 
--image-repository registry.aliyuncs.com/google_containers 
--service-cidr=10.1.0.0/16 
--pod-network-cidr=10.244.0.0/16

参数说明：

• image-repository 设置镜像源，因为国内网络，必须设置一个可用的镜源
• apiserver-advertise-address 通过该 ip 地址向集群其他节点公布 api server 的信息，必须能够被其他节点访问
• pod-network-cidr 指定pod网络地址范围
• service-cidr 指定service网络地址范围

4.1.1 遇到错误1

[ERROR Swap]: running with swap on is not supported. Please disable swap

• 解决：需要关闭swap

--永久关闭(建议)
vi /etc/fstab 注掉以下行
/swap.img	none	swap	sw	0	0

--临时关闭
swapoff -a 

4.1.2 遇到错误2

[init] Using Kubernetes version: v1.19.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

• 修改docker配置

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

• 重启docker服务

systemctl restart docker

4.1.3 遇到错误3

[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1

处理办法：

echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables

4.1.4 遇到错误4

registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0 无法下载

处理办法：

# 在master想办法通过其它registry pullcoredns镜像
docker pull coredns/coredns:v1.8.0
# 调整tag符合当前image-repository版本信息
docker tag k8s.gcr.io/coredns:v1.8.0 registry.aliyuncs.com/google_containers/coredns/coredns:v1.8.0

4.2 集群master节点初始化成功

[root@centos7-1 ~]# kubeadm init 
> --apiserver-advertise-address=192.168.56.201 
> --image-repository registry.aliyuncs.com/google_containers 
> --service-cidr=10.1.0.0/16 
> --pod-network-cidr=10.244.0.0/16
.........
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
    --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 

4.6 根据成功提示配置kubectl

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf
$HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.7 测试kubectl命令

[root@centos7-1 vagrant]# kubectl get nodes
NAME        STATUS   ROLES    AGE   VERSION
centos7-1   Ready    master   55m   v1.19.3

4.8 记录集群加入join地址

其它工作节点需要通过此命令加入到集群中

注：需要等网络插件成功安装，才能正常加入集群

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
    --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 

5. 安装网络插件

5.1 查看master节点组件状态

[root@centos7-1 vagrant]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-fg7g4            0/1     Pending   0          38m
kube-system   coredns-6d56c8448f-mw84l            0/1     Pending   0          38m
kube-system   etcd-centos7-1                      1/1     Running   0          38m
kube-system   kube-apiserver-centos7-1            1/1     Running   0          38m
kube-system   kube-controller-manager-centos7-1   1/1     Running   0          38m
kube-system   kube-proxy-8xbkk                    1/1     Running   0          38m
kube-system   kube-scheduler-centos7-1            1/1     Running   0          38m

我们发现，因为corednspod没有启动，因为缺少网络pod

5.2 安装网络插件

k8s网络插件很多，此处选择flannel

https://kubernetes.io/zh/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-

• 5.2.1 部署flannel网络

https://github.com/coreos/flannel#flannel

在线方式：

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

备注：有时网络问题无法下载，请切换网络试下

离线方式：
想办法下载kube-flannel.yml到本地

kubectl apply -f kube-flannel.yml

遇到问题1：

[root@centos7-1 vagrant]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
The connection to the server raw.githubusercontent.com was refused - did you specify the right host or port?

解决: 配置KUBECONFIG环境变量
因为kubectl命令需要使用kubernetes-admin来运行，所以需要通过KUBECONFIG找配置文件

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile

5.3 检查网络插件状态(网络组件启动要一会)

从以下看出，网络插件和corednspod都运行正常了

[root@centos7-1 vagrant]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system   coredns-6d56c8448f-fg7g4            1/1     Running   0          70m
kube-system   coredns-6d56c8448f-mw84l            1/1     Running   0          70m
kube-system   etcd-centos7-1                      1/1     Running   0          70m
kube-system   kube-apiserver-centos7-1            1/1     Running   0          70m
kube-system   kube-controller-manager-centos7-1   1/1     Running   1          70m
kube-system   kube-flannel-ds-wx592               1/1     Running   0          8m36s
kube-system   kube-proxy-8xbkk                    1/1     Running   0          70m
kube-system   kube-scheduler-centos7-1            1/1     Running   0          70m

6. 加入工作节点到集群

前提：各工作节点要已安装docker和三架马车相关工具，各工作节点和master是网络是互通的

6.1 获取集群加入join命令

两种方式获取

1. 通过以上4.8章节master节点集群初始成功后记录join地址
   2.在master节点再次执行以下命令获取

kubeadm token create --print-join-command

6.1 完成工作节点加入集群

在相应的工作节点使用root用户执行以下命令：

kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
    --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 

问题：有可能失败，参考以上4.2-4.4的错误处理

• 未启动docker服务
• 禁用swap

工作节点加入集群成功结果：

[root@centos7-2 ~]# kubeadm join 192.168.56.201:6443 --token ng780y.14hb11g0iya7djc6 
>     --discovery-token-ca-cert-hash sha256:342b40ba65b93f2316492176cc479a5415b6c60250b2723f4c7050aa2ab3ea07 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

6.2 查看节点集群状态

回到master节点执行以下命令，查看node加入状态

[root@centos7-1 ~]# kubectl get nodes
NAME        STATUS   ROLES    AGE    VERSION
centos7-1   Ready    master   6m2s   v1.19.3
centos7-2   Ready    <none>   102s   v1.19.3

7. 体验k8s上应用部署

开启命令补全功能，方便命令输入

yum install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)

7.1 创建部署

kubectl create deployment hello-echo --image=registry.aliyuncs.com/google_containers/echoserver:1.10

7.2 发布应用

kubectl expose deployment hello-echo --type=NodePort --port=8080

参数说明：

• –type = NodePort 指定 Service 的类型

7.2 查看应用的外部仿问地址

[root@centos7-1 vagrant]# kubectl get service
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hello-echo   NodePort    10.1.184.187   <none>        8080:31038/TCP   15m
kubernetes   ClusterIP   10.1.0.1       <none>        443/TCP          3h21m

从以上输入PORT中看出，hello-echo对应的外部端口为31038

7.3 仿问发布的应用

8. 安装dashboard

https://kubernetes.io/zh/docs/tasks/access-application-cluster/web-ui-dashboard/

dashboard 可以通过kubectl proxy和NodePort方式对外开放,因为proxy只能在本机仿问,所以以下介绍NodePort方式

8.1 下载dashboard的yaml

wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml

备注：有时网络问题无法下载，请切换网络试下

8.2 修改recommended.yaml服务暴露方式

暴露到端口: 30443

8.3. 部署dashboard

kubectl apply -f recommended.yaml

注：需要等待一会的时间，dashboard相关镜像拉取和容器初始化费点时间,可以使用kubectl get pods --all-namespaces查看部署状态

8.4. 仿问dashboard

注意使用https方式仿问
dashboard正常启动了,但是需要身份验证才能登陆

8.5 创建帐号得到令牌

注: 请在控制面板所在机器操作

• 创建account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
# Create ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

• 执行创建

kubectl create -f account.yaml

• 获取admin-user用户的令牌

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

• 登陆dashboard