一、系统环境要求：

1.各节点配置时间同步

通过安装chrony与网络时间同步

 [root@master1 ~]# yum install chrony -y  

2.主机名称解析

编辑/etc/hosts文件，添加主机名解析记录

3.关闭防火墙

关闭iptables或ufw或firewalld相关服务

 [root@master1 ~]# systemctl stop firewalld
[root@master1 ~]# systemctl disable firewalld
Ubuntu系统：
root@master2:~# ufw disable
Firewall stopped and disabled on system startup  

4.禁用SELinx

getenforce或sestatus查看selinux状态

 [root@master1 ~]# getenforce
Enforcing
[root@master1 ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31  

编辑/etc/sysconfig/selinux的SELINUX=enforcing选项，使SELINUX的值为disabled，重启服务器生效；若不想重启可临时设置如下：

 [root@master1 ~]# setenforce
usage: setenforce [ Enforcing | Permissive | 1 | 0 ]
[root@master1 ~]# setenforce 0  

5.禁用Swap设备；

 swapoff -a
然后，在/etc/fstab中注释掉swap挂载记录行  

6.配置 bridge -nf-call-iptables

设置/proc/sys/net/bridge/bridge-nf-call-iptables和/proc/sys/net/bridge/bridge-nf-call-ip6tables文件内容为1，创建/etc/sysctl.d/k8s.conf文件，并添加如下内容：

 vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net. ipv4 .ip_forward = 1

sysctl -p /etc/sysctl.d/k8s.conf  # 使配置生效  

二、安装应用程序

所有主机需执行相同的操作

1.安装docker

查看：Docker安装部署方法总结【二】

2.Docker配置

各节点安装并启用 Docker容器 运行时，配置好容器加速服务；

• 建议使用阿里云容器加速；
• 需要制定cgroup驱动为 systemd ;

 {
  "exec-opts":["native.cgroupdriver=systemd"],
  "log-driver":"json-file",
  "log-opts":{
    "max-size":"100m"
  },
  "storage-driver":"overlay2",
  "registry-mirrors": ["#34;,"#34;,"#34;]
}  

配置生效

配置完docker配置文件，需要daemon-reload和重启docker生效，可以用docker info命令查看

 [root@master1 ~]# systemctl daemon-reload
[root@master1 ~]# systemctl restart docker  

3.安装kubeadm、kubectl和kubelet;

由于kubenetes使用Google主导研发，所有程序镜像包大陆无法访问（如gcr.io），只能通过内陆的某些比如 阿里巴巴 和清华大学镜像来获取。阿里巴巴k8s镜像访问点击这里查看，根据提示进行配置，然后进行安装。

参考资料：

1.For Debian/Ubuntu

 apt-get update && apt-get install -y apt-transport-https
curl  | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb  kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl  # 默认安装当前最新版本
root@master2:/etc/docker# apt-get install -y kubectl=1.18.1-00 kubeadm=1.18.1-00 kubelet=1.18.1-00  # 安装指定版本程序  

2.For CentOS / RHEL / Fedora

 cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey= 
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet  # 默认安装最新版本软件
[root@node1 system]# yum install -y kubectl-1.18.1 kubeadm-1.18.1 kubelet-1.18.1 # 安装指定版本的软件包  

ps: 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用 yum install -y –nogpgcheck kubelet kubeadm kubectl 安装

4.设置开机自启动

 root@master2:~# systemctl enable docker kubelet  

三、初始化控制平面

1..初始化命令选项查看帮助

 [root@master1 lib]# kubeadm init --help
Run this command in order to set up the Kubernetes control plane

The "init" command executes the following phases:
```
preflight Run pre-flight checks
kubelet-start Write kubelet settings and (re)start the kubelet
certs Certificate generation
/ca Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
/apiserver Generate the certificate for serving the Kubernetes API
/apiserver-kubelet-client Generate the certificate for the API server to connect to kubelet
/front-proxy-ca Generate the self-signed CA to provision identities for front proxy
/front-proxy-client Generate the certificate for the front proxy client
/etcd-ca Generate the self-signed CA to provision identities for etcd
/etcd-server Generate the certificate for serving etcd
/etcd-peer Generate the certificate for etcd nodes to communicate with each other
/etcd-healthcheck-client Generate the certificate for liveness probes to healthcheck etcd
/apiserver-etcd-client Generate the certificate the apiserver uses to access etcd
/sa Generate a private key for signing service account tokens along with its public key
kubeconfig Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
/admin Generate a kubeconfig file for the admin to use and for kubeadm itself
/kubelet Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
/controller-manager Generate a kubeconfig file for the controller manager to use
/scheduler Generate a kubeconfig file for the scheduler to use
control-plane Generate all static Pod manifest files necessary to establish the control plane
/apiserver Generates the kube-apiserver static Pod manifest
/controller-manager Generates the kube-controller-manager static Pod manifest
/scheduler Generates the kube-scheduler static Pod manifest
etcd Generate static Pod manifest file for local etcd
/local Generate the static Pod manifest file for a local, single-node local etcd instance
upload-config Upload the kubeadm and kubelet configuration to a ConfigMap
/kubeadm Upload the kubeadm ClusterConfiguration to a ConfigMap
/kubelet Upload the kubelet component config to a ConfigMap
upload-certs Upload certificates to kubeadm-certs
mark-control-plane Mark a node as a control-plane
bootstrap-token Generates bootstrap tokens used to join a node to a cluster
kubelet-finalize Updates settings relevant to the kubelet after TLS bootstrap
/experimental-cert-rotation Enable kubelet client certificate rotation
addon Install required addons for passing Conformance tests
/coredns Install the CoreDNS addon to a Kubernetes cluster
/kube-proxy Install the kube-proxy addon to a Kubernetes cluster
```
Usage:
  kubeadm init [flags]
  kubeadm init [command]

Available Commands:
  phase       Use this command to invoke single phase of the init workflow

Flags:
      --apiserver-advertise-address string   The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
      --apiserver-bind-port int32            Port for the API Server to bind to. (default 6443)
      --apiserver-cert-extra-sans strings    Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
      --cert-dir string                      The path where to save and store the certificates. (default "/etc/kubernetes/pki")
      --certificate-key string               Key used to encrypt the control-plane certificates in the kubeadm-certs Secret.
      --config string                        Path to a kubeadm configuration file.
      --control-plane-endpoint string        Specify a stable IP address or DNS name for the control plane.
      --cri-socket string                    Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
      --dry-run                              Don't apply any changes; just output what would be done.
  -k, --experimental-kustomize string        The path where kustomize patches for static pod manifests are stored.
      --feature-gates string                 A set of key=value pairs that describe feature gates for various features. Options are:
                                             IPv6DualStack=true|false (ALPHA - default=false)
                                             PublicKeysECDSA=true|false (ALPHA - default=false)
  -h, --help                                 help for init
      --ignore-preflight-errors strings      A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
      --image-repository string              Choose a container registry to pull control plane images from (default "k8s.gcr.io")
      --kubernetes-version string            Choose a specific Kubernetes version for the control plane. (default "stable-1")
      --node-name string                     Specify the node name.
      --pod-network-cidr string              Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
      --service-cidr string                  Use alternative range of IP address for service VIPs. (default "10.96.0.0/12")
      --service-dns-domain string            Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local")
      --skip-certificate-key-print           Don't print the key used to encrypt the control-plane certificates.
      --skip-phases strings                  List of phases to be skipped
      --skip-token-print                     Skip printing of the default bootstrap token generated by 'kubeadm init'.
      --token string                         The token to use for establishing bidirectional trust between nodes and control-plane nodes. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
      --token-ttl duration                   The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire (default 24h0m0s)
      --upload-certs                         Upload control-plane certificates to the kubeadm-certs Secret.

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm init [command] --help" for more information about a command.  

2.开始初始化

初始化命令（可根据如下选项释义和自己实际情况进行修改）

[erphpdown]

 [root@master1 ~]# kubeadm init \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.18.1 \
--control-plane-endpoint k8s-api.ilinux.io \
--apiserver-advertise-address 192.168.222.153 \
--pod-network-cidr 10.244.0.0/16 \
--token-ttl 0  

选项意义：

• –image-repository: 指定要使用的镜像仓库；
• –kubernetes-version: kubernetes程序组件的版本号，应该与kubelet的版本号相同
• –control-plane-endpoint: 控制平面固定端点，可以是IP地址或DNS名称，会被用于集群管理员即集群组件的kubeconflg配置文件的API Server的访问地址；单控制平面部署可以不使用该选项；
• –pod-network-cidr: Pod网络的地址范围，其值为CIDR格式的网络地址，flannel网络插件的默认为10.244.0.0/16, callco插件的默认值为192.168.0.0/16;
• –service-cidr: Service的网络地址范围，其值为CIDR格式的网络地址，默认为10.96.0.0/12;
• –apiserver-advertise-address: apiserver通告给其他组件的IP地址，一般应该为Master节点的用于集群内部通信的IP地址，0.0.0.0表示节点上所有可用地址；
• –token-ttl: 共享令牌（token）的过期时长，默认为24小时，0表示不过期；为防止不安全存储等原因导致的令牌泄露危机到集群安全，建议为其设定过期时长；

1.若初始化报错，可以在/etc/sysconfig/kubelet文件中添加参数，忽略报错。

 [root@master1 package]# more /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=  

如设置KUBELET_EXTRA_ARGS=”–fail-swap-on=false”可忽略交换空间报错，表示swap出错的时候，不让其出错，相当于忽略。初始化的时候可添加忽略选项–ignore-preflight-errors=Swap

2.若报错如下

 May 14 13:44:55 master1 kubelet: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See  for more information.
May 14 13:44:55 master1 kubelet: F0514 13:44:55.571137 16841 server.go:199] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory  

这是因为初始化还没有完成，相关配置文件还没有自动生成，可以忽略，等初始化成功即可。

[/erphpdown]

初始化成功会有如下提示（包括安装环境预检、生成证书、私钥、配置文件、pod清单文件、部署addon等过程）：

 W0514 16:26:38.198975 3142 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-api.ilinux.io] and IPs [10.96.0.1 192.168.222.150]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master1 localhost] and IPs [192.168.222.150 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master1 localhost] and IPs [192.168.222.150 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0514 16:28:24.490184 3142 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0514 16:28:24.494827 3142 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 28.512491 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: ezl1f1.ha4fc5werojbu359
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:


You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

kubeadm join k8s-api.ilinux.io:6443 --token ezl1f1.ha4fc5werojbu359 \
--discovery-token-ca-cert-hash sha256:c84d70dd5e0ce8f9305f09955c524a8aac9bb189445b58d026dba78d9981ed43 \
--control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8s-api.ilinux.io:6443 --token ezl1f1.ha4fc5werojbu359 \
--discovery-token-ca-cert-hash sha256:c84d70dd5e0ce8f9305f09955c524a8aac9bb189445b58d026dba78d9981ed43  

初始化成功后操作

安装提示执行提示要执行的命令即可：

 [root@master1 opt]# mkdir -p $HOME/.kube
[root@master1 opt]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master1 opt]# chown $(id -u):$(id -g) $HOME/.kube/config
[root@master1 opt]# ll -h $HOME/.kube/config 
-rw------- 1 root root 5.4K May 14 16:58 /root/.kube/config  

node加入集群

在node节点执行命令，使其添加到集群中：

 [root@node1 system]# kubeadm join k8s-api.ilinux.io:6443 --token ezl1f1.ha4fc5werojbu359 \
> --discovery-token-ca-cert-hash sha256:c84d70dd5e0ce8f9305f09955c524a8aac9bb189445b58d026dba78d9981ed43
W0514 17:42:23.395760    4416 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.  

四、安装网络插件Flannel

1.Flannel在Github上的项目

地址:

2.开始安装部署flannel

 [root@master1 opt]# kubectl apply -f   

可能由于国内网络被qiang等原因，很难下载kube-flannel.yml和pull到flannel镜像。本人费九牛二虎之力用global服务器下载了kube-flannel.yml和pull到相关镜像，然后docker save打包为一个文件导入所需服务器load后即可使用，做如下操作记录：

 [root@ecs-e84a ~]# docker pull quay.io/coreos/flannel:v0.12.0-arm64 && \
> docker pull quay.io/coreos/flannel:v0.12.0-arm && \
> docker pull quay.io/coreos/flannel:v0.12.0-ppc64le && \
> docker pull quay.io/coreos/flannel:v0.12.0-s390x
v0.12.0-arm64: Pulling from coreos/flannel
8fa90b21c985: Pull complete 
c4b41df13d81: Pull complete 
a73758d03943: Pull complete 
d09921139b63: Pull complete 
17ca61374c07: Pull complete 
6da2b4782d50: Pull complete 
Digest: sha256:a2f5081b71ee4688d0c7693d7e5f2f95e9eea5ea3b4147a12179f55ede42c185
Status: Downloaded newer image for quay.io/coreos/flannel:v0.12.0-arm64
quay.io/coreos/flannel:v0.12.0-arm64
v0.12.0-arm: Pulling from coreos/flannel
832e07764099: Pull complete 
d0888001d791: Pull complete 
f0e1b4ffe531: Pull complete 
a0af26dd2937: Pull complete 
8701ece6be98: Pull complete 
bf9c02e8240d: Pull complete 
Digest: sha256:ca8249ab50424e07d15298a2fda29003be3fd930750565d7b068b5ef9d36bd53
Status: Downloaded newer image for quay.io/coreos/flannel:v0.12.0-arm
quay.io/coreos/flannel:v0.12.0-arm
v0.12.0-ppc64le: Pulling from coreos/flannel
cd95c8a93e39: Pull complete 
324d08c752b6: Pull complete 
7e5cf18f4805: Pull complete 
eef736355019: Pull complete 
9d84e68c2ffe: Pull complete 
aeb40b202208: Pull complete 
Digest: sha256:28314b287f84b62f62e2bdb30e0cdba17ac97dd4822efb4664a7391637b0f052
Status: Downloaded newer image for quay.io/coreos/flannel:v0.12.0-ppc64le
quay.io/coreos/flannel:v0.12.0-ppc64le
v0.12.0-s390x: Pulling from coreos/flannel
176bad61a3a4: Pull complete 
13b80a37370b: Pull complete 
42d8e66fa893: Pull complete 
266247e2e603: Pull complete 
1b56fbc8a8e1: Pull complete 
85ecb68de469: Pull complete 
Digest: sha256:3ce5b8d40451787e1166bf6b207c7834c13f7a0712b46ddbfb591d8b5906bfa6
Status: Downloaded newer image for quay.io/coreos/flannel:v0.12.0-s390x
quay.io/coreos/flannel:v0.12.0-s390x
[root@ecs-e84a ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.12.0-s390x 57eade024bfb 2 months ago 56.9MB
quay.io/coreos/flannel v0.12.0-ppc64le 9225b871924d 2 months ago 70.3MB
quay.io/coreos/flannel v0.12.0-arm64 7cf4a417daaa 2 months ago 53.6MB
quay.io/coreos/flannel v0.12.0-arm 767c3d1f8cba 2 months ago 47.8MB
quay.io/coreos/flannel v0.12.0-amd64 4e9f801d2217 2 months ago 52.8MB
[root@ecs-e84a flannel]# docker save quay.io/coreos/flannel:v0.12.0-amd64 quay.io/coreos/flannel:v0.12.0-arm64 quay.io/coreos/flannel:v0.12.0-arm quay.io/coreos/flannel:v0.12.0-ppc64le quay.io/coreos/flannel:v0.12.0-s390x -o flannelv0.12.0.tar
[root@ecs-e84a flannel]# ll -h
total 274M
-rw------- 1 root root 274M May 15 22:24 flannelv0.12.0.tar  

查看node几点状态，如下显示Ready表示集群正常，K8S已安装成功！(k8s的相关管理和操作使用等请参考其他相关文章…)

 [root@master1 flannel]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 2d5h v1.18.1
node1 Ready <none> 9h v1.18.1
node2 Ready <none> 2d4h v1.18.1