Ubuntu22.04中安装Kubernetes1.27高可用（Docker作为容器运行时） Kubernetes1.27版本安装记录过程 安装前准备 集群规划 前提准备 开始安装 1. 安装容器运行时（所有节点执行） 安装前准备 更新apt 并 安装必要工具包 添加Docker官方GPG key 设置apt库 安装Docker 安装24.0.0版本Docker 验证是否成功 手动设置Docker使用systemd cgroup驱动 启动并设置开机启动 查看Docker的cgroup驱动是systemd 安装cri-dockerd 下载cri-docker版本：0.3.2 安装deb软件包 添加infra容器镜像配置 启动并设置开机启动 2. 安装 kubeadm、kubelet 和 kubectl （所有节点执行） 3. 配置cgoup驱动程序（所有节点执行） 4. 配置负载均衡（控制平面节点执行） 说明 安装KeepAlive（k8s静态Pod方式安装） 创建/etc/keepalived/keepalived.conf配置文件（主： k8s-master01节点） 创建/etc/keepalived/keepalived.conf配置文件（备： k8s-master02、k8s-master03节点） 创建心跳检测文件（主备都执行： k8s-master01、k8s-master02、k8s-master03） 创建Keepalive Pod yaml文件（主备都执行： k8s-master01、k8s-master02、k8s-master03） 安装HAProxy（k8s静态Pod方式安装） 创建HAProxy配置文件 （k8s-master01、k8s-master02、k8s-master03执行） 创建HAProxy Pod 需要的yaml文件（k8s-master01、k8s-master02、k8s-master03执行） 针对Keepalive和HAProxy的说明 5. 初始化Master节点（控制平面节点执行） 参考界面 查看kubeadm init命令默认配置文件 （参考） 下载镜像 手动安装1.27.1版本需要的镜像 启动初始化 配置用户可以使用kubectl 命令 验证kubectl 命令是否可用 安装容器网络（CNI）插件 移除控制平面污点 7. Worker节点加入集群（所有Woker节点执行，必须使用root用户执行） 8. 高可用Master主节点加入集群（比如使用Root用户执行） 9. 可选操作 （可选）从控制平面节点以外的计算机控制集群 （可选）将 API 服务器代理到本地主机 Kubernetes1.27版本安装记录过程 安装参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ 安装前准备 集群规划 系统： Ubuntu22.04 root@k8s-master01:~# lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy 安装工具：kubeadm 集群分布： 3主节点 + 2个工作节点 容器运行时： 选用Docker 因为1.24版本在k8s主干分支中移除了 dockershim ，并将其给了docker维护，后将其改名为cni-dockerd,因此我们需要安装： 安装Docker 安装cni-dockerd 采用堆叠的方式创建高可用集群（etcd和控制平面其他组件在同一个节点上）： 集群节点图： 高可用参考： https://developer.aliyun.com/article/853054 前提准备 ​ 节点之中不可以有重复的主机名、MAC 地址或 product_uuid。请参见这里了解更多详细信息。 主机名验证 # 查看主机名 hostname # 修改为k8s-master (替换成每个节点主机名) hostnamectl hostname k8s-master Mac地址验证 使用命令 ip link 或 ifconfig -a 来获取网络接口的 MAC 地址 Product_uuid验证 sudo cat /sys/class/dmi/id/product_uuid 配置主机名映射 ​ 配置文件：/etc/hosts 192.168.0.18 k8s-master 192.168.0.19 k8s-master01 192.168.0.20 k8s-master02 192.168.0.21 k8s-master03 192.168.0.22 k8s-slave01 192.168.0.23 k8s-slave02 192.168.0.18 k8s-master 这一条是为了配置高可用集群准备的。 关闭防火墙 ufw disable 转发 IPv4 并让 iptables 看到桥接流量 执行下述指令： cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay sudo modprobe br_netfilter # 设置所需的 sysctl 参数，参数在重新启动后保持不变 cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF # 应用 sysctl 参数而不重新启动 sudo sysctl --system 通过运行以下指令确认 br_netfilter 和 overlay 模块被加载 lsmod | grep br_netfilter lsmod | grep overlay 通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1： sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward 一台兼容的 Linux 主机。Kubernetes 项目为基于 Debian 和 Red Hat 的 Linux 发行版以及一些不提供包管理器的发行版提供通用的指令。 每台机器 2 GB 或更多的 RAM（如果少于这个数字将会影响你应用的运行内存）。 CPU 2 核心及以上。 集群中的所有机器的网络彼此均能相互连接（公网和内网都可以）。 开启机器上的某些端口。请参见这里了解更多详细信息。 nc 127.0.0.1 6443 无输出，表示正常 禁用交换分区。为了保证 kubelet 正常工作，你必须禁用交换分区。 例如，sudo swapoff -a 将暂时禁用交换分区。要使此更改在重启后保持不变，请确保在如 /etc/fstab、systemd.swap 等配置文件中禁用交换分区，具体取决于你的系统如何配置。 #关闭swap swapoff -a sed -ri 's/.*swap.*/#&/' /etc/fstab ​ 查看主机cgroup版本 stat -fc %T /sys/fs/cgroup/ 对于 cgroup v1，输出为 tmpfs。 对于 cgroup v2，输出为 cgroup2fs。 cgroup v2 具有以下要求： 操作系统发行版启用 cgroup v2 Linux 内核为 5.8 或更高版本 容器运行时支持 cgroup v2。例如： containerd v1.4 和更高版本 cri-o v1.20 和更高版本 kubelet 和容器运行时被配置为使用 systemd cgroup 驱动 参考： https://kubernetes.io/zh-cn/docs/concepts/architecture/cgroups/ 开始安装 1. 安装容器运行时（所有节点执行） https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#docker 安装前准备 更新apt 并 安装必要工具包 sudo apt-get remove docker docker-engine docker.io containerd runc sudo apt-get update sudo apt-get install ca-certificates curl gnupg 添加Docker官方GPG key sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg sudo chmod a+r /etc/apt/keyrings/docker.gpg 设置apt库 echo \ "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null 安装Docker 安装24.0.0版本Docker sudo apt-get update VERSION_STRING=5:24.0.0-1~ubuntu.22.04~jammy sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin -y 验证是否成功 root@k8s-master01:~# docker version Client: Docker Engine - Community Version: 24.0.0 API version: 1.43 Go version: go1.20.4 Git commit: 98fdcd7 Built: Mon May 15 18:49:22 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.0 API version: 1.43 (minimum version 1.12) Go version: go1.20.4 Git commit: 1331b8c Built: Mon May 15 18:49:22 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.21 GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8 runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061 docker-init: Version: 0.19.0 GitCommit: de40ad0 手动设置Docker使用systemd cgroup驱动 修改： /etc/docker/daemon.json文件,添加如下内容 保存重启docker cat > /etc/docker/daemon.json << EOF { "registry-mirrors": ["https://84bkfzte.mirror.aliyuncs.com"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF systemctl daemon-reload && systemctl restart docker 启动并设置开机启动 systemctl enable docker --now 查看Docker的cgroup驱动是systemd root@k8s-master01:~# docker info | grep -i cgroup Cgroup Driver: systemd Cgroup Version: 2 cgroupns 安装cri-dockerd 下载cri-docker版本：0.3.2 ​ https://github.com/Mirantis/cri-dockerd/releases/tag/v0.3.2 安装deb软件包 sudo dpkg -i ./cri-dockerd_0.3.2.3-0.ubuntu-jammy_amd64.deb 添加infra容器镜像配置 修改镜像地址为国内，否则kubelet拉取不了镜像导致启动失败 sudo sed -i 's|ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://|ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9|' /usr/lib/systemd/system/cri-docker.service 修改以后内容如下： cat /usr/lib/systemd/system/cri-docker.service ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 启动并设置开机启动 systemctl daemon-reload systemctl enable cri-docker --now 2. 安装 kubeadm、kubelet 和 kubectl （所有节点执行） 你需要在每台机器上安装以下的软件包： kubeadm：用来初始化集群的指令。 kubelet：在集群中的每个节点上用来启动 Pod 和容器等。 kubectl：用来与集群通信的命令行工具。 更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包， 并配置阿里apt源： sudo apt-get update sudo apt-get install -y apt-transport-https ca-certificates curl curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/kubernetes-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list 更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本： sudo apt-get update sudo apt-get install -y kubelet=1.27.1-00 kubeadm=1.27.1-00 kubectl=1.27.1-00 sudo apt-mark hold kubelet kubeadm kubectl 查看对应版本 root@k8s-master01:~# kubectl version -o yaml clientVersion: buildDate: "2023-04-14T13:21:19Z" compiler: gc gitCommit: 4c9411232e10168d7b050c49a1b59f6df9d7ea4b gitTreeState: clean gitVersion: v1.27.1 goVersion: go1.20.3 major: "1" minor: "27" platform: linux/amd64 kustomizeVersion: v5.0.1 The connection to the server localhost:8080 was refused - did you specify the right host or port? root@k8s-master01:~# kubelet --version Kubernetes v1.27.1 root@k8s-master01:~# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:20:04Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"} root@k8s-master01:~# 注意：kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环（正常现象，初始化好主节点就好了）。 3. 配置cgoup驱动程序（所有节点执行） 配置配置kubelet cgroup驱动 https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#%E6%9B%B4%E6%96%B0%E6%89%80%E6%9C%89%E8%8A%82%E7%82%B9%E7%9A%84-cgroup-%E9%A9%B1%E5%8A%A8 警告： 你需要确保容器运行时和 kubelet 所使用的是相同的 cgroup 驱动，否则 kubelet 进程会失败。 相, 我们都是使用 systemd, 新建文件并写入内容： root@k8s-master01:~# cat << EOF >> /etc/default/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" EOF root@k8s-master01:~# cat /etc/default/kubelet KUBELET_EXTRA_ARGS="--cgroup-driver=systemd" root@k8s-master01:~# systemctl restart kubelet 4. 配置负载均衡（控制平面节点执行） ​ 参考页面：https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#options-for-software-load-balancing 说明 我们使用KeepAlive和HAProxy完成负载均衡高可用的配置。 安装KeepAlive（k8s静态Pod方式安装） 创建/etc/keepalived/keepalived.conf配置文件（主： k8s-master01节点） sudo mkdir -p /etc/keepalived sudo cat << EOF > /etc/keepalived/keepalived.conf ! /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script check_apiserver { # 指定验证KeepAlive是否存活脚本位置 script "/etc/keepalived/check_apiserver.sh" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { ! 指定MASTER 或者 BACKUP， 这里我们使用k8s-master01节点作为MASTER state MASTER ! 网卡名称 interface enp0s3 ! 指定router_id，集群该值必须相同，这里指定为：51 virtual_router_id 51 ! 优先级： MASTER节点优先级要高于BACKUP节点，我们MASTER节点配置：101， BACKUP节点设置：100 priority 101 authentication { auth_type PASS ! 验证密码： 集群该值必须相同，这里指定为：42 auth_pass 42 } virtual_ipaddress { ! 虚拟IP地址： 改地址将作为KeepAlive对外暴露的地址，指定的IP必须是你集群所在的网络里面没有被使用的IP地址，这里指定：192.168.0.18 ! 同时改地址也是将要指定kubeadm init 命令 --control-plane-endpoint 参数中的，至于端口， 需要在HAProxy里面指定 192.168.0.18 } track_script { check_apiserver } } EOF cat /etc/keepalived/keepalived.conf 创建/etc/keepalived/keepalived.conf配置文件（备： k8s-master02、k8s-master03节点） sudo mkdir -p /etc/keepalived sudo cat << EOF > /etc/keepalived/keepalived.conf ! /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script check_apiserver { # 指定验证KeepAlive是否存活脚本位置 script "/etc/keepalived/check_apiserver.sh" interval 3 weight -2 fall 10 rise 2 } vrrp_instance VI_1 { ! 指定MASTER 或者 BACKUP， 这里我们使用k8s-master01节点作为MASTER state BACKUP ! 网卡名称 interface enp0s3 ! 指定router_id，集群该值必须相同，这里指定为：51 virtual_router_id 51 ! 优先级： MASTER节点优先级要高于BACKUP节点，我们MASTER节点配置：101， BACKUP节点设置：100 priority 100 authentication { auth_type PASS ! 验证密码： 集群该值必须相同，这里指定为：42 auth_pass 42 } virtual_ipaddress { ! 虚拟IP地址： 改地址将作为KeepAlive对外暴露的地址，指定的IP必须是你集群所在的网络里面没有被使用的IP地址，这里指定：192.168.0.18 ! 同时改地址也是将要指定kubeadm init 命令 --control-plane-endpoint 参数中的，至于端口， 需要在HAProxy里面指定 192.168.0.18 } track_script { check_apiserver } } EOF cat /etc/keepalived/keepalived.conf 创建心跳检测文件（主备都执行： k8s-master01、k8s-master02、k8s-master03） sudo mkdir -p /etc/keepalived sudo cat << EOF > /etc/keepalived/check_apiserver.sh # /etc/keepalived/check_apiserver.sh #!/bin/sh errorExit() { echo "*** $*" 1>&2 exit 1 } curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null || errorExit "Error GET https://localhost:8443/" if ip addr | grep -q 192.168.0.18; then curl --silent --max-time 2 --insecure https://192.168.0.18:8443/ -o /dev/null || errorExit "Error GET https://192.168.0.18:8443/" fi EOF cat /etc/keepalived/check_apiserver.sh 创建Keepalive Pod yaml文件（主备都执行： k8s-master01、k8s-master02、k8s-master03） 文件名： /etc/kubernetes/manifests/keepalived.yaml sudo cat << EOF > /etc/kubernetes/manifests/keepalived.yaml apiVersion: v1 kind: Pod metadata: creationTimestamp: null name: keepalived namespace: kube-system spec: containers: - image: osixia/keepalived:2.0.17 name: keepalived resources: {} securityContext: capabilities: add: - NET_ADMIN - NET_BROADCAST - NET_RAW volumeMounts: - mountPath: /usr/local/etc/keepalived/keepalived.conf name: config - mountPath: /etc/keepalived/check_apiserver.sh name: check hostNetwork: true volumes: - hostPath: path: /etc/keepalived/keepalived.conf name: config - hostPath: path: /etc/keepalived/check_apiserver.sh name: check status: {} EOF cat /etc/kubernetes/manifests/keepalived.yaml 安装HAProxy（k8s静态Pod方式安装） 说明： ​ 由于现在没有进行kubeadm init 操作，因此现在kubelet组件启动不了，因此想要看到效果需要等到kubeadm init 指定完成以后。 创建HAProxy配置文件 （k8s-master01、k8s-master02、k8s-master03执行） sudo mkdir -p /etc/haproxy sudo cat << EOF > /etc/haproxy/haproxy.cfg # /etc/haproxy/haproxy.cfg #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global log /dev/log local0 log /dev/log local1 notice daemon #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 1 timeout http-request 10s timeout queue 20s timeout connect 5s timeout client 20s timeout server 20s timeout http-keep-alive 10s timeout check 10s #--------------------------------------------------------------------- # apiserver frontend which proxys to the control plane nodes #--------------------------------------------------------------------- frontend apiserver # 指定负载均衡绑定的地址和端口，这里的端口需要和/etc/keepalived/check_apiserver.sh文件中监控的端口相同 bind *:8443 mode tcp option tcplog default_backend apiserver #--------------------------------------------------------------------- # round robin balancing for apiserver #--------------------------------------------------------------------- backend apiserver option httpchk GET /healthz http-check expect status 200 mode tcp option ssl-hello-chk balance roundrobin # HAProxy负载均衡器代理的后端节点 server k8s-master01 192.168.0.19:6443 check server k8s-master02 192.168.0.20:6443 check server k8s-master03 192.168.0.21:6443 check # [...] EOF cat /etc/haproxy/haproxy.cfg 创建HAProxy Pod 需要的yaml文件（k8s-master01、k8s-master02、k8s-master03执行） sudo cat << EOF > /etc/kubernetes/manifests/haproxy.yaml apiVersion: v1 kind: Pod metadata: name: haproxy namespace: kube-system spec: containers: - image: haproxy:2.1.4 name: haproxy livenessProbe: failureThreshold: 8 httpGet: host: localhost path: /healthz # 指定HAProxy代理的端口，该端口必须和/etc/haproxy/haproxy.cfg配置的端口相同 port: 8443 scheme: HTTPS volumeMounts: - mountPath: /usr/local/etc/haproxy/haproxy.cfg name: haproxyconf readOnly: true hostNetwork: true volumes: - hostPath: path: /etc/haproxy/haproxy.cfg type: FileOrCreate name: haproxyconf status: {} EOF cat /etc/kubernetes/manifests/haproxy.yaml 针对Keepalive和HAProxy的说明 我们配置负载均衡使用8443端口而没有使用默认的6443端口是因为我们Keepalive和HAProxy都部署在主节点上，而主节点上也部署Kubernetes的api-server组件，而api-server主键已经占用了 6443端口，因此，我们这里配置了8443端口。 当使用Keepalived和HAProxy组合作为高可用负载均衡时，可以构建一个可靠的架构来提供高可用性和负载均衡功能。下面是一个简化的架构图，以可视化方式展示Keepalived和HAProxy的组合： +----------------------+ | Load Balancer | | (HAProxy) | +----------------------+ | | | | +--------+ +--------+ | | +--------------+ +--------------+ | Backend 1 | | Backend 2 | +--------------+ +--------------+ 在上面的架构图中，有以下组件： Load Balancer (HAProxy)：负责接收客户端请求并将其转发到后端服务器。HAProxy是一种高性能的负载均衡器，能够根据不同的负载均衡算法将请求分发到后端服务器，以实现负载均衡和高可用性。 Backend 1 和 Backend 2：这些是真实的后端服务器，用于处理客户端请求。可以有多个后端服务器，以实现负载均衡和高可用性。这些后端服务器可以是应用服务器、数据库服务器等。 Keepalived：用于实现高可用性的组件。Keepalived监测Load Balancer节点的可用性，并在主节点发生故障时将其切换到备份节点。Keepalived使用虚拟IP地址（VIP）来提供无缝的故障转移和高可用性。 在这个架构中，客户端请求首先到达Load Balancer（HAProxy），然后根据负载均衡算法选择一个后端服务器进行处理。如果Load Balancer节点出现故障，Keepalived会自动检测到，并将主节点的VIP切换到备份节点，以确保服务的持续可用性。 访问HAProxy的入口是Keepalived提供的虚拟IP（VIP）。Keepalived会将虚拟IP绑定到主节点上，以便客户端可以通过该虚拟IP与负载均衡器通信。 在高可用负载均衡架构中，客户端不需要直接连接到单个负载均衡器节点。相反，客户端将请求发送到虚拟IP地址，该虚拟IP地址由Keepalived管理并绑定到当前的主节点上。通过这种方式，客户端可以无需关心主节点和备份节点之间的切换，而始终通过虚拟IP与负载均衡器通信。 当Keepalived检测到主节点故障时，它会自动将虚拟IP迁移到备份节点上，以实现无缝的故障转移。这样，客户端可以继续使用相同的虚拟IP与负载均衡器通信，无需感知主备节点的切换。 总结起来，访问HAProxy的入口就是通过Keepalived提供的虚拟IP。客户端可以使用该虚拟IP来连接负载均衡器，并由负载均衡器将请求转发到后端服务器。 5. 初始化Master节点（控制平面节点执行） 参考界面 kubeadm init 命令运行过程： ​ https://kubernetes.io/zh-cn/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images kubeadm config print 打印kubeadm join 或者 kubeadm init 命令默认值： ​ https://kubernetes.io/zh-cn/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-print ​ 查看kubeadm init命令默认配置文件 （参考） 这里输出的yaml格式不正确： 正确格式参考： ​ https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/ root@k8s-master01:~# kubeadm config print init-defaults | tee kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent name: node taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: registry.k8s.io kind: ClusterConfiguration kubernetesVersion: 1.27.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} 下载镜像 手动安装1.27.1版本需要的镜像 # 注意： 这里 EOF 是使用单引号括起来的，不适用单引号，后面的脚本文件会执行，导致结果错误 sudo cat << 'EOF' > download_image.sh #!/bin/bash # Kubernetes 安装的版本 KUBERNETES_VERSION=$(kubeadm version | grep -oP 'GitVersion:"v\K[^"]+') # 阿里Kubernetes官方镜像库 AILI_KUBERNETES_REGISTRY="registry.cn-hangzhou.aliyuncs.com/google_containers" echo "KUBERNETES_VERSION => ${KUBERNETES_VERSION}" echo "AILI_KUBERNETES_REGISTRY => ${AILI_KUBERNETES_REGISTRY}" # 下载并重命名镜像 function download_and_tag_image() { # 官方镜像全称： registry.k8s.io/xxx/xxx:xxx # 比如： registry.k8s.io/kube-proxy:v1.27.1 local full_official_image=$1 local ali_image ali_image=$(echo "$full_official_image" | sed -E "s|(.*/)(.*)|$AILI_KUBERNETES_REGISTRY/\2|") echo "downloading image => $ali_image" echo "downloading image => $ali_image" sudo docker pull "$ali_image" # 重命名镜像 echo "rename image $ali_image to $full_official_image" sudo docker tag "$ali_image" "$full_official_image" } # 官方镜像列表 OFFICIAL_IMAGE_LIST=$(kubeadm config images list --kubernetes-version "$KUBERNETES_VERSION" 2>/dev/null | grep "$OFFICIAL_KUBERNETES_REGISTRY") for official_image in $OFFICIAL_IMAGE_LIST; do download_and_tag_image "$official_image" done EOF cat download_image.sh sudo chmod u+x ./download_image.sh && ./download_image.sh 启动初始化 root@k8s-master01:~# kubeadm init \ --apiserver-advertise-address=192.168.0.19 \ --kubernetes-version v1.27.1 \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --cri-socket=unix:///var/run/cri-dockerd.sock \ --control-plane-endpoint=k8s-master:8443 \ --upload-certs [init] Using Kubernetes version: v1.27.1 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' W0520 20:12:40.038258 30744 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0) W0520 20:12:40.293262 30744 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image. [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.19] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.19 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.0.19 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" W0520 20:12:42.465414 30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file W0520 20:12:42.555482 30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file W0520 20:12:42.934781 30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file W0520 20:12:43.058171 30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" W0520 20:12:43.628841 30744 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0) [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 9.011713 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947 [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: th5i1f.fnzc9v0yb6z3aok8 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS W0520 20:12:54.340208 30744 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of the control-plane node running the following command on each as root: kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \ --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \ --control-plane --certificate-key 3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947 Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \ --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 --apiserver-advertise-address=192.168.0.20 : 用于为控制平面节点的 API server 设置广播地址（必须指定master节点IP） --cri-socket=unix:///var/run/cri-dockerd.sock : 指定CRI 套接字的路径，我们这里有两个运行时环境： containerd 和 cri-dockerd， 这里我们指定cri-dockerd。 注意： kubeadm join 命令的时候也需要指定。 --control-plane-endpoint=k8s-master:8443 : 用于为所有控制平面(高可用环境必须指定)节点设置共享端点，集群节点都要需要配置 /etc/hosts 将k8s-master 指定我们Keepalive中配置的虚拟IP地址, 8443这个端口是我们之前在HAProxy中配置的。 ​ kubeadm 不支持将没有 --control-plane-endpoint 参数的单个控制平面集群转换为高可用性集群。 --service-cidr=10.96.0.0/12 : 指定service网段 --pod-network-cidr=10.244.0.0/16: 指定pod IP地址网段 --kubernetes-version 1.27.0 : 指定k8s版本 --upload-certs : 指定上传证书 ， 高可用集群建议指定。 如果不指定，也可以手动指定: 参考如下部分证书分配手册。 或者安装完成集群以后，重新上传证书： sudo kubeadm init phase upload-certs --upload-certs 这里有可能失败，如果失败就手动下载这个镜像：registry.k8s.io/pause:3.6 cat << 'EOF' > download.sh #!/bin/bash # 官方镜像列表 OFFICIAL_IMAGE_LIST=("$@") # 阿里Kubernetes官方惊镜像库 AILI_KUBERNETES_REGISTRY="registry.cn-hangzhou.aliyuncs.com/google_containers" echo "AILI_KUBERNETES_REGISTRY => ${AILI_KUBERNETES_REGISTRY}" # 下载并重命名镜像 function download_and_tag_image() { # 官方镜像全称： registry.k8s.io/xxx/xxx:xxx # 比如： registry.k8s.io/kube-proxy:v1.27.1 local full_official_image=$1 local ali_image ali_image=$(echo "$full_official_image" | sed -E "s|(.*/)(.*)|$AILI_KUBERNETES_REGISTRY/\2|") echo "downloading image => $ali_image" sudo docker pull "$ali_image" # 重命名镜像 echo "rename image $ali_image to $full_official_image" sudo docker tag "$ali_image" "$full_official_image" } for official_image in "${OFFICIAL_IMAGE_LIST[@]}"; do download_and_tag_image "$official_image" done EOF sudo chmod u+x ./download.sh && ./download.sh registry.k8s.io/pause:3.6 也有可能报错：Nameserver limits exceeded root@k8s-master01:/etc/kubernetes# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Tue 2023-05-30 00:34:59 CST; 3s ago Docs: https://kubernetes.io/docs/home/ Main PID: 7672 (kubelet) Tasks: 13 (limit: 13832) Memory: 25.7M CPU: 588ms CGroup: /system.slice/kubelet.service └─7672 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --containe> May 30 00:35:01 k8s-master01 kubelet[7672]: E0530 00:35:01.033186 7672 dns.go:158] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, > 解决：Nameserver limits exceeded ​ 参考： https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues root@k8s-master01:~# cat /etc/resolv.conf # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8). # Do not edit. # # This file might be symlinked as /etc/resolv.conf. If you're looking at # /etc/resolv.conf and seeing this text, you have followed the symlink. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs should typically not access this file directly, but only # through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a # different way, replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. ## 注释： 这里只有三条解析，我们做修改 nameserver 127.0.0.53 options edns0 trust-ad search . root@k8s-master01:~# cat /etc/systemd/resolved.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it under the # terms of the GNU Lesser General Public License as published by the Free # Software Foundation; either version 2.1 of the License, or (at your option) # any later version. # # Entries in this file show the compile time defaults. Local configuration # should be created by either modifying this file, or by creating "drop-ins" in # the resolved.conf.d/ subdirectory. The latter is generally recommended. # Defaults can be restored by simply deleting this file and all drop-ins. # # Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config. # # See resolved.conf(5) for details. ## 注释： 这个文件没有解析，不做修改 [Resolve] # Some examples of DNS servers which may be used for DNS= and FallbackDNS=: # Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com # Google: 8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google # Quad9: 9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net #DNS= #FallbackDNS= #Domains= #DNSSEC=no #DNSOverTLS=no #MulticastDNS=no #LLMNR=no #Cache=no-negative #CacheFromLocalhost=no #DNSStubListener=yes #DNSStubListenerExtra= #ReadEtcHosts=yes #ResolveUnicastSingleLabel=no root@k8s-master01:~# cat /run/systemd/resolve/resolv.conf # This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8). # Do not edit. # # This file might be symlinked as /etc/resolv.conf. If you're looking at # /etc/resolv.conf and seeing this text, you have followed the symlink. # # This is a dynamic resolv.conf file for connecting local clients directly to # all known uplink DNS servers. This file lists all configured search domains. # # Third party programs should typically not access this file directly, but only # through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a # different way, replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. # 这个文件有多条nameserver解析，我们注释其中的两条，因为我们不使用ip6,所以这里注释掉ip6解析 nameserver 192.168.0.1 nameserver 192.168.1.1 # 注释掉 # nameserver fe80::1%2 # Too many DNS servers configured, the following entries may be ignored. # 注释掉 # nameserver 240c::6666 search . root@k8s-master01:~# vim /run/systemd/resolve/resolv.conf # 重启 root@k8s-master01:~# systemctl restart kubelet # 重置kubeadm后,重新执行以上kubeadm init命令，即可解决 root@k8s-master01:~# kubeadm reset \ --cri-socket=unix:///var/run/cri-dockerd.sock [reset] Reading configuration from the cluster... [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' W0530 00:45:44.483361 37269 reset.go:106] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: configmaps "kubeadm-config" not found W0530 00:45:44.483476 37269 preflight.go:56] [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted. [reset] Are you sure you want to proceed? [y/N]: y [preflight] Running pre-flight checks W0530 00:45:46.625805 37269 removeetcdmember.go:106] [reset] No kubeadm config, using etcd pod spec to get data directory [reset] Deleted contents of the etcd data directory: /var/lib/etcd [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of directories: [/etc/kubernetes/manifests /var/lib/kubelet /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file. 重要的说明： 要重新配置一个已经创建的集群， 请参见重新配置一个 kubeadm 集群。 要再次运行 kubeadm init，你必须首先卸载集群。 配置用户可以使用kubectl 命令 非root用户执行（root用户也可以）： mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config root用户（如果执行完上面的语句， 下面可以不执行） export KUBECONFIG=/etc/kubernetes/admin.conf 验证kubectl 命令是否可用 root@k8s-master01:~# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane 21m v1.27.1 安装容器网络（CNI）插件 为了保证pod之间能够正常访问，需要安装容器网络插件。 注意： 每个集群只能安装一个 Pod 网络（容器网络插件）。 你必须部署一个基于 Pod 网络插件的 容器网络接口 (CNI)，以便你的 Pod 可以相互通信。 在安装网络之前，集群 DNS (CoreDNS) 将不会启动。 这里有多种网络插件：https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/addons/#networking-and-network-policy 我们选择： Calico root@k8s-master01:~# kubectl apply -f https://projectcalico.docs.tigera.io/archive/v3.25/manifests/calico.yaml poddisruptionbudget.policy/calico-kube-controllers created serviceaccount/calico-kube-controllers created serviceaccount/calico-node created configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created deployment.apps/calico-kube-controllers created 查看是否安装完成： root@k8s-master01:~# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-dmwg7 0/1 Pending 0 118s kube-system calico-node-t5bc7 0/1 Init:0/3 0 118s kube-system coredns-5d78c9869d-gm5vt 0/1 Pending 0 33m kube-system coredns-5d78c9869d-xgkbj 0/1 Pending 0 33m kube-system etcd-k8s-master 1/1 Running 0 34m kube-system kube-apiserver-k8s-master 1/1 Running 0 34m kube-system kube-controller-manager-k8s-master 1/1 Running 0 34m kube-system kube-proxy-d26m7 1/1 Running 0 33m kube-system kube-scheduler-k8s-master 1/1 Running 0 34m root@k8s-master01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-cfqg9 1/1 Running 0 41s kube-system calico-node-rczss 1/1 Running 0 41s kube-system coredns-5d78c9869d-gm5vt 1/1 Running 0 80m kube-system coredns-5d78c9869d-xgkbj 1/1 Running 0 80m kube-system etcd-k8s-master 1/1 Running 1 (10m ago) 80m kube-system kube-apiserver-k8s-master 1/1 Running 1 (10m ago) 80m kube-system kube-controller-manager-k8s-master 1/1 Running 1 (10m ago) 80m kube-system kube-proxy-d26m7 1/1 Running 1 (10m ago) 80m kube-system kube-scheduler-k8s-master 1/1 Running 1 (10m ago) 80m 需要等10多分钟。。。 最后 calico 相关的pod都 Running 状态， 并且都是 READY 1/1 ，表示安装完成， 可以看到 calico 安装完成以后， coredns 相关的pod也都是Running 状态了。 移除控制平面污点 默认控制平面会打上污点，pod是不会被调度到控制平面上的， 如果想要让pod调度到控制平面上，可以执行以下命令移除污点： root@k8s-master01:~# kubectl taint nodes --all node-role.kubernetes.io/control-plane- node/k8s-master untainted 7. Worker节点加入集群（所有Woker节点执行，必须使用root用户执行） ​ 我这里就是： k8s-slave-1 和 k8s-slave02 这两台集群执行： root@k8s-slave01:~# kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \ --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \ --cri-socket=unix:///var/run/cri-dockerd.sock [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. ​ 如果在等一段时间还是node还是NotReady状态，则使用在master节点中执行以下命令： root@k8s-master01:~# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 27m v1.27.1 k8s-master02 Ready control-plane 10m v1.27.1 k8s-master03 Ready control-plane 9m45s v1.27.1 k8s-slave01 NotReady <none> 49s v1.27.1 k8s-slave02 NotReady <none> 44s v1.27.1 root@k8s-master01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-fslqk 1/1 Running 0 23m kube-system calico-node-6wgk4 0/1 Init:Error 4 (76s ago) 2m37s kube-system calico-node-grb97 1/1 Running 0 11m kube-system calico-node-ltczv 0/1 Init:CrashLoopBackOff 4 (56s ago) 2m42s kube-system calico-node-pffcg 1/1 Running 0 12m kube-system calico-node-vtcqg 1/1 Running 0 23m kube-system coredns-5d78c9869d-m5zgd 1/1 Running 0 28m kube-system coredns-5d78c9869d-mnxzj 1/1 Running 0 28m kube-system etcd-k8s-master01 1/1 Running 0 29m kube-system etcd-k8s-master02 1/1 Running 0 12m kube-system etcd-k8s-master03 1/1 Running 0 11m kube-system haproxy-k8s-master01 1/1 Running 0 29m kube-system haproxy-k8s-master02 1/1 Running 0 12m kube-system haproxy-k8s-master03 1/1 Running 0 11m kube-system keepalived-k8s-master01 1/1 Running 0 29m kube-system keepalived-k8s-master02 1/1 Running 0 12m kube-system keepalived-k8s-master03 1/1 Running 0 10m kube-system kube-apiserver-k8s-master01 1/1 Running 0 29m kube-system kube-apiserver-k8s-master02 1/1 Running 0 12m kube-system kube-apiserver-k8s-master03 1/1 Running 1 (11m ago) 11m kube-system kube-controller-manager-k8s-master01 1/1 Running 1 (12m ago) 29m kube-system kube-controller-manager-k8s-master02 1/1 Running 0 12m kube-system kube-controller-manager-k8s-master03 1/1 Running 0 10m kube-system kube-proxy-lmw7g 1/1 Running 0 11m kube-system kube-proxy-mb8hx 0/1 ErrImagePull 0 2m42s kube-system kube-proxy-nvx8b 0/1 ImagePullBackOff 0 2m37s kube-system kube-proxy-phvcm 1/1 Running 0 28m kube-system kube-proxy-psst7 1/1 Running 0 12m kube-system kube-scheduler-k8s-master01 1/1 Running 1 (12m ago) 29m kube-system kube-scheduler-k8s-master02 1/1 Running 0 12m kube-system kube-scheduler-k8s-master03 1/1 Running 0 10m # 可以看到kube-proxy镜像可能现在失败，我们看一下Pod的详情 root@k8s-master01:~# kubectl describe pod kube-proxy-mb8hx -n kube-system # 在Events里面可以看到是应为镜像没有pull下来，我们去两个node节点手动下载一下镜像 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m57s default-scheduler Successfully assigned kube-system/kube-proxy-mb8hx to k8s-slave01 Warning Failed 3m24s kubelet Failed to pull image "registry.k8s.io/kube-proxy:v1.27.1": rpc error: code = Unknown desc = Error response from daemon: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy/manifests/v1.27.1": dial tcp 64.233.187.82:443: i/o timeout Normal Pulling 62s (x4 over 3m54s) kubelet Pulling image "registry.k8s.io/kube-proxy:v1.27.1" Warning Failed 31s (x4 over 3m24s) kubelet Error: ErrImagePull Warning Failed 31s (x3 over 2m42s) kubelet Failed to pull image "registry.k8s.io/kube-proxy:v1.27.1": rpc error: code = Unknown desc = Error response from daemon: Head "https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy/manifests/v1.27.1": dial tcp 64.233.188.82:443: i/o timeout Normal BackOff 5s (x6 over 3m23s) kubelet Back-off pulling image "registry.k8s.io/kube-proxy:v1.27.1" Warning Failed 5s (x6 over 3m23s) kubelet Error: ImagePullBackOff # k8s-slave01节点下载镜像 root@k8s-slave01:~# bash download.sh registry.k8s.io/kube-proxy:v1.27.1 AILI_KUBERNETES_REGISTRY => registry.cn-hangzhou.aliyuncs.com/google_containers downloading image => registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 v1.27.1: Pulling from google_containers/kube-proxy b6425c1785a5: Pull complete 5730c7a042b6: Pull complete Digest: sha256:958ddb03a4d4d7a567d3563c759a05f3e95aa42ca8af2964aa76867aafc43610 Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 rename image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 to registry.k8s.io/kube-proxy:v1.27.1 # k8s-slave02节点下载镜像 root@k8s-slave02:~# bash download.sh registry.k8s.io/kube-proxy:v1.27.1 AILI_KUBERNETES_REGISTRY => registry.cn-hangzhou.aliyuncs.com/google_containers downloading image => registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 v1.27.1: Pulling from google_containers/kube-proxy b6425c1785a5: Pull complete 5730c7a042b6: Pull complete Digest: sha256:958ddb03a4d4d7a567d3563c759a05f3e95aa42ca8af2964aa76867aafc43610 Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 rename image registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.1 to registry.k8s.io/kube-proxy:v1.27. # 看到kube-proxy已经Running并且READY 了 root@k8s-master01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-fslqk 1/1 Running 0 28m kube-system calico-node-6wgk4 0/1 Init:CrashLoopBackOff 6 (57s ago) 7m38s kube-system calico-node-grb97 1/1 Running 0 16m kube-system calico-node-ltczv 0/1 Init:CrashLoopBackOff 6 (79s ago) 7m43s kube-system calico-node-pffcg 1/1 Running 0 17m kube-system calico-node-vtcqg 1/1 Running 0 28m kube-system coredns-5d78c9869d-m5zgd 1/1 Running 0 33m kube-system coredns-5d78c9869d-mnxzj 1/1 Running 0 33m kube-system etcd-k8s-master01 1/1 Running 0 34m kube-system etcd-k8s-master02 1/1 Running 0 17m kube-system etcd-k8s-master03 1/1 Running 0 16m kube-system haproxy-k8s-master01 1/1 Running 0 34m kube-system haproxy-k8s-master02 1/1 Running 0 17m kube-system haproxy-k8s-master03 1/1 Running 0 16m kube-system keepalived-k8s-master01 1/1 Running 0 34m kube-system keepalived-k8s-master02 1/1 Running 0 17m kube-system keepalived-k8s-master03 1/1 Running 0 15m kube-system kube-apiserver-k8s-master01 1/1 Running 0 34m kube-system kube-apiserver-k8s-master02 1/1 Running 0 17m kube-system kube-apiserver-k8s-master03 1/1 Running 1 (16m ago) 16m kube-system kube-controller-manager-k8s-master01 1/1 Running 1 (17m ago) 34m kube-system kube-controller-manager-k8s-master02 1/1 Running 0 17m kube-system kube-controller-manager-k8s-master03 1/1 Running 0 15m kube-system kube-proxy-lmw7g 1/1 Running 0 16m kube-system kube-proxy-mb8hx 1/1 Running 0 7m43s kube-system kube-proxy-nvx8b 1/1 Running 0 7m38s kube-system kube-proxy-phvcm 1/1 Running 0 33m kube-system kube-proxy-psst7 1/1 Running 0 17m kube-system kube-scheduler-k8s-master01 1/1 Running 1 (17m ago) 34m kube-system kube-scheduler-k8s-master02 1/1 Running 0 17m kube-system kube-scheduler-k8s-master03 1/1 Running 0 15m # 但是现在还有Calico网络插件没有Running， 我们看一下原因 root@k8s-master01:~# kubectl describe pod calico-node-ltczv -n kube-system # Events看到镜像已经下载成功了， 但是Back-off了，这里我们重新启动一下 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 9m38s default-scheduler Successfully assigned kube-system/calico-node-ltczv to k8s-slave01 Normal Pulled 9m35s kubelet Container image "docker.io/calico/cni:v3.25.0" already present on machine Normal Created 9m35s kubelet Created container upgrade-ipam Normal Started 9m35s kubelet Started container upgrade-ipam Normal Pulled 7m55s (x5 over 9m35s) kubelet Container image "docker.io/calico/cni:v3.25.0" already present on machine Normal Created 7m55s (x5 over 9m35s) kubelet Created container install-cni Normal Started 7m54s (x5 over 9m34s) kubelet Started container install-cni Warning BackOff 4m32s (x22 over 9m27s) kubelet Back-off restarting failed container install-cni in pod calico-node-ltczv_kube-system(c89e2e76-5045-4474-af93-9b839e1d2206) # 重启一下Calico DaemonSet控制器控制的Pod root@k8s-master01:~# kubectl -n kube-system rollout restart DaemonSet/calico-node daemonset.apps/calico-node restarted root@k8s-master01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-fslqk 1/1 Running 0 33m kube-system calico-node-bbfx2 1/1 Running 0 15s kube-system calico-node-cf55q 0/1 Init:2/3 0 4s kube-system calico-node-ltczv 1/1 Running 0 12m kube-system calico-node-pffcg 1/1 Running 0 22m kube-system calico-node-vtcqg 1/1 Running 0 33m kube-system coredns-5d78c9869d-m5zgd 1/1 Running 0 38m kube-system coredns-5d78c9869d-mnxzj 1/1 Running 0 38m kube-system etcd-k8s-master01 1/1 Running 0 39m kube-system etcd-k8s-master02 1/1 Running 0 22m kube-system etcd-k8s-master03 1/1 Running 0 21m kube-system haproxy-k8s-master01 1/1 Running 0 39m kube-system haproxy-k8s-master02 1/1 Running 0 22m kube-system haproxy-k8s-master03 1/1 Running 0 21m kube-system keepalived-k8s-master01 1/1 Running 0 39m kube-system keepalived-k8s-master02 1/1 Running 0 22m kube-system keepalived-k8s-master03 1/1 Running 0 20m kube-system kube-apiserver-k8s-master01 1/1 Running 0 39m kube-system kube-apiserver-k8s-master02 1/1 Running 0 22m kube-system kube-apiserver-k8s-master03 1/1 Running 1 (21m ago) 21m kube-system kube-controller-manager-k8s-master01 1/1 Running 1 (22m ago) 39m kube-system kube-controller-manager-k8s-master02 1/1 Running 0 22m kube-system kube-controller-manager-k8s-master03 1/1 Running 0 20m kube-system kube-proxy-lmw7g 1/1 Running 0 21m kube-system kube-proxy-mb8hx 1/1 Running 0 12m kube-system kube-proxy-nvx8b 1/1 Running 0 12m kube-system kube-proxy-phvcm 1/1 Running 0 38m kube-system kube-proxy-psst7 1/1 Running 0 22m kube-system kube-scheduler-k8s-master01 1/1 Running 1 (22m ago) 39m kube-system kube-scheduler-k8s-master02 1/1 Running 0 22m kube-system kube-scheduler-k8s-master03 1/1 Running 0 20m # 过一会就都启动了 root@k8s-master01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-6c99c8747f-fslqk 1/1 Running 0 34m kube-system calico-node-82c5z 1/1 Running 0 48s kube-system calico-node-9vrzk 1/1 Running 0 37s kube-system calico-node-bbfx2 1/1 Running 0 69s kube-system calico-node-cf55q 1/1 Running 0 58s kube-system calico-node-scrp4 1/1 Running 0 26s kube-system coredns-5d78c9869d-m5zgd 1/1 Running 0 39m kube-system coredns-5d78c9869d-mnxzj 1/1 Running 0 39m kube-system etcd-k8s-master01 1/1 Running 0 40m kube-system etcd-k8s-master02 1/1 Running 0 23m kube-system etcd-k8s-master03 1/1 Running 0 22m kube-system haproxy-k8s-master01 1/1 Running 0 40m kube-system haproxy-k8s-master02 1/1 Running 0 23m kube-system haproxy-k8s-master03 1/1 Running 0 22m kube-system keepalived-k8s-master01 1/1 Running 0 40m kube-system keepalived-k8s-master02 1/1 Running 0 23m kube-system keepalived-k8s-master03 1/1 Running 0 20m kube-system kube-apiserver-k8s-master01 1/1 Running 0 40m kube-system kube-apiserver-k8s-master02 1/1 Running 0 23m kube-system kube-apiserver-k8s-master03 1/1 Running 1 (22m ago) 22m kube-system kube-controller-manager-k8s-master01 1/1 Running 1 (23m ago) 40m kube-system kube-controller-manager-k8s-master02 1/1 Running 0 23m kube-system kube-controller-manager-k8s-master03 1/1 Running 0 21m kube-system kube-proxy-lmw7g 1/1 Running 0 22m kube-system kube-proxy-mb8hx 1/1 Running 0 13m kube-system kube-proxy-nvx8b 1/1 Running 0 13m kube-system kube-proxy-phvcm 1/1 Running 0 39m kube-system kube-proxy-psst7 1/1 Running 0 23m kube-system kube-scheduler-k8s-master01 1/1 Running 1 (23m ago) 40m kube-system kube-scheduler-k8s-master02 1/1 Running 0 23m kube-system kube-scheduler-k8s-master03 1/1 Running 0 21m # 节点都是 Ready 状态 root@k8s-master01:~# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 41m v1.27.1 k8s-master02 Ready control-plane 24m v1.27.1 k8s-master03 Ready control-plane 24m v1.27.1 k8s-slave01 Ready <none> 15m v1.27.1 k8s-slave02 Ready <none> 15m v1.27.1 可能会出现镜像下载失败问题等等。 8. 高可用Master主节点加入集群（比如使用Root用户执行） 参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/high-availability/ ​ 如果使用kubeadm init时，没有指定--upload-certs 选项， 则可以现在重新配置上传证书阶段： root@k8s-master01:~# kubeadm init phase upload-certs --upload-certs [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: c82900d92a026aa6f6498b41ea70c9602e052c88eaca3e019d99b297af43230e 不要忘记默认情况下，--certificate-key 中的解密秘钥会在两个小时后过期。 ​ 我们这里就是root@k8s-master02和 root@k8s-master03 这台机器， 使用上面命令生成的 certificate key: c82900d92a026aa6f6498b41ea70c9602e052c88eaca3e019d99b297af43230e： root@k8s-master02:~# kubeadm join k8s-master:8443 --token th5i1f.fnzc9v0yb6z3aok8 \ --discovery-token-ca-cert-hash sha256:25357bff7f44a787886222dc9439916ab271dc5af5d5bbef274288fdd8e245b4 \ --control-plane \ --certificate-key 3636bc7d84515aeb36ca79597792b07cc64a888ebdea9221ab68a5bae93ac947 \ --cri-socket=unix:///var/run/cri-dockerd.sock [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks before initializing the new control plane instance [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' W0520 20:29:27.062790 14892 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0) W0520 20:29:27.337990 14892 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image. [download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki" [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master k8s-master02 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.20] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.0.20 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-master02 localhost] and IPs [192.168.0.20 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Valid certificates and keys now exist in "/etc/kubernetes/pki" [certs] Using the existing "sa" key [kubeconfig] Generating kubeconfig files [kubeconfig] Using kubeconfig folder "/etc/kubernetes" W0520 20:29:29.131836 14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file W0520 20:29:29.206366 14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file W0520 20:29:29.479200 14892 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [check-etcd] Checking that the etcd cluster is healthy [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [etcd] Announced new etcd member joining to the existing etcd cluster [etcd] Creating static Pod manifest for "etcd" W0520 20:29:31.931154 14892 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.1, falling back to the nearest etcd version (3.5.7-0) [etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master02 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] This node has joined the cluster and a new control plane instance was created: * Certificate signing request was sent to apiserver and approval was received. * The Kubelet was informed of the new secure connection details. * Control plane label and taint were applied to the new node. * The Kubernetes control plane instances scaled up. * A new etcd member was added to the local/stacked etcd cluster. To start administering your cluster from this node, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Run 'kubectl get nodes' to see this node join the cluster. 配置节点kubectl命令 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config 查看节点是否正常 （到此基本集群安装完成） root@k8s-master03:~# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane 18m v1.27.1 k8s-master02 Ready control-plane 90s v1.27.1 k8s-master03 Ready control-plane 37s v1.27.1 9. 可选操作 （可选）从控制平面节点以外的计算机控制集群 为了使 kubectl 在其他计算机（例如笔记本电脑）上与你的集群通信， 你需要将管理员 kubeconfig 文件从控制平面节点复制到工作站，如下所示： scp root@<control-plane-host>:/etc/kubernetes/admin.conf . kubectl --kubeconfig ./admin.conf get nodes 说明： 上面的示例假定为 root 用户启用了 SSH 访问。如果不是这种情况， 你可以使用 scp 将 admin.conf 文件复制给其他允许访问的用户。 admin.conf 文件为用户提供了对集群的超级用户特权。 该文件应谨慎使用。对于普通用户，建议生成一个你为其授予特权的唯一证书。 你可以使用 kubeadm alpha kubeconfig user --client-name <CN> 命令执行此操作。 该命令会将 KubeConfig 文件打印到 STDOUT，你应该将其保存到文件并分发给用户。 之后，使用 kubectl create (cluster)rolebinding 授予特权。 （可选）将 API 服务器代理到本地主机 如果要从集群外部连接到 API 服务器，则可以使用 kubectl proxy： scp root@<control-plane-host>:/etc/kubernetes/admin.conf . kubectl --kubeconfig ./admin.conf proxy 你现在可以在本地访问 API 服务器 http://localhost:8001/api/v1。

使用frp进行内网穿透，实现远程ssh连接内网主机 需求 我想在公司的网络环境中连接到家里主机进行远程操作。 目前的解决方案： 远程控制软件： TeamViewer 向日葵 QQ 等等... 内网穿透工具 frp (前提：需要一台具有公网IP的主机) ... 但是，由于我的需求只是需要远程SSH到家里的主机，进行控制台操作，基本不涉及到图形化界面，因此，这里我选择frp内网穿透（前提： 我目前已经有一台能够具有公网ip的阿里云服务器）, 顺便说一下， 现在阿里云云主机（99元一年，新老客户都可以）： https://www.aliyun.com/minisite/goods?userCode=ej7w2qnx。 frp简介 通过在具有公网 IP 的节点上部署 frp 服务端，您可以轻松地将内网服务穿透到公网，并享受以下专业特性： 多种协议支持：客户端服务端通信支持 TCP、QUIC、KCP 和 Websocket 等多种协议。 TCP 连接流式复用：在单个连接上承载多个请求，减少连接建立时间，降低请求延迟。 代理组间的负载均衡。 端口复用：多个服务可以通过同一个服务端端口暴露。 P2P 通信：流量不必经过服务器中转，充分利用带宽资源。 客户端插件：提供多个原生支持的客户端插件，如静态文件查看、- -HTTPS/HTTP 协议转换、HTTP、SOCKS5 代理等，以便满足各种需求。 服务端插件系统：高度可扩展的服务端插件系统，便于根据自身需求进行功能扩展。 用户友好的 UI 页面：提供服务端和客户端的用户界面，使配置和监控变得更加方便。 下面是frp一些相关链接： GitHub项目地址(Go语言实现)： https://github.com/fatedier/frp 文档地址： https://gofrp.org/zh-cn/docs/overview/ 实例配置： https://gofrp.org/zh-cn/docs/examples/ssh/ frp安装 frp是使用Go语言编写，支持跨平台的。 因此，只需要在具有公网IP的服务端下载软件和家里主机上下载进行简单配置即可。 目前（2023年11月26日13:01:37）， 最新版本为： v0.52.3, 因此我们使用这个版本。 注意： 服务器和家里主机的版本最好一致。 下载地址： https://github.com/fatedier/frp/releases/tag/v0.52.3 如果下载比较缓慢，请开启VPN进行下载。 公网服务器安装fpr 解压安装包[root@iZ2zeb2i87ctar29rg4v28Z frp]# pwd /root/frp [root@iZ2zeb2i87ctar29rg4v28Z frp]# ll 总用量 11508 -rw-r--r-- 1 root root 11781267 11月 26 13:20 frp_0.52.3_linux_amd64.tar.gz [root@iZ2zeb2i87ctar29rg4v28Z frp]# tar -zxvf frp_0.52.3_linux_amd64.tar.gz frp_0.52.3_linux_amd64/ frp_0.52.3_linux_amd64/LICENSE frp_0.52.3_linux_amd64/frps.toml frp_0.52.3_linux_amd64/frpc frp_0.52.3_linux_amd64/frpc.toml frp_0.52.3_linux_amd64/frps [root@iZ2zeb2i87ctar29rg4v28Z frp]# ll 总用量 11512 drwxr-xr-x 2 1001 docker 4096 10月 24 10:57 frp_0.52.3_linux_amd64 -rw-r--r-- 1 root root 11781267 11月 26 13:20 frp_0.52.3_linux_amd64.tar.gz [root@iZ2zeb2i87ctar29rg4v28Z frp]# 配置 这里不需要配置， 只是看一下frps.toml文件中默认绑定的端口为： 7000即可。 [root@iZ2zeb2i87ctar29rg4v28Z frp]# ls frp_0.52.3_linux_amd64 frp_0.52.3_linux_amd64.tar.gz [root@iZ2zeb2i87ctar29rg4v28Z frp]# cd frp_0.52.3_linux_amd64/ [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# ls frpc frpc.toml frps frps.toml LICENSE [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# cat frps.toml bindPort = 7000 [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# 使用Systemd管理frp 注意：将/root/frp/frp_0.52.3_linux_amd64 替换成你的目录。 sudo cat > /etc/systemd/system/frps.service << EOF [Unit] # 服务名称 Description = frp server After = network.target syslog.target Wants = network.target [Service] Type = simple # 启动frps的命令 ExecStart = /root/frp/frp_0.52.3_linux_amd64/frps -c /root/frp/frp_0.52.3_linux_amd64/frps.toml [Install] WantedBy = multi-user.target EOF 执行过程： [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# sudo cat > /etc/systemd/system/frps.service << EOF > [Unit] > # 服务名称 > Description = frp server > After = network.target syslog.target > Wants = network.target > > [Service] > Type = simple > # 启动frps的命令 > ExecStart = /root/frp/frp_0.52.3_linux_amd64/frps -c /root/frp/frp_0.52.3_linux_amd64/frps.toml > > [Install] > WantedBy = multi-user.target > EOF [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# cat /etc/systemd/system/frps.service [Unit] # 服务名称 Description = frp server After = network.target syslog.target Wants = network.target [Service] Type = simple # 启动frps的命令 ExecStart = /root/frp/frp_0.52.3_linux_amd64/frps -c /root/frp/frp_0.52.3_linux_amd64/frps.toml [Install] WantedBy = multi-user.target [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# 启动frp [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# sudo systemctl start frps [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# sudo systemctl status frps ● frps.service - frp server Loaded: loaded (/etc/systemd/system/frps.service; disabled; vendor preset: enabled) Active: active (running) since Sun 2023-11-26 13:32:30 CST; 10s ago Main PID: 93946 (frps) Tasks: 5 (limit: 11849) Memory: 8.7M CGroup: /system.slice/frps.service └─93946 /root/frp/frp_0.52.3_linux_amd64/frps -c /root/frp/frp_0.52.3_linux_amd64/frps.toml 11月 26 13:32:30 iZ2zeb2i87ctar29rg4v28Z systemd[1]: Started frp server. 11月 26 13:32:30 iZ2zeb2i87ctar29rg4v28Z frps[93946]: 2023/11/26 13:32:30 [I] [root.go:102] frps uses config file: /root/frp/frp_0.52.3_linux_amd64/frps.toml 11月 26 13:32:30 iZ2zeb2i87ctar29rg4v28Z frps[93946]: 2023/11/26 13:32:30 [I] [service.go:200] frps tcp listen on 0.0.0.0:7000 11月 26 13:32:30 iZ2zeb2i87ctar29rg4v28Z frps[93946]: 2023/11/26 13:32:30 [I] [root.go:111] frps started successfully [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# 开启开机启动 [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# sudo systemctl enable frps Created symlink /etc/systemd/system/multi-user.target.wants/frps.service → /etc/systemd/system/frps.service. [root@iZ2zeb2i87ctar29rg4v28Z frp_0.52.3_linux_amd64]# 配置云主机放开： 7000和6000端口 （略） 家里主机安装fpr 解压安装包 localhost:Documents zxy$ ls frp_0.52.3_darwin_amd64.tar.gz localhost:Documents zxy$ tar -xf frp_0.52.3_darwin_amd64.tar.gz localhost:Documents zxy$ ls frp_0.52.3_darwin_amd64.tar.gz frp_0.52.3_darwin_amd64 localhost:Documents zxy$ cd frp_0.52.3_darwin_amd64 localhost:frp_0.52.3_darwin_amd64 zxy$ ls LICENSE frpc frpc.toml frps frps.toml localhost:frp_0.52.3_darwin_amd64 zxy$ 配置 编辑 frpc.toml 文件，假设 frps 所在服务器的公网 IP 地址为 x.x.x.x。以下是示例配置： sudo cat > /Users/zxy/Documents/frp_0.52.3_darwin_amd64/frpc.toml << EOF serverAddr = "x.x.x.x" serverPort = 7000 [[proxies]] name = "ssh" type = "tcp" localIP = "127.0.0.1" localPort = 22 remotePort = 6000 EOF serverAddr frp服务端公网IP。 serverPort frp服务端端口。 localIP 和 localPort 配置为需要从公网访问的内网服务的地址和端口。 remotePort 表示在 frp 服务端监听的端口，访问此端口的流量将被转发到本地服务的相应端口。 启动 命令： ./frpc -c frpc.toml 过程： localhost:frp_0.52.3_darwin_amd64 zxy$ ./frpc -c frpc.toml 2023/11/26 13:48:29 [I] [root.go:139] start frpc service for config file [frpc.toml] 2023/11/26 13:48:29 [I] [service.go:299] [3d90255be67b269a] login to server success, get run id [3d90255be67b269a] 2023/11/26 13:48:29 [I] [proxy_manager.go:156] [3d90255be67b269a] proxy added: [ssh] 2023/11/26 13:48:29 [I] [control.go:173] [3d90255be67b269a] [ssh] start proxy success 家里电脑（MacOS）开启SSH访问 系统偏好设置 -> 共享 -> 勾选 远程登录 验证是否可以远程SSH连接 使用以下命令通过 SSH 访问内网机器，假设用户名为 test： ssh -o Port=6000 test@x.x.x.x frp 将请求发送到 x.x.x.x:6000 的流量转发到内网机器的 22 端口。 总结 到此，就可以完成远程ssh访问了。

Gitlab和Redmine集成问题追踪系统，Intellij IDEA中集成问题追踪系统 问题引入 技术工具 开发流程（大致） 痛点 解决 参考Apache Flink项目 Gitlab和Redmine集成 Intellij IDEA本地配置 总结 问题引入 技术工具 目前， 使用以下工具进行项目开发工作。 开发工具： Intellij IDEA 代码版本控制： Git 代码托管平台(内部搭建)： Gitlab Issue追踪系统： Redmine 开发流程（大致） 产品提出相关需求，并在Redmine上创建需求对应的Issue 技术人员给出实现需求的设计，并在Redmine上产品创建Issue下创建对应的子Issue。 技术人员进行开发，拉取分支（分支名包含Redmine中的Issue编号），开发完成。 提交测试 测试完成，产品验收。 上线 痛点 我想要的最终目的： 快速定位：开发人员可以根据Git的提交信息快速定位代码改动所属的需求。因此，我们在提交代码填写提交信息的时候，都会带上本地提交的所属的Redmine上创建的Issue编号。 该目的表现在以下两个方面： 本地：git提交的消息（包含Redmine的Issue编号）怎么和Redmine系统中的Issue进行关联。 Gitlab： git提交的消息（包含Redmine的Issue编号）怎么和Redmine系统中的Issue进行关联。 解决 参考Apache Flink项目 我发现Apache Flink项目是通过JIRA作为问题追踪系统，GitHub作为代码库托管平台，Git作为代码版本管理工具。这和我使用的工具很像。 于是，通过查询Gitlab文档发现它也是可以和Redmine进行集成的，让Redmine作为问题追踪系统。 这就解决了痛点中的2. 另外，我发现Apache Flink 项目在Intellij IDEA中的提交信息中包含JIRA上创建的ISSUE也是通过点击进入到ISSUE对应的页面的。查看Intellij IDEA文档得到结局方案。 Gitlab和Redmine集成 官网文档参考： https://docs.gitlab.com/ee/user/project/integrations/redmine.html#reference-redmine-issues-in-gitlab 使用Redmine作为Gitlab的外部问题追踪系统。需要进行以下配置： 进入你Gitlab项目界面 点击Setting -> 找到Integrations 滑动鼠标，点击选择： Redmine 勾选 Action复选框 填写以下几个必填项 Project URL： 填写你Redmine中为该Gitlab创建的项目地址 Issue地址： 这个地址就是你Issue页面的URL，只是把最后的数字替换成:id 新建Issue地址： 这个URL目前没有使用，在以后得版本中可能会被删除 假设： Redmine的域名为：https://redmine.example.com 配置Redmine的项目名为： gitlab 则，这里给出一个配置的具体示例 Project URL: https://redmine.example.com/projects/gitla Issue URL: https://redmine.example.com/issues/:id New Issue URL: https://redmine.example.com/projects/gitlab/issues/new 填写好以后， 点击： Test setting. 出现successful字样就是成功了。 点击： Save changes 保存 你配置好以后，就可以看到在你项目的左侧栏中会出现Redmine选项，点击可以进入你配置的Redmine项目链接。 配置完成以后，如何在Git commit提交的消息中引用呢？ 短引用（不推荐，会和Gitlab自身的Issue系统冲突，如果两个Issue编号相同，发生冲突后，以Gitlab内存为准）：#<ID>，比如：Redmine中的Issue编号为100, 则你可以在git提交信息中使用: #100 引用该Issue编号。 长引用（推荐）： <PROJECT>-<ID> ， 比如: 你配置的项目为gitlab, Redmine中的Issue编号为100, 则你可以在git提交信息中使用: GITLAB-100(注意：这里项目名需要全大写) 引用该Issue编号。 代码提交到Gitlab以后，就可以根据<PROJECT>-<ID>自动生成链接到Redmine的链接了。 Intellij IDEA本地配置 本地配置参考Intellij IDEA文档： https://www.jetbrains.com/help/idea/handling-issues.html 以下是配置步骤： Intellij IDEA -> Setting... (Mac系统： command + ,) Version Controll -> Issue Navigation -> 点击 + -> Add Issue Navigation Link  : 正则匹配Git commit提交信息里面的内容 Issue Link: 将匹配到的Issue ID内容， 添加这里配置的超链接。 Explame 这里是让你实时验证你上面填写的内容 - Issue ID ：输入待验证的内容 - Issue link : 显示通过以上填写的正则匹配到以后添加的链接地址 假设： 我们需要匹配提交信息中包含POLARIS-111这种格式的内容，匹配到以后给这个匹配内容添加一个超链接，连接到https://example.com/issue/111, 我们可以进行一下配置： 配置说明： POLARIS-(\d+): 这个正则匹配内容 https://example.com/issue/$1 : 这里是给上面匹配内容刚添加的超链接， 其中$1使用正则中反向引用，这里代码匹配到的数字，也就是111 点击 OK -> 点击 OK 查看Git提交历史记录 （Mac: command + 9） 可以看到已经添加上超链接了。 点击是可以跳转的。 总结 通过以上配置，我们将Gitlab和Redmine进行的集成，可以轻松的通过提交的消息包含Issue链接到Redmine系统。同时本地Intellij IDEA也可以此操作，可以很方便定位问题。

Typecho安装支持生成目录、流程图（Mermaid）、数学公式（MathJax）Markdown解析器插件Parsedown 需求 安装 下载插件到插件目录 解压，修改插件目录名 启用插件 修改文章，查看效果 一些其他设置 总结 需求 安装完成Typecho以后，使用Markdown写完博客发现页面不会自动生成文章的目录，google了一会，于是就找到这款Markdown解析器Parsedown。 下面介绍一下该解析器相比于默认解析器额外功能： 支持[TOC] 语法，生成文章目录 支持 MathJax 数学公式的$和$$语法 支持 Mermaid各种图表语法 安装 作者GitHub地址： https://github.com/mrgeneralgoo/typecho-markdown 作者博客地址： https://www.chengxiaobai.cn/php/markdown-parser-library.html 作者给的示例地址： https://www.chengxiaobai.cn/record/markdown-concise-grammar-manual.html 下载插件到插件目录 [root@iZ2zeb2i87ctar29rg4v28Z ~]# curl -SL https://github.com/mrgeneralgoo/typecho-markdown/archive/master.zip -O ~/typecho/php-fpm/typecho/usr/plugins/master.zip % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:01:33 --:--:-- 0 可以看到，我们已经等待了1分33秒，但是仍然没有下载一点。这是就不要等等待了，直接ctrl + c 取消吧（因为从github.com下载不使用科学上网会非常慢，有时还会失败）。 还是挂上科学上网， 在本地下载下载完成（直接将上面的下载链接复制到浏览器URL栏即可）。 下载完成以后，上传到typecho部署的插件目录： (base) iMac-2:~ xx$ scp -r ~/Downloads/typecho-markdown-master.zip root@99:~/typecho/php-fpm/typecho/usr/plugins typecho-markdown-master.zip 100% 20KB 721.8KB/s 00:00 (base) zxys-iMac-2:~ zxy$ 解压，修改插件目录名 [root@iZ2zeb2i87ctar29rg4v28Z plugins]# pwd /root/typecho/php-fpm/typecho/usr/plugins [root@iZ2zeb2i87ctar29rg4v28Z plugins]# [root@iZ2zeb2i87ctar29rg4v28Z plugins]# unzip typecho-markdown-master.zip Archive: typecho-markdown-master.zip e5dc8b7c6d7fb544e87da617b99a8febf0993264 creating: typecho-markdown-master/ inflating: typecho-markdown-master/CHANGELOG.md inflating: typecho-markdown-master/LICENSE.md inflating: typecho-markdown-master/Parsedown.php inflating: typecho-markdown-master/ParsedownExtension.php inflating: typecho-markdown-master/ParsedownExtra.php inflating: typecho-markdown-master/Plugin.php inflating: typecho-markdown-master/README.md [root@iZ2zeb2i87ctar29rg4v28Z plugins]# ll 总用量 36 drwxr-xr-x 2 82 82 4096 11月 21 16:00 HelloWorld drwxr-xr-x 2 root root 4096 6月 17 2022 typecho-markdown-master -rw-r--r-- 1 root root 20861 11月 21 19:44 typecho-markdown-master.zip [root@iZ2zeb2i87ctar29rg4v28Z plugins]# [root@iZ2zeb2i87ctar29rg4v28Z plugins]# mv typecho-markdown-master MarkdownParse [root@iZ2zeb2i87ctar29rg4v28Z plugins]# 启用插件 访问Typecho后台地址： https://timewentby.com/admin/plugins.php 找到MardownParsr， 然后点击启用 : 修改文章，查看效果 修改任意Markdown文章，在文章第一行新增： [TOC] 保存， 可以看到会自动生成文章目录。 比如这篇文章： https://timewentby.com/docker/install-docker-and-docker-compose-on-linux.html 文章内容： [TOC] ## 环境说明 - 阿里云云主机（99元一年，新老客户都可以）： https://www.aliyun.com/minisite/goods?userCode=ej7w2qnx - 系统(内核)： - 系统： Alibaba Cloud Linux release 3 (Soaring Falcon) - 内核： 5.10.134-15.al8.x86_64 ..... 省略很多文字 文章目录预览： 一些其他设置 查看设置 设置页面 默认开启[TOC]语法（文章目录） 默认开启MathJax支持 默认关闭Mermaid 总结 这款插件满足了我的需求，生成了文章目录，解决了问题。 另外，还额外支持数学公式和图表功能， 以后可以探索。 哈哈。

记录Linux安装docker和docker-compose过程 环境说明 安装docker 官网参考链接 安装（这里按照CenOS安装） 安装docker-compose 前提知识 官方参考链接 安装 环境说明 阿里云云主机（99元一年，新老客户都可以）： https://www.aliyun.com/minisite/goods?userCode=ej7w2qnx 系统(内核)： 系统： Alibaba Cloud Linux release 3 (Soaring Falcon) 内核： 5.10.134-15.al8.x86_64 [root@iZ2zeb2i87ctar29rg4v28Z ~]# cat /etc/redhat-release Alibaba Cloud Linux release 3 (Soaring Falcon) [root@iZ2zeb2i87ctar29rg4v28Z ~]# uname -r 5.10.134-15.al8.x86_64 [root@iZ2zeb2i87ctar29rg4v28Z ~]# uname -a Linux iZ2zeb2i87ctar29rg4v28Z 5.10.134-15.al8.x86_64 #1 SMP Thu Jul 20 00:44:04 CST 2023 x86_64 x86_64 x86_64 GNU/Linux [root@iZ2zeb2i87ctar29rg4v28Z ~]# CPU 核心： 2核 架构： x86_64 [root@iZ2zeb2i87ctar29rg4v28Z ~]# cat /proc/cpuinfo | grep processor processor : 0 processor : 1 [root@iZ2zeb2i87ctar29rg4v28Z ~]# uname -m x86_64 内存 : 2G [root@iZ2zeb2i87ctar29rg4v28Z ~]# free -h total used free shared buff/cache available Mem: 1.8Gi 170Mi 739Mi 1.0Mi 977Mi 1.5Gi Swap: 0B 0B 0B 带宽： 包月：3M 安装docker 官网参考链接 Centos系统 ： https://docs.docker.com/engine/install/centos/ Debian系统 ： https://docs.docker.com/engine/install/debian/ Ubuntu系统 ： https://docs.docker.com/engine/install/ubuntu/ Fedora系统 ： https://docs.docker.com/engine/install/fedora/ RHEL 系统 ： https://docs.docker.com/engine/install/rhel/ sles 系统 ： https://docs.docker.com/engine/install/sles/ Raspberry Pi OS (32-bit) 系统 ： https://docs.docker.com/engine/install/raspberry-pi-os/ 安装（这里按照CenOS安装） 防止之前安装过，先移除docker相关软件 执行命令： sudo yum remove docker \ docker-client \ docker-client-latest \ docker-common \ docker-latest \ docker-latest-logrotate \ docker-logrotate \ docker-engine 命令记录： [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo yum remove docker \ > docker-client \ > docker-client-latest \ > docker-common \ > docker-latest \ > docker-latest-logrotate \ > docker-logrotate \ > docker-engine 未找到匹配的参数: docker 未找到匹配的参数: docker-client 未找到匹配的参数: docker-client-latest 未找到匹配的参数: docker-common 未找到匹配的参数: docker-latest 未找到匹配的参数: docker-latest-logrotate 未找到匹配的参数: docker-logrotate 未找到匹配的参数: docker-engine 没有软件包需要移除。 依赖关系解决。 无需任何处理。 完毕！ 安装yum-utils包并设置docker安装库 sudo yum install -y yum-utils sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo yum install -y yum-utils 上次元数据过期检查：3:04:52 前，执行于 2023年11月21日 星期二 11时47分02秒。 软件包 yum-utils-4.0.21-14.1.al8.noarch 已安装。 依赖关系解决。 =================================================================================================================================================================================================== 软件包 架构 版本 仓库 大小 =================================================================================================================================================================================================== 升级: dnf-plugins-core noarch 4.0.21-19.al8 alinux3-updates 75 k python3-dnf-plugin-post-transaction-actions noarch 4.0.21-19.al8 alinux3-updates 57 k python3-dnf-plugins-core noarch 4.0.21-19.al8 alinux3-updates 261 k yum-utils noarch 4.0.21-19.al8 alinux3-updates 75 k 安装依赖关系: python3-systemd x86_64 234-8.2.al8 alinux3-os 81 k 事务概要 =================================================================================================================================================================================================== 安装 1 软件包 升级 4 软件包 总下载：550 k 下载软件包： (1/5): python3-dnf-plugin-post-transaction-actions-4.0.21-19.al8.noarch.rpm 4.7 MB/s | 57 kB 00:00 (2/5): python3-dnf-plugins-core-4.0.21-19.al8.noarch.rpm 11 MB/s | 261 kB 00:00 (3/5): yum-utils-4.0.21-19.al8.noarch.rpm 8.0 MB/s | 75 kB 00:00 (4/5): dnf-plugins-core-4.0.21-19.al8.noarch.rpm 1.3 MB/s | 75 kB 00:00 (5/5): python3-systemd-234-8.2.al8.x86_64.rpm 1.4 MB/s | 81 kB 00:00 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 总计 9.1 MB/s | 550 kB 00:00 运行事务检查 事务检查成功。 运行事务测试 事务测试成功。 运行事务 准备中 : 1/1 安装 : python3-systemd-234-8.2.al8.x86_64 1/9 升级 : python3-dnf-plugins-core-4.0.21-19.al8.noarch 2/9 升级 : dnf-plugins-core-4.0.21-19.al8.noarch 3/9 升级 : yum-utils-4.0.21-19.al8.noarch 4/9 升级 : python3-dnf-plugin-post-transaction-actions-4.0.21-19.al8.noarch 5/9 清理 : yum-utils-4.0.21-14.1.al8.noarch 6/9 清理 : dnf-plugins-core-4.0.21-14.1.al8.noarch 7/9 清理 : python3-dnf-plugin-post-transaction-actions-4.0.21-14.1.al8.noarch 8/9 清理 : python3-dnf-plugins-core-4.0.21-14.1.al8.noarch 9/9 运行脚本: python3-dnf-plugins-core-4.0.21-14.1.al8.noarch 9/9 验证 : python3-systemd-234-8.2.al8.x86_64 1/9 验证 : dnf-plugins-core-4.0.21-19.al8.noarch 2/9 验证 : dnf-plugins-core-4.0.21-14.1.al8.noarch 3/9 验证 : python3-dnf-plugin-post-transaction-actions-4.0.21-19.al8.noarch 4/9 验证 : python3-dnf-plugin-post-transaction-actions-4.0.21-14.1.al8.noarch 5/9 验证 : python3-dnf-plugins-core-4.0.21-19.al8.noarch 6/9 验证 : python3-dnf-plugins-core-4.0.21-14.1.al8.noarch 7/9 验证 : yum-utils-4.0.21-19.al8.noarch 8/9 验证 : yum-utils-4.0.21-14.1.al8.noarch 9/9 已升级: dnf-plugins-core-4.0.21-19.al8.noarch python3-dnf-plugin-post-transaction-actions-4.0.21-19.al8.noarch python3-dnf-plugins-core-4.0.21-19.al8.noarch yum-utils-4.0.21-19.al8.noarch 已安装: python3-systemd-234-8.2.al8.x86_64 完毕！ [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo 添加仓库自：https://download.docker.com/linux/centos/docker-ce.repo [root@iZ2zeb2i87ctar29rg4v28Z ~]# 安装最新社区版docker sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 注意： 这里需要验证查看指纹是否为：060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35 [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 上次元数据过期检查：0:00:22 前，执行于 2023年11月21日 星期二 14时53分34秒。 依赖关系解决。 =================================================================================================================================================================================================== 软件包 架构 版本 仓库 大小 =================================================================================================================================================================================================== 安装: containerd.io x86_64 1.6.24-3.1.el8 docker-ce-stable 34 M docker-buildx-plugin x86_64 0.11.2-1.el8 docker-ce-stable 13 M docker-ce x86_64 3:24.0.7-1.el8 docker-ce-stable 24 M docker-ce-cli x86_64 1:24.0.7-1.el8 docker-ce-stable 7.2 M docker-compose-plugin x86_64 2.21.0-1.el8 docker-ce-stable 13 M 安装依赖关系: fuse-overlayfs x86_64 1.11-1.0.1.al8 alinux3-updates 75 k fuse3 x86_64 3.3.0-16.al8 alinux3-updates 54 k fuse3-libs x86_64 3.3.0-16.al8 alinux3-updates 97 k libcgroup x86_64 0.41-19.2.al8 alinux3-os 70 k libslirp x86_64 4.4.0-1.al8 alinux3-updates 71 k slirp4netns x86_64 1.2.0-2.al8 alinux3-updates 54 k 安装弱的依赖: docker-ce-rootless-extras x86_64 24.0.7-1.el8 docker-ce-stable 4.9 M 事务概要 =================================================================================================================================================================================================== 安装 12 软件包 总下载：96 M 安装大小：368 M 下载软件包： (1/12): libcgroup-0.41-19.2.al8.x86_64.rpm 4.7 MB/s | 70 kB 00:00 (2/12): fuse3-libs-3.3.0-16.al8.x86_64.rpm 8.5 MB/s | 97 kB 00:00 (3/12): libslirp-4.4.0-1.al8.x86_64.rpm 7.6 MB/s | 71 kB 00:00 (4/12): slirp4netns-1.2.0-2.al8.x86_64.rpm 8.2 MB/s | 54 kB 00:00 (5/12): fuse-overlayfs-1.11-1.0.1.al8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 (6/12): fuse3-3.3.0-16.al8.x86_64.rpm 363 kB/s | 54 kB 00:00 (7/12): docker-buildx-plugin-0.11.2-1.el8.x86_64.rpm 3.7 MB/s | 13 MB 00:03 (8/12): docker-ce-cli-24.0.7-1.el8.x86_64.rpm 4.1 MB/s | 7.2 MB 00:01 (9/12): docker-ce-24.0.7-1.el8.x86_64.rpm 3.9 MB/s | 24 MB 00:06 (10/12): docker-ce-rootless-extras-24.0.7-1.el8.x86_64.rpm 3.4 MB/s | 4.9 MB 00:01 (11/12): containerd.io-1.6.24-3.1.el8.x86_64.rpm 4.2 MB/s | 34 MB 00:08 (12/12): docker-compose-plugin-2.21.0-1.el8.x86_64.rpm 4.0 MB/s | 13 MB 00:03 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 总计 10 MB/s | 96 MB 00:09 Docker CE Stable - x86_64 2.9 kB/s | 1.6 kB 00:00 导入 GPG 公钥 0x621E9F35: Userid: "Docker Release (CE rpm) <docker@docker.com>" 指纹: 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35 来自: https://download.docker.com/linux/centos/gpg 导入公钥成功 运行事务检查 事务检查成功。 运行事务测试 事务测试成功。 运行事务 准备中 : 1/1 安装 : docker-compose-plugin-2.21.0-1.el8.x86_64 1/12 运行脚本: docker-compose-plugin-2.21.0-1.el8.x86_64 1/12 安装 : fuse3-libs-3.3.0-16.al8.x86_64 2/12 运行脚本: fuse3-libs-3.3.0-16.al8.x86_64 2/12 安装 : fuse3-3.3.0-16.al8.x86_64 3/12 安装 : fuse-overlayfs-1.11-1.0.1.al8.x86_64 4/12 运行脚本: fuse-overlayfs-1.11-1.0.1.al8.x86_64 4/12 安装 : docker-buildx-plugin-0.11.2-1.el8.x86_64 5/12 运行脚本: docker-buildx-plugin-0.11.2-1.el8.x86_64 5/12 安装 : docker-ce-cli-1:24.0.7-1.el8.x86_64 6/12 运行脚本: docker-ce-cli-1:24.0.7-1.el8.x86_64 6/12 安装 : containerd.io-1.6.24-3.1.el8.x86_64 7/12 运行脚本: containerd.io-1.6.24-3.1.el8.x86_64 7/12 安装 : libslirp-4.4.0-1.al8.x86_64 8/12 安装 : slirp4netns-1.2.0-2.al8.x86_64 9/12 运行脚本: libcgroup-0.41-19.2.al8.x86_64 10/12 安装 : libcgroup-0.41-19.2.al8.x86_64 10/12 运行脚本: libcgroup-0.41-19.2.al8.x86_64 10/12 安装 : docker-ce-rootless-extras-24.0.7-1.el8.x86_64 11/12 运行脚本: docker-ce-rootless-extras-24.0.7-1.el8.x86_64 11/12 安装 : docker-ce-3:24.0.7-1.el8.x86_64 12/12 运行脚本: docker-ce-3:24.0.7-1.el8.x86_64 12/12 验证 : libcgroup-0.41-19.2.al8.x86_64 1/12 验证 : fuse-overlayfs-1.11-1.0.1.al8.x86_64 2/12 验证 : fuse3-3.3.0-16.al8.x86_64 3/12 验证 : fuse3-libs-3.3.0-16.al8.x86_64 4/12 验证 : libslirp-4.4.0-1.al8.x86_64 5/12 验证 : slirp4netns-1.2.0-2.al8.x86_64 6/12 验证 : containerd.io-1.6.24-3.1.el8.x86_64 7/12 验证 : docker-buildx-plugin-0.11.2-1.el8.x86_64 8/12 验证 : docker-ce-3:24.0.7-1.el8.x86_64 9/12 验证 : docker-ce-cli-1:24.0.7-1.el8.x86_64 10/12 验证 : docker-ce-rootless-extras-24.0.7-1.el8.x86_64 11/12 验证 : docker-compose-plugin-2.21.0-1.el8.x86_64 12/12 已安装: containerd.io-1.6.24-3.1.el8.x86_64 docker-buildx-plugin-0.11.2-1.el8.x86_64 docker-ce-3:24.0.7-1.el8.x86_64 docker-ce-cli-1:24.0.7-1.el8.x86_64 docker-ce-rootless-extras-24.0.7-1.el8.x86_64 docker-compose-plugin-2.21.0-1.el8.x86_64 fuse-overlayfs-1.11-1.0.1.al8.x86_64 fuse3-3.3.0-16.al8.x86_64 fuse3-libs-3.3.0-16.al8.x86_64 libcgroup-0.41-19.2.al8.x86_64 libslirp-4.4.0-1.al8.x86_64 slirp4netns-1.2.0-2.al8.x86_64 完毕！ [root@iZ2zeb2i87ctar29rg4v28Z ~]# 如果是安装指定版本， 参考这里： 查看可用版本 $ yum list docker-ce --showduplicates | sort -r docker-ce.x86_64 3:24.0.0-1.el8 docker-ce-stable docker-ce.x86_64 3:23.0.6-1.el8 docker-ce-stable 安装指定版本： <VERSION_STRING> : 3:24.0.0-1.el8 sudo yum install -y docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io docker-buildx-plugin docker-compose-plugin 启动docker [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo systemctl start docker [root@iZ2zeb2i87ctar29rg4v28Z ~]# 设置开机启动 [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo systemctl enable docker [root@iZ2zeb2i87ctar29rg4v28Z ~]# 查看docker版本（目前2023-11-21 14:58:51 最新版本： 24.0.7） 恭喜你， 到这里就是安装成功了。 [root@iZ2zeb2i87ctar29rg4v28Z ~]# docker version Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:09:18 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:20 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.24 GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523 runc: Version: 1.1.9 GitCommit: v1.1.9-0-gccaecfc docker-init: Version: 0.19.0 GitCommit: de40ad0 安装docker-compose 前提知识 最好已经安装完成docker docker-compose安装方式 通过docker桌面版安装 作为docker插件方式安装 单独安装（我们采用这种方式） 官方参考链接 参考官网： https://docs.docker.com/compose/install/standalone/ 安装 下载可执行二进制文件 [root@iZ2zeb2i87ctar29rg4v28Z ~]# curl -SL https://github.com/docker/compose/releases/download/v2.23.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 7 56.8M 7 4242k 0 0 20453 0 0:48:33 0:03:32 0:45:01 26892 可以看到，在国内下载的数据是在是太慢了。 也可以在本地科学上网下载完成之后，上传到服务器上。 本地执行上传： (base) iMac-2:~ xx$ scp -r ~/Downloads/docker-compose-linux-x86_64 root@99:/usr/local/bin/ docker-compose-linux-x86_64 100% 57MB 961.6KB/s 01:00 (base) zxys-iMac-2:~ zxy$ 服务器： [root@iZ2zeb2i87ctar29rg4v28Z ~]# mv /usr/local/bin/docker-compose-linux-x86_64 /usr/local/bin/docker-compose [root@iZ2zeb2i87ctar29rg4v28Z ~]# [root@iZ2zeb2i87ctar29rg4v28Z ~]# [root@iZ2zeb2i87ctar29rg4v28Z ~]# chmod 755 /usr/local/bin/docker-compose [root@iZ2zeb2i87ctar29rg4v28Z ~]# [root@iZ2zeb2i87ctar29rg4v28Z ~]# sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose [root@iZ2zeb2i87ctar29rg4v28Z ~]# 查看是否安装成功 [root@iZ2zeb2i87ctar29rg4v28Z ~]# docker-compose -v Docker Compose version v2.23.1 [root@iZ2zeb2i87ctar29rg4v28Z ~]# 恭喜?， 出现版本信息就是成功了， 到此，结束。