K8s --The connection to the server 192.168.56.120:6443 was refused报错解决

🕗 发布于 2024-07-18 09:10 kubernetes 容器云原生

K8s The connection to the server 192.168.56.120:6443 was refused报错解决

背景说明
环境说明
部署信息
场景复现
问题排查
- 基础环境排查
- 组件服务排查
根因定位
问题解决

背景说明

自己本地虚拟机有一套使用kuebadm部署的3节点k8s集群，因为很长时间不使用了，今天登陆后在使用kubectl命令时一直报The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port?错误，接下来就将解决的整个过程分享给大家。

环境说明

操作系统信息以及k8s集群版本信息请看下面

[root@k8s-master][/usr/local]
$cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)

[root@k8s-master][/usr/local]
$kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

部署信息

序号	ip	角色
1	192.168.56.120	k8s-master
2	192.168.56.121	k8s-node1
3	192.168.56.122	k8s-node2

场景复现

[root@k8s-master][/usr/local]
$kubectl get node
The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port?

问题排查

基础环境排查

#1 基础环境排查
#1.0 端口排查（端口已经不存在）
[root@k8s-master][~]
$ss -antpl |grep 6443

#1.1 查看内存（都够用）
[root@k8s-master][~]
$free -h
              total        used        free      shared  buff/cache   available
Mem:           4.7G        1.2G        366M         58M        3.1G        3.1G
Swap:            0B          0B          0B

#1.2 查看存储（也没问题）
[root@k8s-master][~]
$df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 2.4G     0  2.4G   0% /dev
tmpfs                    2.4G     0  2.4G   0% /dev/shm
tmpfs                    2.4G   58M  2.3G   3% /run
tmpfs                    2.4G     0  2.4G   0% /sys/fs/cgroup
/dev/mapper/centos-root   37G   22G   16G  59% /

#1.3 查看防火墙（防火墙是关闭的）
[root@k8s-master][~]
$systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

组件服务排查

#2 排查组件
#2.1 查看docker信息（没有报错信息）
[root@k8s-master][~]
$systemctl status docker

#2.2 查看kubelet（发现master节点丢失的信息，但是内容不详细）
[root@k8s-master][/usr/local]
$systemctl status kubelet

Jul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.603977   14485 kubelet.go:2422] "Error getting node" err="node \"k8s-master\" not found"
Jul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.634017   14485 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://192.168....ion refused

#3 查看etcd情况
[root@k8s-master][~]
$docker ps |grep k8s_etcd_etcd

#从以下的信息可以看出证书已经过期了，在确认一遍
[root@k8s-master][~]
$docker logs dc2c0e42972e
{"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"}

#4 查看证书过期情况（发现已经过期了，到此找到报错根因）
[root@k8s-master][~]
$kubeadm certs check-expiration

根因定位

#从以下的信息可以看出证书过期，导致api-server服务未启动
[root@k8s-master][~]
$docker logs dc2c0e42972e
{"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"}

问题解决

既然已经定位到是证书过期的问题，那接下来就针对证书过期进行解决。

安装go环境

# 如果 k8s 版本较高，则需要 go 语言版本 1.19 以上(尽量使用和k8s版本一致的go版本，要不会报很多错误)
kubectl version
[root@k8s-master][~]
$kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}


#1  下载 go 语言软件包
mkdir -p /root/pac
cd /root/pac
wget https://studygolang.com/dl/golang/go1.17.3.linux-amd64.tar.gz

#2 解压并配置环境变量
tar xf go1.19.1.linux-amd64.tar.gz -C /usr/local

vim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/go

source /etc/profile

#3 验证
[root@k8s-master][~]
$go version
go version go1.17.3 linux/amd64

下载k8s源码

#1 查看当前环境 k8s 版本
kubectl version

#2 下载源码
cd /root/pac
wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.23.0.tar.gz

#2 解压
tar xf v1.23.0.tar.gz

修改CA有效期

#1 进入到指定目录
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cd /root/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert

#2 备份cert.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cp cert.go{,.bak}

#3 修改cert.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# vim cert.go

// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {
        now := time.Now()
        tmpl := x509.Certificate{
                SerialNumber: new(big.Int).SetInt64(0),
                Subject: pkix.Name{
                        CommonName:   cfg.CommonName,
                        Organization: cfg.Organization,
                },
                DNSNames:              []string{cfg.CommonName},
                NotBefore:             now.UTC(),
                //NotAfter:              now.Add(duration365d * 10).UTC(),
                // 修改为 100 年，默认为 10 年
                NotAfter:              now.Add(duration365d * 100).UTC(),
                KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
                BasicConstraintsValid: true,
                IsCA:                  true,
        }

        certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)
        if err != nil {
                return nil, err
        }
        return x509.ParseCertificate(certDERBytes)
}

修改证书有效期

[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cd /root/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants

#2 备份constants.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants]
# cp constants.go{,.bak}

#3 修改文件
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants]
# vim constants.go

const (
        // KubernetesDir is the directory Kubernetes owns for storing various configuration files
        KubernetesDir = "/etc/kubernetes"
        // ManifestsSubDirName defines directory name to store manifests
        ManifestsSubDirName = "manifests"
        // TempDirForKubeadm defines temporary directory for kubeadm
        // should be joined with KubernetesDir.
        TempDirForKubeadm = "tmp"

        // CertificateValidity defines the validity for all the signed certificates generated by kubeadm
        //CertificateValidity = time.Hour * 24 * 365
        //修改为 100 年，默认为 1 年
        CertificateValidity = time.Hour * 24 * 365 * 100
        // DefaultCertificateDir defines default certificate directory
        DefaultCertificateDir = "pki"

        // CACertAndKeyBaseName defines certificate authority base name
        CACertAndKeyBaseName = "ca"
        // CACertName defines certificate name
        CACertName = "ca.crt"
        // CAKeyName defines certificate name

编译 kubeadm

# 提前装rsync
yum install rsync -y

#1 重新编译kubeadm
[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# pwd
/root/xxx/go/kubernetes-1.23.0

[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# make all WHAT=cmd/kubeadm GOFLAGS=-v

#2 替换原来的 kubeadm
[root@k8s-master][~]
# mv /usr/bin/kubeadm{,.bak}

[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# pwd
/root/xxx/go/kubernetes-1.23.0

[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# ll _output/local/bin/linux/amd64/kubeadm
-rwxr-xr-x. 1 root root 43667456 Mar  6 11:26 _output/local/bin/linux/amd64/kubeadm

[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm


[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# chmod +x /usr/bin/kubeadm

更新证书

#1 备份原文件
[root@k8s-master][~]
# cp -a /etc/kubernetes{,.bak_20240717}


#2 更新证书
[root@k8s-master][/etc/kubernetes]
# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

检查结果

[root@k8s-master][~]
$kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 23, 2124 02:49 UTC   99y                                     no      
apiserver                  Jun 23, 2124 02:49 UTC   99y             ca                      no      
apiserver-etcd-client      Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Jun 23, 2124 02:49 UTC   99y             ca                      no      
controller-manager.conf    Jun 23, 2124 02:49 UTC   99y                                     no      
etcd-healthcheck-client    Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
etcd-peer                  Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
etcd-server                Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
front-proxy-client         Jun 23, 2124 02:49 UTC   99y             front-proxy-ca          no      
scheduler.conf             Jun 23, 2124 02:49 UTC   99y                                     no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 12, 2033 07:34 UTC   8y              no      
etcd-ca                 Mar 12, 2033 07:34 UTC   8y              no      
front-proxy-ca          Mar 12, 2033 07:34 UTC   8y              no

再次使用kubectl命令，看看问题是否解决

[root@k8s-master][~]
$kubectl get node
NAME         STATUS   ROLES                  AGE    VERSION
k8s-master   Ready    control-plane,master   489d   v1.23.0
k8s-node1    Ready    <none>                 489d   v1.23.0
k8s-node2    Ready    <none>                 489d   v1.23.0

至此，问题得到解决。

原文地址：https://blog.csdn.net/xgysimida/article/details/140493190

免责声明：本站文章内容转载自网络资源，如本站内容侵犯了原著者的合法权益，可联系本站删除。更多内容请关注自学内容网（zxcms.com）！

上一篇：查找运行中 sql中bind variable value 绑定变量值
下一篇：代码随想录训练营第40天 198打家劫舍 213打家劫舍II 337打家劫舍III

如何构建安全可靠的 HarmonyOS 应用
本文将深入探讨 HarmonyOS App 的安全编码规范与最佳实践，帮助开发者在代码编写中避免常见的安全漏洞，如 SQL 注入、XSS攻击等。我们将提供具体的编码示例，并结合ArkUI和ArkTS实
阅读更多2024-11-16
js像循环数组那样循环一个数字，Array.from()
js像循环数组那样循环一个数字，Array.from()
阅读更多2024-11-16
【C++笔记】vector使用详解及模拟实现
vector的文档使用STL的三个境界：能用、明理、能扩展，下面学习vector，我们也按照这个境界去学习。vector是可以改变大小的数组序列容器，也就是数据结构的顺序表。构造函数声明接口说明vec
阅读更多2024-11-16
Java线程池：ThreadPoolExecutor原理解析
本文介绍了线程池的基本概念、主要参数、工作流程，以及 execute() 方法的源码分析，此外，还讨论了在实际应用中可能遇到的陷阱和问题。
阅读更多2024-11-16
1.两数之和-力扣（LeetCode）
1.两数之和-力扣（LeetCode）
阅读更多2024-11-16
Xss挑战（跨脚本攻击）
这里将script，on，src，data，href，进行了过滤，并且在尝试的时候关键字双写不能用了，那么这里直接选择不去闭合标签，直接使用伪协议，但是发现javascript也被拆开了，这里可以对伪
阅读更多2024-11-16
《Python 网络爬虫》
本文介绍了 Python 网络爬虫的基本概念、技术原理、常用工具以及实战案例，希望能够帮助读者快速掌握 Python 网络爬虫技术。在实际应用中，需要根据具体的需求选择合适的工具和方法，并注意遵守法律
阅读更多2024-11-16
UEFI学习（五）——启动框架
https://www.zhihu.com/question/36313402/answer/2398532123UEFI（统一可扩展固件接口）在启动过程主要有以下几个阶段：
阅读更多2024-11-16
【洛谷】T539820 202411A Giants
C++ LGR-207-Div.4】洛谷入门赛 #29 第一题
阅读更多2024-11-16
Python 正则表达式进阶用法：量词与范围
匹配前面的字符零次或多次，相当于“任意多次”。：匹配前面的字符一次或多次，相当于“至少一次”。?：匹配前面的字符零次或一次，相当于“可有可无”。{n}：匹配前面的字符n次。{n,}：匹配前面的字符至少
阅读更多2024-11-16