在使用kubenetes的过程中,如何将服务开放到集群外部访问是一个重要的问题。当使用云平台(阿里云、腾讯云、AWS等)的容器服务时,我们可以通过配置 service 为 LoadBalancer 模式来绑定云平台的负载均衡器,从而实现外网的访问。但是,如果对于自建的 kubernetes裸机集群,这个问题则要麻烦的多。
祼机集群不支持负载均衡的方式,可用的不外乎NodePort、HostNetwork、ExternalIPs等方式来实现外部访问。但这些方式并不完美,他们或多或少都存在的一些缺点,这使得裸机集群成为Kubernetes生态系统中的二等公民。
MetalLB 旨在通过提供与标准网络设备集成的Network LB实施来解决这个痛点,从而使裸机群集上的外部服务也尽可能“正常运行”,减少运维上的管理成本。它是一种纯软件的解决方案,参考 https://kubernetes.github.io/ingress-nginx/deploy/baremetal/。
从 v0.13.0 版本开始,官方对解决方案进行了部分调整,操作步骤简洁一些,建议使用最新版本,参考官方教程 https://metallb.universe.tf/installation/
部署
- 创建namespace
$ kubectl create namespace metallb-system
2. 新建 secret
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
参数 from 前面是两个-
3. 部署
root@sxf-virtual-machine:~# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/metallb.yaml Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/controller created podsecuritypolicy.policy/speaker created serviceaccount/controller created serviceaccount/speaker created clusterrole.rbac.authorization.k8s.io/metallb-system:controller created clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created role.rbac.authorization.k8s.io/config-watcher created role.rbac.authorization.k8s.io/pod-lister created role.rbac.authorization.k8s.io/controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created rolebinding.rbac.authorization.k8s.io/config-watcher created rolebinding.rbac.authorization.k8s.io/pod-lister created rolebinding.rbac.authorization.k8s.io/controller created daemonset.apps/speaker created deployment.apps/controller created
查看部署结果
root@sxf-virtual-machine:~# kubectl get pod -n metallb-system NAME READY STATUS RESTARTS AGE controller-6cc57c4567-cn766 1/1 Running 0 29m speaker-4fv86 1/1 Running 0 32m
root@sxf-virtual-machine:~# kubectl get sa -n metallb-system NAME SECRETS AGE controller 1 33m default 1 33m speaker 1 33m
- 配置 MetalLB
配置 MetalLB 为 Layer 2模式 (使用 yaml 文件部署), 文件 MetalLB-Layer2-Configmap.yaml
kind: ConfigMap apiVersion: v1 metadata: name: config namespace: metallb-system data: config: | address-pools: - name: default protocol: layer2 addresses: - 192.168.3.251-192.168.3.254
这里配置一个由 MetalLB 二层模式控制的 service 外部 IP 段为 192.168.3.251 - 192.168.3.254
。
还可以指定多个网段
addresses: - 192.168.12.0/24 - 192.168.144.0/20
除了自动分配IP外,Metallb 还支持在定义服务的时候,通过 spec.loadBalancerIP
指定一个静态IP 。
$ kubectl apply -f MetalLB-Layer2-Configmap.yaml
使用kubectl log -f [matellb-contoller-pod]能看到配置更新过程
查看结果
root@sxf-virtual-machine:~# kubectl get cm -n metallb-system NAME DATA AGE config 1 22s kube-root-ca.crt 1 43m
从版本 v0.13.0 版本开始使用 IPAddressPool 代替,参考 https://metallb.universe.tf/configuration/_advanced_ipaddresspool_config/
测试
创建一个Nginx服务,服务类型为LoadBalancer:
deploy.yaml:
apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.23-alpine ports: - name: http containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx spec: #loadBalancerIP: x.y.z.a # 指定公网IP ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: LoadBalancer
服务创建,并查看服务信息:
$ kubectl apply -f deploy.yaml
查看pod
root@sxf-virtual-machine:~# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-8c9df995d-pvmkl 1/1 Running 0 2m10s
查看服务
root@sxf-virtual-machine:~# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default details ClusterIP 10.110.60.221 <none> 9080/TCP 3d1h default hostnames ClusterIP 10.111.190.216 <none> 80/TCP 3d1h default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d23h default nginx LoadBalancer 10.108.230.26 192.168.3.253 80:30829/TCP 65s default productpage ClusterIP 10.98.222.204 <none> 9080/TCP 3d1h default ratings ClusterIP 10.104.86.58 <none> 9080/TCP 3d1h default reviews ClusterIP 10.101.246.133 <none> 9080/TCP 3d1h istio-system istio-eastwestgateway LoadBalancer 10.99.130.139 192.168.3.252 15021:31544/TCP,15443:32575/TCP,15012:30747/TCP,15017:30099/TCP 3d istio-system istio-ingressgateway LoadBalancer 10.96.196.70 192.168.3.251 15021:30035/TCP,80:31245/TCP,443:31639/TCP 3d22h istio-system istiod ClusterIP 10.107.246.136 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 3d23h kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d23h
这里就发现一些 LoadBalancer 类型的服务,被分配到了固定的 IP地址信息。
服务自动生成的 yaml 文件
root@sxf-virtual-machine:~# kubectl get svc nginx -o yaml apiVersion: v1 kind: Service metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"nginx","namespace":"default"},"spec":{"ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":80}],"selector":{"app":"nginx"},"type":"LoadBalancer"}} creationTimestamp: "2021-09-29T11:26:35Z" name: nginx namespace: default resourceVersion: "1274438" uid: f2fb62dc-e277-4432-8496-2b5fb5984a55 spec: clusterIP: 10.108.230.26 clusterIPs: - 10.108.230.26 externalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: http nodePort: 30355 port: 80 protocol: TCP targetPort: 80 selector: app: nginx sessionAffinity: None type: LoadBalancer status: loadBalancer: ingress: - ip: 192.168.3.253
这时如果你 ping
这个 ip 的话,会发现无法 ping 通
root@sxf-virtual-machine:~# curl 192.168.3.253 PING 192.168.3.253 (192.168.3.253): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3
这个是正常的,因为它是一个虚拟IP地址,所以根本无法ping 通(此虚拟IP与物理网卡共用同一个 MAC
地址,那么这个IP是如何工作的呢?又是如何收到流量请求的呢?很值得思考)。
那如何测试这个虚拟IP是否能正常提供服务呢,其实只需要使用 telnet 命令就可以了,如
root@sxf-virtual-machine:~# telnet 192.168.3.253 80
这里我们提供的是一个 web
服务,暴露的监听端口是 80
,对应容器的端口为 30829
。
root@sxf-virtual-machine:~# telnet 192.168.3.253 80 Trying 192.168.3.253... Connected to 192.168.3.253. Escape character is '^]'.
看到这个信息则说明此虚拟IP可以正常提供服务了。
这里我们用 curl 192.168.3.253
再次验证一下nginx服务,就会发现返回了 Nginx 的欢迎信息。
root@sxf-virtual-machine:~# curl 192.168.3.253 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
可以看到可以正常访问pod的内容。
剩下的就是我们如何使用了,经常部署方案是为集群服务创建一个 Ingress 的服务,参考 https://stackoverflow.com/questions/65789968/microk8s-metallb-ingress
最后我们清理pod
root@sxf-virtual-machine:~# kubectl delete -f deploy.yaml deployment.apps "nginx" deleted service "nginx" deleted
参考
- https://github.com/metallb/metallb
- https://metallb.universe.tf/usage/
- https://kind.sigs.k8s.io/docs/user/loadbalancer/
- https://stackoverflow.com/questions/65789968/microk8s-metallb-ingress
- https://kubernetes.io/zh/docs/concepts/services-networking/ingress/
- https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#a-pure-software-solution-metallb
除了本文介绍的 Metallb 另,还有另一种解决方案为 openELB ,它是由国内互联网 青云 公司开源的一种方法,目前在 CNCF 沙盒 在托管,已在国内不少企业如 本来生活、苏州电视台、视源股份、云智天下、Jollychic、QingCloud、百旺、Rocketbyte 等海内外多家企业采用。