istio 多集羣鏈路追蹤實操指南

【導讀】本文非常詳細地介紹了 istio 做 tracing 的多控制面、多網格實操步驟。

理論篇

什麼是可觀測性

這裏的可觀察性主要指服務網格的可觀察性，也就是需要觀測服務網格中運行的微服務。爲什麼可觀察性很重要，因爲隨着微服務架構的流行，一個系統可能運行成百上千微服務，如果系統出現故障，定位問題帶來很大的問題。有了觀測系統，就能更好的分析問題發生的原因，已經更好的監控告警。

服務網格可觀察性主要分爲三個大類，分別是 log，metrics，tracing。

log 是指將分佈式系統的日誌收集起來集中存儲，用於日誌分析，常用的工具如 efk。

metrics 是指收集服務網格的監控指標，進行監控告警，常用工具比如 prometheus。

tracing 是指分佈式鏈路追蹤，用於可視化顯示服務調用的依賴關係，及獲取延遲數據，常用工具如 zipkin，jaeger 等。

本文關注的是 tracing，這裏我們用到了 zipkin 作爲 tracing 工具。

什麼是鏈路追蹤

在分佈式系統，尤其是微服務系統中，一次外部請求往往需要內部多個模塊，多箇中間件，多臺機器的相互調用才能完成。在這一系列的調用中，可能有些是串行的，而有些是並行的。在這種情況下，我們如何才能確定這整個請求調用了哪些應用？哪些模塊？哪些節點？以及它們的先後順序和各部分的性能如何呢？

鏈路追蹤是分佈式系統下的一個概念，它的目的就是要解決上面所提出的問題，也就是將一次分佈式請求還原成調用鏈路，將一次分佈式請求的調用情況集中展示，比如，各個服務節點上的耗時、請求具體到達哪臺機器上、每個服務節點的請求狀態等等。

多集羣鏈路追蹤

我們這裏要演示的鏈路追蹤不是單個 istio 集羣的，而是多個 istio 集羣的。

我們把多個 istio 集羣部署成一個聯邦的 istio 集羣，把多個集羣的 tracing 數據在 zipkin 集中存儲分析。單個 istio 集羣的鏈路追蹤相對比較簡單，只需配置 istio 的 comfigmap 就行，多個集羣考慮到集羣的部署方式有很多，需要所有 proxy 將信息傳送到統一的一個 zipkin，相對來說複雜一點。

這裏我們展示兩集羣 istio 聯邦和三集羣 istio 聯邦，一共 14 個案例。

實操篇

環境說明

兩集羣部署用的機子是：

cluster1

192.168.229.128  master

192.168.229.129  master

192.168.229.130  node

cluster2

192.168.229.131 master

192.168.229.132  master

192.168.229.133 node

三集羣部署用的機子是；

cluster1

192.168.229.137  master

192.168.229.138  master

192.168.229.139  node

cluster2

192.168.229.140  master

192.168.229.141  master

192.168.229.142  node

cluster3

192.168.229.143  master

192.168.229.144  master

192.168.229.145  node

k8s 版本

[root@node01 ~]# kubectl version --short
Client Version: v1.21.0
Server Version: v1.21.0

istio 版本

[root@node01 ~]# istioctl version
client version: 1.11.2
control plane version: 1.11.2
data plane version: none

兩集羣準備

首先需要創建 root-ca，多個 istio 集羣的 root-ca 必須是一樣的：

cluster1:
 mkdir -p certs
 make -f ../tools/certs/Makefile.selfsigned.mk root-ca
 make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
 make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts
 scp -r cluster2 root@192.168.229.131:/root/cluster2

  kubectl create namespace istio-system
 kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster1/ca-cert.pem \
      --from-file=cluster1/ca-key.pem \
      --from-file=cluster1/root-cert.pem \
      --from-file=cluster1/cert-chain.pem

       cluster2:
  kubectl create namespace istio-system
 kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster2/ca-cert.pem \
      --from-file=cluster2/ca-key.pem \
      --from-file=cluster2/root-cert.pem \
      --from-file=cluster2/cert-chain.pem

兩集羣

單個控制面板

在同一個網絡中

Image

部署步驟：

集羣 1
128,129,130
集羣 2
131,132,133
128。129.130
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.131
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.133
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.132
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.131

131,132，133
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.128
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.129
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.130
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.128
生成部署 operator 文件
 cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

#  export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network1
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF
scp cluster2.yaml root@192.168.229.131:/root
安裝 cluster1
istioctl install -f cluster1.yaml
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 |  istioctl  install -y  -f -
 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'
  kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

cluster2 生成訪問 apiserver secret

 istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.128:/root

cluster1 應用 secret

  kubectl apply -f remote-secret-cluster2.yaml

cluster2 安裝 cluster2

 istioctl install  -f cluster2.yaml

cluster1  重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster2   重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster1 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口 kubectl edit svc -n istio-system istio-eastwestgateway

  - name: http-zipkin
    nodePort: 32197
    port: 15018
    protocol: TCP
    targetPort: 15018

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1,cluster2: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

清理：

cluster1:

kubectl delete vs istiod-vs -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

istioctl x uninstall -f cluster2.yaml

reboot
在不同網絡中

Image

集羣1
128,129,130
集羣2
131,132,133

給 istio-system namespace 打標籤 cluster1:

kubectl  label namespace istio-system topology.istio.io/network=network1

cluster2:

kubectl  label namespace istio-system topology.istio.io/network=network2
cluster1:

生成 istio operator 部署文件

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

#  export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到另一個集羣

scp cluster2.yaml root@192.168.229.131:/root

安裝 istio

istioctl install  -f cluster1.yaml

安裝東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh  --mesh mesh1 --cluster cluster1 --network network1 |  istioctl install -y  -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster2:

生成 istiod 訪問 apiserver secret

istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.128:/root

cluster1 安裝 secret

kubectl apply -f remote-secret-cluster2.yaml -n istio-system

部署 cluster2

istioctl install  -f cluster2.yaml

生成東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster1 重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster1 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口

 kubectl edit svc -n istio-system istio-eastwestgateway   - name: http-zipkin    nodePort: 32197    port: 15018    protocol: TCP    targetPort: 15018

cluster1 暴露 zipkin visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1,cluster2 : cm istio

cluster1,cluster2,cluster3: cm istio

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

    修改
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

清理：

cluster1:kubectl  label namespace istio-system topology.istio.io/network-kubectl delete vs istiod-vs -n istio-systemkubectl delete gw istiod-gateway -n istio-systemkubectl delete gw cross-network-gateway -n istio-systemkubectl delete secret istio-remote-secret-cluster2 -n istio-systemkubectl delete gw zipkin-gateway -n istio-systemkubectl delete vs zipkin-vs -n istio-systemistioctl x uninstall -f cluster1.yamlrebootcluster2:kubectl  label namespace istio-system topology.istio.io/network-kubectl delete gw cross-network-gateway -n istio-systemistioctl x uninstall -f cluster2.yamlreboot

兩個控制面板

在同一個網絡中

兩集羣網絡聯通

集羣1
128,129,130
集羣2
131,132,133
#兩個網絡聯通
128。129.130
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.131
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.133
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.132
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.131

131,132，133
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.128
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.129
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.130
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.128

生成部署 operator 文件

 cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

#  export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network1
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到另一個集羣

scp cluster2.yaml root@192.168.229.131:/root

安裝 cluster1

istioctl install -f cluster1.yaml

生成東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 |  istioctl  install -y  -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

  kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

cluster2 生成訪問 apiserver secret

istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

scp remote-secret-cluster2.yaml root@192.168.229.128:/root

cluster1 應用 secret

kubectl apply -f remote-secret-cluster2.yaml

cluster2 安裝 cluster2

istioctl install  -f cluster2.yaml

cluster1  重啓 pod

kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster2 重啓 pod

kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

由於 cluster2 dns 無法解析 zipkin.istio-system，所以 cluster1 需要安裝東西向網關

# 部署東西向網關
/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl  install -y  -f -

#配置東西向網關ip 
 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

cluster1 增加東西向網關端口

kubectl edit svc -n istio-system istio-eastwestgateway

  - name: http-zipkin
    nodePort: 32197
    port: 15018
    protocol: TCP
    targetPort: 15018

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1,cluster2: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'
 cluster1: 
  重啓pod
 kubectl rollout restart deploy -n istio

  cluster2:
   重啓pod
 kubectl rollout restart deploy -n istio

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

清理：

cluster1:

kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

kubectl delete secret istio-remote-secret-cluster1 -n istio-system
istioctl x uninstall -f cluster2.yaml

reboot
在不同網絡中

 

集羣1
128,129,130
集羣2
131,132,133

給 istio-system namespace 打標籤 cluster1:

 kubectl  label namespace istio-system topology.istio.io/network=network1

cluster2:

 kubectl  label namespace istio-system topology.istio.io/network=network2

cluster1 生成 istio operator 部署文件

 cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

cluster2 生成 istio operator 部署文件

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到 cluster2

scp cluster2.yaml root@192.168.229.131:/root

生成監控 apiserver secret

傳輸 secret 到 cluster2

scp remote-secret-cluster1.yaml root@192.168.229.131:/root

cluster2 生成監控 apiserver secret

 istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.131:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.128:/root

cluster1 部署監控 apiserver secret

kubectl apply -f remote-secret-cluster2.yaml

部署 istio

istioctl install  -f cluster1.yaml

部署東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 | istioctl  install -y  -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露服務

 kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster2 部署監控 apiserver secret

kubectl apply -f remote-secret-cluster1.yaml

部署 istio

 istioctl install -f cluster2.yaml

部署東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 |  istioctl install -y -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster1 重啓 pod

kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster2 重啓 pod

kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster1: 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 : cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: zipkin.istio-system:9411
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

cluster1: 暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

增加東西向網關端口 kubectl edit svc -n istio-system istio-eastwestgateway

  - name: http-zipkin
    nodePort: 32197
    port: 15018
    protocol: TCP
    targetPort: 15018

暴露 zipkin 到 cluster2

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster2 : cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

清理：

cluster1:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete secret istio-remote-secret-cluster1 -n istio-system
istioctl x uninstall -f cluster2.yaml

reboot

三集羣

單控制面板

單網絡

Image

三個網絡聯通
集羣1
137,138,139
集羣2
140,141,142
集羣3
143,144,145

網絡聯通

137,138,139
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140

route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143

route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140

140,141,142
route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.139
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.138
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.137

route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143

route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.137

143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140

route add -net 172.20.2.0 netmask 255.255.255.0 gw 192.168.229.139
route add -net 172.20.0.0 netmask 255.255.255.0 gw 192.168.229.138
route add -net 172.20.1.0 netmask 255.255.255.0 gw 192.168.229.137

route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140
route add -net 10.68.0.0 netmask 255.255.0.0 gw 192.168.229.137

cluster1: 生成 istio operator 部署文件

 cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

#  export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 istio operator 部署文件

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network1
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到 cluster2

scp cluster2.yaml root@192.168.229.140:/root

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

#  export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 istio operator 部署文件

cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster3
      network: network1
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到 cluster3

scp cluster3.yaml root@192.168.229.143:/root

部署 istio

istioctl install -f cluster1.yaml

生成東西向網關

 /root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster1 --network network1 |  istioctl  install  -y -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

kubectl apply  -n istio-system -f  /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

cluster2: 生成訪問 apiserver secret

 istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.137:/root

cluster3: 生成訪問 apiserver secret

istioctl x create-remote-secret --name=cluster3  --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster3.yaml root@192.168.229.137:/root

cluster1 應用 secret

kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml

cluster2: 部署 istio

 istioctl install  -f cluster2.yaml

cluster3: 部署 istio

 istioctl install  -f cluster3.yaml

cluster1: 重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster2: 重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster3: 重啓 pod

 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster1: 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口

  kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1,cluster2,cluster3: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

    修改
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
 cluster1: 
  重啓pod
 kubectl rollout restart deploy -n istio

  cluster2:
   重啓pod
 kubectl rollout restart deploy -n istio

    cluster3:
   重啓pod
 kubectl rollout restart deploy -n istio

我的集羣的應用部署情況：

cluster1:
[root@node01 istio-teaching]# kubectl get pod -n istio
NAME                             READY   STATUS    RESTARTS   AGE
productpage-v1-655c9d8c9-dln7x   2/2     Running   0          2m50s
ratings-v1-86ccf5754f-bz867      2/2     Running   0          2m50s

cluster2:
[root@node01 ~]# kubectl get pod -n istio
NAME                          READY   STATUS    RESTARTS   AGE
reviews-v2-77f86758bd-9fb4n   2/2     Running   0          11m

cluster3:
[root@node01 ~]# kubectl get pod -n istio
NAME                          READY   STATUS    RESTARTS   AGE
details-v1-548fbfb4d5-2xhkk   2/2     Running   0          11m
ratings-v1-678964777c-wkg4c   2/2     Running   0          11m
reviews-v3-76857cf4bf-5vhck   2/2     Running   0          11m

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

清理：

cluster1:

kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete vs istiod-vs -n istio-system
kubectl delete gw istiod-gateway -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

istioctl x uninstall -f cluster2.yaml

reboot

cluster3:

istioctl x uninstall -f cluster3.yaml

reboot
兩網絡
兩網關

Image

兩個網絡 network2 東西向網管可以在 cluster2 也可以在 cluster3 cluster2 有網關，cluster3 沒有網關 不建議使用，按地域負載均衡的時候會有問題

集羣1
137,138,139
集羣2
140,141,142
集羣3
143,144,145

打通 cluster2，cluster3 網絡

140,141,142
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143

route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143

143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140

route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140

給 isito-system namespace 打標籤

cluster1:
kubectl  label namespace istio-system topology.istio.io/network=network1

cluster1:
kubectl  label namespace istio-system topology.istio.io/network=network2

cluster1:
kubectl  label namespace istio-system topology.istio.io/network=network2

生成 operator 部署文件

cluster1:
cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 operator 部署文件

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 operator 部署文件

cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster3
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

把部署文件傳到 cluster2

scp cluster2.yaml root@192.168.229.140:/root

把部署文件傳到 cluster3

scp cluster3.yaml root@192.168.229.143:/root

部署 cluster1

istioctl install  -f cluster1.yaml

部署東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh  --mesh mesh1 --cluster cluster1 --network network1 |  istioctl install -y  -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster2: 生成監控 apiserver secret

istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.137:/root

cluster3: 生成監控 apiserver secret

istioctl x create-remote-secret --name=cluster3  --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster3.yaml root@192.168.229.137:/root

cluster1: 應用監控 apiserver secret

kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml

cluster2: 部署 cluster2

istioctl install  -f cluster2.yaml

安裝東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y  -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster3: 部署 cluster3

istioctl install  -f cluster3.yaml

cluster1: 重啓 pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system

cluster2: 重啓 pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system

cluster3: 重啓 pod kubectl rollout restart deploy -n istio kubectl rollout restart deploy -n istio-system

cluster1: 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口

  kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1，cluster2,cluster3: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

    修改
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
cluster1: 
重啓pod
kubectl rollout restart deploy -n istio

cluster2:
重啓pod
kubectl rollout restart deploy -n istio

cluster3:
重啓pod
 kubectl rollout restart deploy -n istio

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

Image

清理：

cluster1:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml

reboot

cluster3:

kubectl  label namespace istio-system topology.istio.io/network-
istioctl x uninstall -f cluster3.yaml

reboot
三網關
兩個網絡
三個東西向網關

集羣1
137,138,139
集羣2
140,141,142
集羣3
143,144,145

打通 cluster2，cluster3 網絡

140,141,142
route add -net 172.22.2.0 netmask 255.255.255.0 gw 192.168.229.145
route add -net 172.22.0.0 netmask 255.255.255.0 gw 192.168.229.144
route add -net 172.22.1.0 netmask 255.255.255.0 gw 192.168.229.143

route add -net 10.70.0.0 netmask 255.255.0.0 gw 192.168.229.143

143,144,145
route add -net 172.21.2.0 netmask 255.255.255.0 gw 192.168.229.142
route add -net 172.21.0.0 netmask 255.255.255.0 gw 192.168.229.141
route add -net 172.21.1.0 netmask 255.255.255.0 gw 192.168.229.140

route add -net 10.69.0.0 netmask 255.255.0.0 gw 192.168.229.140

給 isito-system namespace 打標籤

cluster1:
kubectl  label namespace istio-system topology.istio.io/network=network1

cluster2:
kubectl  label namespace istio-system topology.istio.io/network=network2

cluster3:
kubectl  label namespace istio-system topology.istio.io/network=network2

生成 operator 部署文件 cluster1:

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 operator 部署文件

cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 operator 部署文件

cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      imagePullPolicy: IfNotPresent
      meshID: mesh1
      multiCluster:
        clusterName: cluster3
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

把部署文件傳到 cluster2

scp cluster2.yaml root@192.168.229.140:/root

把部署文件傳到 cluster3

scp cluster3.yaml root@192.168.229.143:/root

部署 cluster1

istioctl install  -f cluster1.yaml

部署東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh  --mesh mesh1 --cluster cluster1 --network network1 |  istioctl install -y  -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster2: 生成監控 apiserver secret

istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.137:/root

cluster3: 生成監控 apiserver secret

istioctl x create-remote-secret --name=cluster3  --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml

傳輸 secret 到 cluster1

 scp remote-secret-clu

cluster1: 應用監控 apiserver secret

kubectl apply -f remote-secret-cluster2.yaml
kubectl apply -f remote-secret-cluster3.yaml

cluster2: 部署 cluster2

istioctl install  -f cluster2.yaml

安裝東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y  -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster3: 部署 cluster3

istioctl install  -f cluster3.yaml

安裝東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster3 --network network2 | istioctl install -y  -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.102"]}}'

暴露服務

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
cluster1:
重啓pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster2:
重啓pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster1:
重啓pod
kubectl rollout restart deploy -n istio
kubectl rollout restart deploy -n istio-system

cluster1: 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1，cluster2,cluster3: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

    修改
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
 cluster1: 
  重啓pod
 kubectl rollout restart deploy -n istio

  cluster2:
   重啓pod
 kubectl rollout restart deploy -n istio

    cluster3:
   重啓pod
 kubectl rollout restart deploy -n istio

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

Image

清理：

cluster1:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml

reboot

cluster3:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster3.yaml

reboot
三網絡
三個網絡
集羣1
137,138,139
集羣2
140,141,142
集羣3
143,144,145

給 istio-system namespace 打標籤

cluster1:
kubectl  label namespace istio-system topology.istio.io/network=network1

cluster2:
 kubectl  label namespace istio-system topology.istio.io/network=network2

cluster3:
 kubectl  label namespace istio-system topology.istio.io/network=network3

cluster1: 生成 istio operator 部署文件

cat <<EOF > cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 istio operator 部署文件

 cat <<EOF > cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

這裏我設置的 cluster1 東西向網關的 ip 試 192.168.229.100 如果用的是 loadblance，可以用下面命令獲取

export DISCOVERY_ADDRESS=$(kubectl  -n istio-system get svc istio-eastwestgateway  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

然後替換 remotePilotAddress

生成 istio operator 部署文件

cat <<EOF > cluster3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: demo
  values:
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster3
      network: network3
      remotePilotAddress: 192.168.229.100
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
EOF

傳輸部署文件到 cluster2

scp cluster2.yaml root@192.168.229.140:/root

傳輸部署文件到 cluster3

scp cluster3.yaml root@192.168.229.143:/root

安裝 istio

istioctl install  -f cluster1.yaml

生成東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh  --mesh mesh1 --cluster cluster1 --network network1 |  istioctl install -y -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.100"]}}'

暴露 istiod

kubectl apply  -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-istiod.yaml

暴露 service

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster2: 生成訪問 apiserver 的 secret

istioctl x create-remote-secret --name=cluster2  --server=https://192.168.229.140:6443 > remote-secret-cluster2.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster2.yaml root@192.168.229.137:/root

cluster3: 生成訪問 apiserver 的 secret

istioctl x create-remote-secret --name=cluster3  --server=https://192.168.229.143:6443 > remote-secret-cluster3.yaml

傳輸 secret 到 cluster1

 scp remote-secret-cluster3.yaml root@192.168.229.137:/root

cluster1: 應用 secret

  kubectl apply -f remote-secret-cluster2.yaml
  kubectl apply -f remote-secret-cluster3.yaml

cluster2: 部署 istio

istioctl install  -f cluster2.yaml

生成東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster2 --network network2 | istioctl install -y -f -

配置東西向網關 ip

kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.101"]}}'

暴露 service

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml

cluster3: 部署 istio

istioctl install  -f cluster3.yaml

生成東西向網關

/root/istio-1.11.2/samples/multicluster/gen-eastwest-gateway.sh --mesh mesh1 --cluster cluster3 --network network3 | istioctl install -y -f -

配置東西向網關 ip

 kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"externalIPs":["192.168.229.102"]}}'

暴露 service

kubectl  apply -n istio-system -f /root/istio-1.11.2/samples/multicluster/expose-services.yaml
   cluster1:
重啓pod
 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

    cluster2:
重啓pod
 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

  cluster3:
 重啓pod
 kubectl rollout restart deploy -n istio
 kubectl rollout restart deploy -n istio-system

cluster1: 部署 zipkin

 kubectl apply -f extras/zipkin.yaml -n istio-system

cluster1 增加東西向網關端口

  kubectl patch svc  -n istio-system istio-eastwestgateway -p '{"spec":{"ports":[{"name": "http-zipkin", "nodePort": 32197,"port": 15018, "protocol": "TCP", "targetPort": 15018}]}}'

cluster1:

暴露 zipkin

visilazation/zipkin-gw-vs.yaml

kubectl apply -f zipkin-gw-vs.yaml -n istio-system

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: zipkin-gateway
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        name: http-zipkin
        number: 15018
        protocol: http        
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: zipkin-vs
spec:
  hosts:
  - "*"
  gateways:
  - zipkin-gateway
  http:
  - route:
    - destination:
        host: zipkin.istio-system.svc.cluster.local
        port:
          number: 9411

cluster1，cluster2,cluster3: cm istio

[root@node01 ~]# kubectl get cm istio -n istio-system -o yaml

apiVersion: v1
data:
  mesh: |-
    accessLogFile: /dev/stdout
    enableTracing: true
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      meshId: mesh1
      proxyMetadata: {}
      tracing:
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
    enablePrometheusMerge: true
    enableTracing: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

    修改
        sampling: 100
        zipkin:
          address: 192.168.229.100:15018
 cluster1: 
  重啓pod
 kubectl rollout restart deploy -n istio

  cluster2:
   重啓pod
 kubectl rollout restart deploy -n istio

  cluster3:
   重啓pod
 kubectl rollout restart deploy -n istio

暴露服務：

kubectl port-forward --address 0.0.0.0 -n istio-system zipkin-6b8c6bdc56-m2b4f 9411:9411

清理：

cluster1:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete secret istio-remote-secret-cluster2 -n istio-system
kubectl delete secret istio-remote-secret-cluster3 -n istio-system
kubectl delete gw cross-network-gateway -n istio-system
kubectl delete gw istiod-gateway -n istio-system
kubectl delete vs istiod-vs -n istio-system
kubectl delete gw zipkin-gateway -n istio-system
kubectl delete vs zipkin-vs -n istio-system
istioctl x uninstall -f cluster1.yaml

reboot

cluster2:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster2.yaml

reboot

cluster3:

kubectl  label namespace istio-system topology.istio.io/network-
kubectl delete gw cross-network-gateway -n istio-system
istioctl x uninstall -f cluster3.yaml

reboot

Go 開發大全

參與維護一個非常全面的 Go 開源技術資源庫。日常分享 Go, 雲原生、k8s、Docker 和微服務方面的技術文章和行業動態。

本文由 Readfog 進行 AMP 轉碼，版權歸原作者所有。
來源：https://mp.weixin.qq.com/s/op7bQ_QjEFbKjhK5NtzVOA

猜你喜歡