阿里云K8S私有Ingress Controller的配置和使用

  1. 云栖社区>
  2. 博客>
  3. 正文

阿里云K8S私有Ingress Controller的配置和使用

IvanYip 2019-06-28 10:37:23 浏览2721
展开阅读全文

阿里云K8S私有Ingress Controller的配置和使用

创建集群

进入阿里云容器服务控制台,创建一个新的k8s集群,此时集群会自动生成一个公网的Ingress Controller和一个公网的SLB监听着Worker的80和443端口。

默认的公网Ingress Controller

默认的公网SLB(SLB名字是自己起的,为了方便看)

创建私有SLB

进入负载均衡控制台创建一个私有SLB,实例规格按实际业务需求。注意:专有网络必须和刚才创建的集群的一样!!!

配置私有Ingress Controller

由于Ingress Controller Pods中的serviceAccountName是引用集群默认创建的,所以在此就不再配置ServiceAccount、ClusterRole和ClusterRoleBinding。

  1. Nginx ConfigMap

        
        apiVersion: v1
        kind: ConfigMap
        metadata:

    name: private-nginx-configuration #名字可以自己改
    namespace: kube-system
    labels:
    app: ingress-nginx

        data:

    proxy-body-size: 20m
    proxy-connect-timeout: "10"
    max-worker-connections: "65536"
    enable-underscores-in-headers: "true"
    reuse-port: "true"
    worker-cpu-affinity: "auto"
    server-tokens: "false"
    ssl-redirect: "false"
    allow-backend-server-header: "true"
    ignore-invalid-headers: "true"
    generate-request-id: "true"
    #forwarded-for-header: "X-Real-IP"
    #compute-full-forwarded-for: "true"
    #hsts: "false"
    #enable-vts-status: "true"
    #use-proxy-protocol: "true"

        ---
        # nginx tcp stream config map
        kind: ConfigMap
        apiVersion: v1
        metadata:

    name: private-tcp-services
    namespace: kube-system

        ---
        # nginx udp stream config map
        kind: ConfigMap
        apiVersion: v1
        metadata:

    name: private-udp-services
    namespace: kube-system

  2. Nginx Ingress Controller Pods

        apiVersion: apps/v1
        kind: Deployment
        metadata:

    name: private-nginx-ingress-controller
    labels:
    app: private-ingress-nginx
    namespace: kube-system
    annotations:
    component.version: '0.22.0'
    component.revision: '5'

        spec:

    replicas: 2
    selector:
    matchLabels:

    app: private-ingress-nginx

    template:
    metadata:

    labels:
      app: private-ingress-nginx
    annotations:
      prometheus.io/port: "10254"
      prometheus.io/scrape: "true"

    spec:

    #tolerations:
    #  - key: node-role.kubernetes.io/master
    #    effect: NoSchedule
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - ingress-nginx
            topologyKey: "kubernetes.io/hostname"
    #use default serviceAccountName
    serviceAccountName: nginx-ingress-controller
    initContainers:
    - name: init-sysctl
      image: registry-vpc.cn-hongkong.aliyuncs.com/acs/busybox:latest
      command:
      - /bin/sh
      - -c
      - |
        sysctl -w net.core.somaxconn=65535
        sysctl -w net.ipv4.ip_local_port_range="1024 65535"
        sysctl -w fs.file-max=1048576
        sysctl -w fs.inotify.max_user_instances=16384
        sysctl -w fs.inotify.max_user_watches=524288
        sysctl -w fs.inotify.max_queued_events=16384
      securityContext:
        privileged: true
    containers:
    - name: nginx-ingress-controller
      image: registry-vpc.cn-hongkong.aliyuncs.com/acs/aliyun-ingress-controller:v0.22.0.5-552e0db-aliyun
      args:
        - /nginx-ingress-controller
        - --configmap=$(POD_NAMESPACE)/private-nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/private-tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/private-udp-services
        - --annotations-prefix=nginx.ingress.kubernetes.io
        - --publish-service=$(POD_NAMESPACE)/private-nginx-ingress-lb
        - --ingress-class=private #自定义名
        - --v=2
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      ports:
      - name: http
        containerPort: 80
      - name: https
        containerPort: 443
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /healthz
          port: 10254
          scheme: HTTP
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /healthz
          port: 10254
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      securityContext:
        capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        runAsUser: 33
      volumeMounts:
      - name: localtime
        mountPath: /etc/localtime
        readOnly: true
    nodeSelector:
      beta.kubernetes.io/os: linux
    volumes:
      - name: localtime
        hostPath:
          path: /etc/localtime
          type: File
  3. Nginx Ingress Service

        apiVersion: v1
        kind: Service
        metadata:

    name: private-nginx-ingress-lb
    namespace: kube-system
    labels:
    app: private-nginx-ingress-lb
    annotations:
    # set loadbalancer to the specified slb id
    service.beta.kubernetes.io/alicloud-loadbalancer-id: lb-xxxx
    # set loadbalancer address type to intranet if using private slb instance
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
    service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'true'

        spec:

    type: LoadBalancer

    do not route traffic to other nodes

    and reserve client ip for upstream

    externalTrafficPolicy: "Local"
    ports:

    • port: 80
      name: http

    targetPort: 80

    • port: 443
      name: https

    targetPort: 443
    selector:
    # select app=private-ingress-nginx pods
    app: private-ingress-nginx

部署私有Ingress Controller

kubectl apply -f private-ingress-controller.yml
  1. Private Ingress Pod
  2. Private Ingress LB Service

更新Clusterrole:nginx-ingress-controller

由于在配置私有Ingress Controller Pod时是引用集群默认的ServiceAccount,新生成的ingress-controller-leader-private配置项没有更新到默认的ClusterRole所以导致Service启动时会报没权限,此时我们需要在默认的ClusterRole中的resourceNames下添加ingress-controller-leader-private


kubectl edit clusterrole nginx-ingress-controller -o yaml

使用阿里云DNS PrivateZone绑定SLB IP

  1. 进入云解析DNS控制台,开通PrivateZone并添加域名。
  2. 关联vpc
  3. 添加域名解析,绑定私有SLB IP

部署测试服务

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
  generation: 1
  labels:
    app: demo
  name: demo
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: demo
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - image: >-
            registry-vpc.cn-hongkong.aliyuncs.com/xxxx/demo:1.0.6-1
          imagePullPolicy: Always
          name: demo
          resources:
            limits:
              cpu: 2048m
              memory: 4Gi
            requests:
              cpu: 2048m
              memory: 4Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          livenessProbe:
            httpGet:
              path: /v1/health
              port: 80
            initialDelaySeconds: 180
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /v1/health
              port: 80
            initialDelaySeconds: 180
            periodSeconds: 10
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 300
      
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: private
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  generation: 1
  name: private-demo-ingress
  namespace: default
spec:
  tls:
  - hosts:
    - k8s-test.internal.abc.com
    # 配置TLS证书
    secretName: abc.com
  rules:
  - host: k8s-test.internal.abc.com
    http:
      paths:
      - backend:
          serviceName: demo-svc
          servicePort: 80
        path: /abc(/|$)(.*)
        
---
apiVersion: v1
kind: Service
metadata:
  name: demo-svc
  namespace: default
spec:
  clusterIP: None
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: demo
  sessionAffinity: None
  type: ClusterIP
  

测试结果

在同一个VPC下的ECS访问集群服务结果如下

网友评论

登录后评论
0/500
评论
IvanYip
+ 关注