In Managing Canary Release of Microservice App based on Istio, KubeSphere implemented grayscale publishing for the Bookinfo microservices sample application based on Istio. Some users have indicated that their project has not yet been on Istio. How do you implement grayscale publishing?
In Ingress-Nginx (v0.21.0), a new Canary feature has been introduced that can be used to configure multiple backend services for gateway portals, as well as to control traffic distribution between multiple backend services using specified annotations. . In v2.0.2, KubeSphere upgraded the Ingress Controller version to 0.24.1 to support Ingress-Nginx-based grayscale publishing.
In the previous article, several application scenarios of grayscale publishing have been introduced in detail. This article will directly introduce and demonstrate the implementation of grayscale publishing based on KubeSphere using Ingress and Ingress Controller.
Note: The sample yaml source files and code used in this article have been uploaded to GitHub and can be cloned for local convenience.
Based on the Nginx Ingress Controller, KubeSphere implements the project's gateway as a proxy for external traffic and a reverse proxy for each service in the project. Ingress-Nginx supports the configuration of Ingress Annotations to implement grayscale publishing and testing in different scenarios. It can meet the business scenarios of Canary Publishing, Blue-Green Deployment, and A/B Testing.
Nginx Annotations supports the following four Canary rules:
nginx.ingress.kubernetes.io/canary-by-header：Flow segmentation based on Request Header for grayscale publishing and A/B testing. When the Request Header is set to
always, the request will be sent to the Canary version all the time. When set to
never, the request will not be sent to the Canary entry. For any other Header value, the Header will be ignored and the request is prioritized by comparison with other Canary rules.
nginx.ingress.kubernetes.io/canary-by-header-value：The value of the Request Header to match to tell Ingress to route the request to the service specified in Canary Ingress. When the Request Header is set to this value, it will be routed to the Canary entry. This rule allows the user to customize the value of the Request Header and must be used with the previous annotation (ie: canary-by-header).
nginx.ingress.kubernetes.io/canary-weight：Service-weighted traffic segmentation for blue-green deployments, weight range 0 - 100 Route requests to services specified in Canary Ingress. A weight of 0 means that the Canary Rules will not send any requests to the services of the Canary portal. A weight of 100 means that all requests will be sent to the Canary portal.
nginx.ingress.kubernetes.io/canary-by-cookie：Cookie-based traffic segmentation for grayscale publishing and A/B testing. A cookie that tells Ingress to route requests to the service specified in Canary Ingress. When the value of cookie is set to
always, it will be routed to the Canary portal; when set to
never, the request will not be sent to the Canary entry; for any other value, the cookie is ignored and the request is prioritized against other Canary rules.
Note: The Canary Rules are sorted in order of priority:
canary-by-header - > canary-by-cookie - > canary-weight
The above four annotation rules can be divided into the following two categories:
1.1. Create a business space (spacespace) and a project (namespace) in KubeSphere. You can refer to Getting Started with Multi-tenant Management. A sample project has been created as follows.
1.2. To create an application easily, create a workload and service in your project by
edit yaml , or use the toolbox in the bottom right corner of KubeSphere to open
web kubectl and create a Production version of the application with the following command and yaml file and expose it to the cluster access. Create a
service for the Production version as follows.
$ kubectl appy -f production.yaml -n ingress-demo deployment.extensions/production created service/production created
The yaml file used is as follows:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: production spec: replicas: 1 selector: matchLabels: app: production template: metadata: labels: app: production spec: containers: - name: production image: mirrorgooglecontainers/echoserver:1.10 ports: - containerPort: 8080 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP --- apiVersion: v1 kind: Service metadata: name: production labels: app: production spec: ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: app: production
1.3. Create a Production version of Ingress.
$ kubectl appy -f production.ingress -n ingress-demo ingress.extensions/production created
The yaml file used is as follows:
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: production annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: kubesphere.io http: paths: - backend: serviceName: production servicePort: 80
2.1. At this point, you can see all the resources under the ingress-demo project under the enterprise space demo-workspace of the KubeSphere UI.
2.2. Access the Production version of the application to ensure that the current project has opened the gateway, open the gateway under the external network access, the type is
2.3. Access the Production version of the app below.
$ curl --resolve kubesphere.io:30205:192.168.0.88 kubesphere.io:30205 # Note that adding the --resolve parameter eliminates the need to configure IP and domain name mappings in /etc/hosts locally, otherwise you need to configure the domain name mapping locally, where 192.168.0.88 is the gateway address within the project. Hostname: production-6b4bb8d58d-7r889 Pod Information: node name: ks-allinone pod name: production-6b4bb8d58d-7r889 pod namespace: ingress-demo pod IP: 10.233.87.165 Server values: server_version=nginx: 1.12.2 - lua: 10010 Request Information: client_address=10.233.87.225 method=GET real path=/ query= request_version=1.1 request_scheme=http request_uri=http://kubesphere.io:8080/ Request Headers: accept=*/* host=kubesphere.io:30205 user-agent=curl/7.29.0 apiVersion: extensions/v1beta1 x-forwarded-for=192.168.0.88 x-forwarded-host=kubesphere.io:30205 x-forwarded-port=80 x-forwarded-proto=http x-original-uri=/ x-real-ip=192.168.0.88 x-request-id=9596df96e994ea05bece2ebbe689a2cc x-scheme=http Request Body: -no body in request-
Refer to the
production.yaml file of the above Production version, and then create a Canary version of the application, including a Canary version of
service (for demonstration, just replace the keyword
production in the
production.yaml deployment and service production to
canary. The actual scenario may involve business code changes).
A typical application scenario for weight-based traffic segmentation is a
Blue-green deployment, which can be achieved by setting the weight to 0 or 100. For example, you can set the Green version to the main part and the Blue version's entry to Canary. Initially, the weight is set to 0, so traffic is not proxied to the Blue version. Once the new version has been tested and verified successfully, you can set the weight of the Blue version to 100, which means that all traffic goes from Green to Blue.
4.1. Use the following
canary.ingress yaml file to create an application routing (Ingress) based on the weighted Canary version.
Note: To enable the grayscale publishing mechanism, first set
nginx.ingress.kubernetes.io/canary: "true"to enable Canary. The following Canary version of the Ingress example uses an annotation rule based on weights for traffic segmentation, which will be assigned 30% traffic request to sent to the Canary version.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: canary annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "30" spec: rules: - host: kubesphere.io http: paths: - backend: serviceName: canary servicePort: 80
4.2. Access the domain name of the app.
Note: After the application's Canary version is based on the weight (30%) for traffic segmentation, the probability of accessing the Canary version is close to 30%, and the traffic ratio may have a small range of fluctuations.
4.3. The classic application scenes based on Request Header's traffic segmentation are
greyscale release or A/B test scenes. Refer to the screenshot below. Add an annotation of
nginx.ingress.kubernetes.io/canary-by-header: canary for Ingress in KubeSphere's Canary edition. (The annotation value here can be random.) Realize the current Ingress and segregate the traffic based on the Ingress.
Note: The Canary Rules are prioritized by
canary-by-header - > canary-by-cookie - > canary-weight, so in the following case, rules for the original canary-weight will be ignored.
4.4. Add a different Header value to the request and access the domain name of the app again.
For examples, as mentioned in the opening paragraph, when the Request Header is set to
always, the request will
alwaysbe sent or always sent to the Canary version; For any other Header value, the Header is ignored and the request is prioritized by comparison with other Canary rules (the second request has a 30% weight as the first priority).
4.5. You can add an
nginx.ingress.kubernetes.io/canary-by-header-value: user-value to the previous annotation (that is, canary-by-header). Used to notify Ingress to route requests to the service specified in Canary Ingress.
4.6. The application's domain name is accessed as follows. When the Request Header satisfies this value, all requests are routed to the Canary version (this rule allows the user to customize the value of the Request Header).
Similar to the Request Header-based annotation usage rules. For example, in the
A/B 测试 scenario, you need to let the users in Beijing have access to the Canary version. Then when the cookie's annotation is set to
nginx.ingress.kubernetes.io/canary-by-cookie: "users_from_Beijing". Now the background user can check the logged in user request, and set the value of the cookie
always if the user access source is from Beijing. For always, this will ensure that users in Beijing only access the Canary version.
The grayscale release can ensure the stability of the overall system. When the initial grayscale is used, the new version can be tested, found and adjusted to ensure its influence. This article demonstrates and demonstrates the use of Ingress and Ingress Controller for grayscale publishing based on Kubeel, and details the four Annotations of Ingress-Nginx. Users who have not used Istio can also easily implement grayscale publishing with Ingress-Nginx. Released with the canary.