NEW! Mirantis Academy -   Learn confidently with expert guidance and On-demand content.   Learn More


Spinnaker Canary Pipelines: How to set up Kayenta with Prometheus

Andrey Pavlov - September 18, 2018
One of the advantages of Spinnaker is the ability to easily create pipelines using different deployment strategies, including canary deployments. In this article, we'll explain how to set up Spinnaker and the default canary provider, Kayenta, to work with metrics from Prometheus.
Our goal is to make it possible for you to create a canary deployment using Prometheus for an existing Kubernetes application.
We're making a couple of assumptions:
  1. You already understand the basics of how to use Spinnaker, how to create pipelines, and so on.
  2. You've got a general idea of how canary deployments work in terms of the way configurations work, and so on. You can get more information and background here.
  3. You've already created (or have access to) a Kubernetes cluster big enough to handle both Spinnaker and Prometheus. In our example, we used a Google Kubernetes Engine cluster with 4 vCPU and 26 GB of RAM.
  4. You've already installed Helm and initialized it with a service account.  Check out this article for information on installing Helm.  The easiest way to do the initialization is:
    kubectl -n kube-system create sa tiller
    kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
    helm init --service-account tiller --upgrade
  5. You have an existing application you want to track. If you need something to work with, feel free to use this YAML file for a sample app:
    apiVersion: extensions/v1beta1
    kind: Deployment
        app: demo-app
        release: canary
      name: demo-app-canary
      replicas: 1
            app: demo-app
            release: canary
          - image: kshatrix/canary-demo
            name: demo-app
                path: /
                port: 80
              initialDelaySeconds: 5
              periodSeconds: 2
    apiVersion: v1
    kind: Service
      annotations: / "8000" "true"
        app: demo-app
        release: canary
      name: demo-app-canary
      - name: http
        port: 80
      - name: http-metrics
        port: 8000
        app: demo-app
  6. You have Prometheus installed. If you don't, you can install it on your Kubernetes cluster using:
    helm install -n prom stable/prometheus --namespace hal
Now that that's out of the way, let's get into the actual project.

Install and configure Spinnaker

The first step is to go ahead and install Spinnaker. Spinnaker is installed and configured using Halyard, so we're going to go ahead and install that using Helm.  Earlier, when we installed Spinnaker using Helm, we went ahead and did the Spinnaker deployment right away, but this time we want to tell Halyard to wait for instructions so we can do some additional configuration before the actual deployment:
helm repo add mirantisworkloads
helm install -n hal mirantisworkloads/halyard  --namespace hal --set command="sleep infinity" --set daemon=true
You should see a series of pods in the hal namespace:
kubectl get pods -n hal
NAME                                                READY STATUS RESTARTS AGE
hal-hal-apply-6967cbdb4d-s457x                      1/1 Running 0 4m
jenkins-hal-5549f48b6b-scq9k                        1/1 Running 0 4m
minio-hal-0                                         1/1 Running 0 5m
minio-hal-1                                         1/1 Running 0 5m
minio-hal-2                                         1/1 Running 0 5m
minio-hal-3                                         1/1 Running 0 5m
prom-prometheus-alertmanager-794b44784f-bchbh       2/2 Running 0 7m
prom-prometheus-kube-state-metrics-7b5b5f55-kfb28   1/1 Running 0 7m
prom-prometheus-node-exporter-dsq9d                 1/1 Running 0 7m
prom-prometheus-pushgateway-78d69775d8-hgk4l        1/1 Running 0 7m
prom-prometheus-server-689675c8f4-wbtzp             2/2 Running 0 7m
Now let’s get into the halyard pod to configure kayenta. Start by getting a command line on the Halyard pod. (Obviously, substitute your own pod here):
kubectl -n hal exec -it hal-hal-apply-6967cbdb4d-s457x bash
Now you can execute these commands inside the hal pod:
  1. Enable canary deployments in general so they're available in the UI:
    hal config canary enable
  2. Configure canary to use Prometheus as a source of metrics:
    hal config canary prometheus enable
    hal config canary prometheus account add my-prometheus --base-url http://prom-prometheus-server:80
  3. Configure canary to use Minio as a storage.  We’ll use the Minio instance deployed during the halyard chart installation - the same instance that stores the Spinnaker configuration itself:
    hal config canary aws enable
    echo "miniosecret" | hal config canary aws account add my-minio --bucket spin-bucket --endpoint http://minio-hal:9000 --access-key-id miniokey --secret-access-key
    hal config canary aws edit --s3-enabled=true
  4. Optionally set the defaults for created accounts:
    hal config canary edit --default-metrics-store prometheus
    hal config canary edit --default-metrics-account my-prometheus
    hal config canary edit --default-storage-account my-minio
  5. Deploy the instance of Spinnaker:
    hal deploy apply
    If you don’t want to apply configuration through the hal cli commands, you can adjust your halconfig with the canary section and pass it to the halyard chart directly:
        enabled: true
        - name: prometheus
          enabled: true
          - name: my-prometheus
              baseUrl: http://prom-prometheus-server:80
            - METRICS_STORE
        - name: aws
          enabled: true
          - name: my-minio
            bucket: spin-bucket
            rootFolder: kayenta
            endpoint: http://minio-hal:9000
            accessKeyId: miniokey
            secretAccessKey: miniosecret
            - OBJECT_STORE
          s3Enabled: true
        reduxLoggerEnabled: true
        defaultMetricsAccount: my-prometheus
        defaultStorageAccount: my-minio
        defaultJudge: NetflixACAJudge-v1.0
        defaultMetricsStore: prometheus
        stagesEnabled: true
        templatesEnabled: true
        showAllConfigsEnabled: true
    You can get more information and more details on how to use halyard chart in How to Deploy Spinnaker on Kubernetes: A Quicker and Dirtier Guide.

Create the test workloads

We need some application to provide metrics that we are going to analyze. You're typically going to deploy two versions (canary and baseline) during your canary pipeline execution, so you'll have something like two deployments in a “prod” namespace, such as:
kubectl get deployment -n prod
demo-app-baseline    1      1      1         1        31m
demo-app-canary      1      1      1         1        31m
Note that you're typically not going to deploy these manually; they're part of your pipeline. So you'll need to make sure that they (or services that are backed by them) have the appropriate annotations that will direct them to Prometheus.
In the case of the sample app these deployments are running two versions of a simple flask application that exports just a few metrics in the prometheus format.
There we also have two services on top of them:
kubectl get svc -n prod
demo-app-baseline   ClusterIP   <none> 80/TCP,8000/TCP   32m
demo-app-canary     ClusterIP    <none> 80/TCP,8000/TCP   32m
Metrics are exported on the 8000 port, so the following annotations have been added for services to be scraped by the prometheus server:
apiVersion: v1
kind: Service
  annotations: / "8000" "true"

Create the Spinnaker Application

Now let’s go to the Spinnaker ui and create an application.  (Look for the hal-deck-hal service, on port 9000.) Start by creating an application:
Enable Canary in features screenshotOnce we've created the application, we need to enable “Canary” in the Features tab of the application config:
Add configuration screenshotIf you can see new Canary options in your Spinnaker UI under Delivery, then we are ready to create our first configuration. Click Canary Configs, then Add Configuration.
Prometheus screenshotThe process of canary configuration creation is very well described in the Spinnaker docs so we will only focus on a Prometheus-related part here.
Now we will take one of the exported metrics and add it into our canary configuration. The sample application is designed to demonstrate the flask_request_count metric, but if you're running a different one, use a metric you know is being emitted by your application.
Let’s query the metric in Prometheus web view to get its labels:
Canary analysis screenshotFor the configuration we need some labels to differentiate the baseline and canary instances. As you can see by looking at the name-value pairs for each instance, we can use combination of “release” and “kubernetes_namespace” to identify the instance of our application. Let’s take a look at the fields that we have in the Metric Scope of the Canary Analysis stage:
Configure metric screenshotThe values of the Baseline and Canary fields will match our metric’s release labels, Baseline Location and Canary Location will match the kubernetes_namespace labels.
Based on it we can go back to our Canary Configuration and create a template for our metric.
To do that  click the Add New Template button under Filter Templates.
ScreenshotHere in our template we’ve assumed that we are only interested in responses with 500 code. We've also added two variable bindings (scope and location) that are implicitly available for all templates.
As you might guess, depending on the target for which the query is made, ${scope} will be interpolated with either “Baseline” or “Canary” and ${location} with “Baseline Location” or “Canary Location”.
Now let’s add the metric itself:
ScreenshotA good way to verify if Kayenta is able to communicate with Prometheus is to get the autocompletion for the “Metric Name”. Let’s give our metric a name, choose previously created “Filter Template” and metric itself in the “Metric Name” field:
ScreenshotFrom this point, configuration is pretty much the same as for any other provider. You might want to add additional metrics, create groups of metrics, and so on. Don’t forget to assign proper weights for the groups and save the configuration. When you done, canary config is ready to be used in “Canary Analysis” stages.
So that's how to create a canary configuration using Kayenta and Prometheus. You can then use it in your canary deployments.
Intrigued?  Want to learn more about Spinnaker?  Check out our SPIN50 class to learn how to get from code check-in to production using Spinnaker in just one day of focused lectures and labs.

Andrey Pavlov

Andrey Pavlov is a software engineer at Mirantis.

Choose your cloud native journey.

Whatever your role, we’re here to help with open source tools and world-class support.


Subscribe to our bi-weekly newsletter for exclusive interviews, expert commentary, and thought leadership on topics shaping the cloud native world.