Setting Up an External OpenStack Testing System – Part 1
February 17, 2014
This post is intended to walk somone through the process of establishing an external testing platform that is linked with the upstream OpenStack continuous integration platform. If you haven’t already, please do read the first article in this series that discusses the upstream OpenStack CI platform in detail. At the end of the article, you should have all the background information on the tools needed to establish your own linked external testing platform.
What Does an External Test Platform Do?
In short, an external testing platform enables third parties to run tests — ostensibly against an OpenStack environment that is configured with that third party’s drivers or hardware — and report the results of those tests on the code review of a proposed patch. It’s easy to see the benefit of this real-time feedback by taking a look at a code review that shows a number of these external platforms providing feedback. In this screenshot, you can see a number
Verified +1 and -1 labels added by external testing systems on a Neutron patch
Each of these systems, when adding a
Comments added to a review by the vendor testing platforms
The developer submitting a patch can use those links to investigate why their patch has caused test failures to occur for that external test platform.
Why Set Up an External Test Platform?
The benefits of external testing integration with upstream code review are numerous:
Why might you be thinking about how to set up an external testing platform? Well, a number of OpenStack projects have had discussions already about requirements for vendors to complete integration of their testing platforms with the upstream OpenStack CI platform. The Neutron developer community is ahead of the game, with more than half a dozen vendors already providing linked testing that appears on Neutron code reviews.
The Cinder project also has had discussions around enforcing a policy the any driver that is in the Cinder source tree have tests run on each commit to validate the driver is working properly. Similarly, the Nova community has discussed the same policy for hypervisor drivers in that project’s source tree. So, while this may be old news for some teams, hopefully this post will help vendors that are new to the OpenStack contribution world get integrated quickly and smoothly.
The Tools You Will Need
The components involved in building a simple linked external testing system that can listen to and notify the upstream OpenStack continuous integration platform are as follows:
I’ll be covering how to install and configure the above components to build your own testing platform using a set of scripts and Puppet modules. Of course, there are a number of ways that you can install and configure any of these components. You can manually install it somewhere by following the install instructions in the component’s documentation. However, I do not recommend that. The problem with manual installation and configuration is two-fold:
A better solution is to use a configuration management system, such as Puppet, Chef,Ansible or SaltStack to manage the deployment of these components, along with a Git repository to store configuration data. In this article, I will show you how to install an external testing system on multiple hosts or virtual machines using a set of Bash scripts and Puppet modules I have collected into a source repository on GitHub. If you don’t like Puppet or would just prefer to use a different configuration management tool, that’s totally fine. You can look at the Puppet modules in this repo for inspiration (and eventually I will write some Ansible scripts in the OpenStack External Testing project, too).
Before I go into the installation instructions, you will need to take care of a few things. Follow these detailed steps and you should be all good.
Getting an Upstream Service Account
In order for your testing platform to post review comments to Gerrit code reviews on openstack.org, you will need to have a service account registered with the OpenStack Infra team. See this link for instructions on getting this account.
In short, you will need to email the OpenStack Infra mailing list an email that includes:
Don’t have an SSH key pair for your Gerrit service account? You can create one like so:
The above will produce the key pair: a pair of files called
Create a Git Repository to Store Configuration Data
When we install our external testing platform, the Puppet modules are fed a set of configuration options and files that are specific to your environment, including the SSH private key for the Gerrit service account. You will need a place to store this private configuration data, and the ideal place is a Git repository, since additions and changes to this data will be tracked just like changes to source code.
I created a source repository on GitHub that you can use as an example. Instead of forking the repository, like you might would normally do, I recommend instead just git clone’ing the repository to some local directory, and making it your own data repository:
Now you’ve got your own data repository to store your private configuration data and you can put it up in some private location somewhere — perhaps in a private organization in GitHub, perhaps on a Git server you have somewhere.
Put Your Gerrit Service Account Private Key Into the Data Repository
The next thing you will want to do is add your SSH key pair to the repository that you used in the step above that had you register an upstream Gerrit service account.
If you created a new key pair using the
If you did not create a new key pair (you used an existing key pair) or you created a key pair that wasn’t called
Set Your Gerrit Account Username
Next, open up the file
Set Your Vendor Name in the Test Jenkins Job
Next, open up the file
Save Changes in Your Data Repository
OK, we’re done with the first changes to your data repository and we’re ready to install a Jenkins master node. But first, save your changes and push your commit to wherever you are storing your data repository privately:
Requirements for Nodes
On the nodes (hosts, virtual machines, or LXC containers) that you are going to install Jenkins master and slaves into, you will want to ensure the following:
Setting up Your Jenkins Master Node
On the host or virtual machine (or LXC container) you wish to run the Jenkins Master node on, run the following:
The above should create an SSL self-signed certificate for Apache to run Jenkins UI with, and then install Jenkins, Jenkins Job Builder, Zuul, Nodepool Scripts, and a bunch of support packages.
When Puppet completes, go ahead and open up the Jenkins web UI, which by default will be at
Once you are done with that, it’s time to load up your Jenkins jobs and restart Zuul:
If you refresh the main Jenkins web UI front page, you should now see two jobs show up:
Jenkins Master Web UI Showing Sandbox Jenkins Jobs Created by JJB
Testing Communication Between Upstream and Your Master
Congratulations. You’ve successfully set up your Jenkins master. Let’s now test connectivity between upstream and our external testing platform using the simple
Note that the
In our Zuul configuration, we have two pipelines: check and gate. There is only a single project listed in the layout.yaml Zuul project configuration file, the
By default, the only job that is enabled is the
Before we do that, let’s go ahead and tail the Zuul debug log, grepping for the term “sandbox”. This will show messages if communication is working properly.
OK, now create a simple test patch in sandbox. Do this on your development workstation, not your Jenkins master:
Output should look like so:
Keep an eye on your tail’d Zuul debug log file. If all is working, you should see something like this:
If you go to the link to the code review in Gerrit (the link that output after you ran git review), you will see your Gerrit testing account has added a
Successful communication between upstream and our external system
Congratulations. You now have an external testing platform that is receiving events from the upstream Gerrit system, triggering Jenkins jobs on your master Jenkins server, and writing reviews back to the upstream Gerrit system. The next article goes over adding a Jenkins slave to your system, which is necessary to run real Jenkins jobs that run devstack-based gate tests. Please do let me know what you think of both this article and the source repository of scripts to set things up. I’m eager for feedback and critique.
(This post was originally published on Join-Fu.)