NEW! Mirantis Academy -   Learn confidently with expert guidance and On-demand content.   Learn More


Rally: OpenStack Tempest Testing Made Simple(r)

Andrey Kurilin - May 29, 2014

Cloud users today often also need to be cloud innovators: upgrading components, integrating new hardware, scaling up, and making lots of configuration changes. But how can you be sure the changes you make don’t break anything?

Testing using Tempest is (part of) the answer. Tempest, OpenStack’s official test suite, is a powerful tool for running a set of functional tests against an OpenStack cluster. Tempest automatically runs against every patch in every project of OpenStack, which lets us avoid merging changes that break things.

Historically, however, there have been a few roadblocks to using Tempest. Accessing its power reminds me of being back at University, learning Assembler by coding inline with a high-level language IDE like Turbo Pascal. It’s sure great to have the IDE there to run the system assembler for you, link the code, and report errors. But, all in all, you’d prefer to be writing Pascal or another high level language.

Tempest complexity can make it difficult and time-consuming to use. Tempest also lacks certain features useful for benchmarking, like the ability to generate simulated loads and present results in graph form.

The solution for these problems is Rally, an OpenStack project to create a framework for validating, performance testing and benchmarking OpenStack at scale with Tempest. Rally automatically installs and configures Tempest, and automates running Tempest tests. In contrast to Tempest, which is installed and executed individually on each cluster, Rally can verify a huge number of clouds -- just add clouds as deployments to Rally, then easily switch between them. Rally’s benchmarking engine can automatically perform tests under simulated user loads. Results of these tests and benchmarks are saved in Rally’s database. You can review them immediately or schedule many tests over time to see how configuration changes are affecting your cloud.

In this article, we're going to look at how to use Tempest and Rally, but before we get to the fun part, here are the main reasons why we’ve integrated Rally with Tempest:

  1. Most OpenStack projects use CI/CD systems that employ Tempest as a ‘gate’: for automated testing of committed changes. So it makes sense for us to help developers use Tempest more widely and with greater ease.

  2. Benchmarking with Tempest tests is efficient:

    1. Tempest already implements a multiplicity of tests, so there’s no need to recreate the same tests in Rally. Less code duplication => better world.

    2. Verification should not be limited to testing with a single user. Rally can simulate loading for Tempest. In other words, Rally Benchmark can launch Tempest tests with a variable number of (simulated) active users.

  3. Debugging tempest & OpenStack. If there are race conditions in gates developers will be able to run the same test (or set of test) under load on their environment => which will simplify detecting source of race condition.

Sounds reasonable. But you'll never know how good Rally is until you try it. So let’s go hands-on!

Install Rally

First of all, you should download Rally from the github repository and run the script “” to install it.

screenshot of install window for Rally

Add your Cloud to Rally

Rally lets you add clouds in two ways. You can source the openrc credentials of your cloud and create the rally deployment from your environment, like this:

screenshot of Rally Konsole when adding a cloud with opencrc credentials

Or you can describe your cloud in json (see link for instructions and details) and offer that to ‘rally deployment create’ using the --file argument:


After adding your deployment, you can use `rally deployment list` to display a list of all known deployments.

screenshot of Rally Deployment List

Check your Cloud

Use the command `rally deployment check` to see if Rally has been deployed correctly, and to determine which Rally services are available:

screenshot of Rally Deployment Check

Prepare Tempest for your Cloud

As mentioned earlier, Rally can customize Tempest for a huge number of clouds. To decrease the time required for cloning the Tempest repository, Rally clones it once, and then uses a local repository to reproduce it on each cloud.

The command `rally-manage tempest install` takes care of cloning the repository, generates the configuration file for Tempest and installs the virtual environment with all dependencies.

screenshot of Rally Tempest installation window

Run Tempest

The command `rally verify start` launches auto-verification. This command expects only one argument: the test set name. If this argument is not specified, smoke tests (the default) will be executed. Output of this command is similar to the Tempest output.

Valid test set names include: full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration, telemetry, and volume.

screenshot of results from running Tempest on Rally

List all Tempest Results

Results of verification tests are stored in Rally’s database, so you can view and compare them. Lists of verifications can be displayed with the command `rally verify list`.

screenshot of Rally Verified List of Tempest results

Show Detailed Information for One Execution

Detailed information for one execution can be derived with two commands: `rally verify show` and `rally verify detailed`. The latter displays tracebacks for failed tests.

screenshot Rally detailed information window showing one execution

screenshot Rally detailed information window showing multiple failed tests

Benchmarking with Tempest Tests

In addition to installing, configuring and running Tempest, Rally will also store and display results for multiple tests, under simulated user loads.

Let's run benchmarking for the test scenario  tempest.api.identity.admin.v3.test_domains.DomainsTestXML.test_create_update_delete_domain, with simulated loads ranging from 2 to 10 users. Once the test is finished, we’ll generate an HTML file of the result, including graphs. Information about the JSON/YAML file used to define this task can be found locally in rally/doc/samples/tasks, or on the OpenStack wiki.

screenshot of Rally window showing benchmarking for the test scenario

For Two Active Users:

screenshot of benchmark test for two active users

For Six Active Users:

screenshot of benchmark test for six active users

For Ten Active Users:

screenshot of benchmark test for ten active users


When you deploy and run a cloud, you must cope with constant change, so how do you verify each change? Tempest is a good solution but is cumbersome to use. Rally makes this simple.

Rally provides benchmarking for OpenStack, and now, thanks to Tempest integration, the scenario library is huge. Will your patch make some operations faster? Is your deployment as fast as your competitor’s? Why not find out with Rally?

Choose your cloud native journey.

Whatever your role, we’re here to help with open source tools and world-class support.


Subscribe to our bi-weekly newsletter for exclusive interviews, expert commentary, and thought leadership on topics shaping the cloud native world.