According to last October’s OpenStack user survey, QA test environments are one of the top ten workloads running on OpenStack clouds. In this post, I’ll describe how staging environments are built, and explore ways that OpenStack can make this process easier and more efficient.
Hardly anyone would argue about the fact that the most important requirement for a test environment is that it be a copy of the relevant production environment (though this is not, in fact, always 100% true – see below). Any other requirements and constraints emerge from your software usage scenarios and testing methodology.
Before going to production, different types of system tests need to be performed, from functional tests to parallel tests. For example, in pre-production testing, you might:
If your production version runs on baremetal, each test environment will need a dedicated server cluster. Hardware, network configuration, software settings and data (anonymized of course) should be mirrored from the production environment to both these test environments. Ideally, both test environments should be built on identical hardware and have as similar a networking configuration as possible.
In most cases, IT administrators manage and allocate such testing environments — testers can’t create on-demand staging environments any time they need them.
Given the above, it’s not surprising that everyone is struggling with testing. Consider these facts:
It would be much better if testing environments were:
For those who know what OpenStack offers, it sounds like a perfect match. OpenStack is open source, free, and can be deployed on cheap commodity hardware. It lets you preconfigure components of a test suite and complete, virtualized test environments, then duplicate these rapidly at any desired scale. Finally, OpenStack has a self-service portal (Horizon) where users can access on-demand resources for:
Before starting to think about cloud design, you need to ask yourself:
If the answer to either question is no, then cloud (and OpenStack of course) may not be for you. But it must be understood that this is a conservative rule, exempting more than a few edge cases. For example, it’s possible to build quite useful cloud-based test environments behind a production instance that runs on baremetal. Doing so, however, demands a deep understanding of the application(s) under test, and the technical ability to:
Also remember: cloud is evolving very fast, and the relationship between VMs and underlying hardware is increasingly configurable, enabling ever-better fine-tuning of VM performance and characteristics. Some careful testing might be in order before you write cloud off as a possible solution for both production and testing.
If you know your application will perform well, and test meaningfully in a cloud environment, however, then cloud (and probably OpenStack) is clearly a good choice.
Let’s see how a cloud-based staging environment can transform the picture.
What would it take to replicate a stack that your application runs on in the virtual environment?
Let’s assume that you build an OpenStack cloud based on existing hardware.
The first, and most important thing you should think about is the cloud architecture, since many aspects must be taken into account. The more layers you have in a solution stack, the more careful you should be when designing each level. Here are some helpful tips.
First and foremost, if your application is “cloud ready”, you can’t go wrong with HA. Even if your cloud won’t run production applications, it needs to support pre-production testing, and should therefore be highly available. (For more information, you can check out Mirantis OpenStack’s implementation of HA.
You should also check out a concise rundown of methods for optimizing compute performance in OpenStack-based cloud environments.
Other design decisions, such as:
… depend on software characteristics, specific data communications and traffic-shaping requirements (e.g., for big bandwidth, low latency, QoS, response time) and testing workflow. Try to find answers to questions such as:
It will be helpful to assemble a list of requirements and transform them to architectural decisions.
Once you’ve decided on your architecture, it’s time to start preparing VM images. Think about making:
The goal, of course, is to develop a library of images to enable rapid deployment of diverse test environments whose components are themselves configured in a disciplined way, pre-documented, and pre-tested.
How will you provide on-premise testing environments?
Provisioning can be done with Heat: the OpenStack component responsible for cloud applications orchestration.
Heat offers a templates mechanism. A Heat template describes the infrastructure for a cloud application, e.g., servers, volumes and their connections, networking settings, including floating ips, security groups, authentication settings, etc. For automatic software configuration, puppet or chef can be used.
This means that one general Heat template can be created for different types of test environments, provided these have similar networking settings and virtual cluster configurations. Specific virtual test environment features can then be added either with a customized Heat template or by manual configuration.
With a cloud-based solution, testers can own their test environments: creating, managing and deleting them, without IT intervention. IT, meanwhile, typically owns the hardware cluster and manages the cloud environment.
So far I’ve provided a generalized approach to building virtual test environments. Now, let’s look at a specific use-case.
Our subject is a web software company, developing a consumer/business application. The app in question has basic availability requirements, and is likely to meet high levels of market demand at introduction. So the company needed to engineer a production platform that offered sub mission-critical HA reliability; and this solution needed to scale very rapidly and cleanly with traffic and demand. Scaling, moreover, would need to be clean in both directions – there are cyclic aspects to this client’s business that could make periodic infrastructure scale-backs necessary for cost-efficiency.
The application uses the standard LAMP stack of back-end technologies. At the opening of our study, the intended production environment was running on VMWare: not baremetal, but still relatively high-cost due to licensing and fees. Their test environment was also running on VMWare, and was maintained by IT administrators.
The company had already identified some problems:
The company wanted to change their approach to building test environments. They wanted their new solution to fulfill these requirements:
The solution architect proposed the building of a test environment using OpenStack. The recommended approach was to build a 10-20 node OpenStack cluster with HA for controller nodes. The resulting solution, when complete, delivers the following benefits, among others:
Building test environments can be a very complicated process, its specifics dependent on the type of software under test, desired testing methodology, people responsible for maintenance of testing environments, and other factors.
Of course, not every test environment can be built on OpenStack, and not every type of test can be performed in a cloud-based test environment. As a cloud operating system, OpenStack has it’s own constraints and issues. For example, it has some performance issues, which need to be taken into account in designing and executing performance tests.
Hardware-dependent tests such as recovery testing can’t be done in a virtualized environment. Of course sometimes hardware emulators can be used, but it’s not the same.
The biggest take-away is: plan carefully. With good engineering and process discipline, many QA organizations will be able to specify cloud-based test environment solutions that will be robust, representative of production infrastructure (whether or not this is cloud-based), and that will solve serious workflow, efficiency, quality and cost issues.
[…] 如何搭建OpenStack测试环境 作者：Evgeniy Shumakeher […]March 7, 201401:33