Real World Business Metrics from Large-Scale OpenStack Superusers

What business outcomes are OpenStack users looking for? How are the most successful OpenStack users winning with open cloud? How (and by how much) do they accelerate their application development cycles? How do they measure progress?

We get answers to some of these questions—along with some anecdotal evidence—from the hundreds of successful user stories compiled by the community from across a broad range of industries. Mirantis’ own case studies have shown that our customers are looking for—and experiencing—increased agility and faster software release cycles.

However, answers weren’t easily forthcoming when it came to concrete measurable business outcomes like:

• Quantified cost savings
• Velocity metrics on the increased speed of innovation and application delivery
• How automation improves operational efficiency

So, Mirantis—in collaboration with Intel—jointly decided to uncover more quantitative and concrete business metrics. Together, we commissioned a consulting firm, Catalyst Strategies, to formulate and conduct an independent study of OpenStack “superusers” to capture specific KPIs for business success and concrete business outcomes. (Superusers are organizations with one or more years' experience running anywhere from several hundred to tens of thousands OpenStack nodes in production.) We wanted to find out exactly how the most successful, largest-scale OpenStack users were winning with cloud and how they were applying OpenStack to real business problems. In short, what, exactly, did “winning with cloud” mean to these big players? And how much were they actually winning?

Catalyst Strategies’ Study

Catalyst selected superusers from across a broad range of industries, from technology to retail. They conducted hour-long, face-to-face interviews with key participants in each superuser's cloud journey to gather concrete metrics and success factors. These interviewees were not just CTOs and others in the IT operations and management side of the business, but came from the “user” and business owners actually consuming these IT services as well—Catalyst spoke with key anchor tenants, development team leads, and other cloud service consumers that work to support the business bottom line.

Interviewees were assured anonymity so they could share more freely. We can't tell you who they are, but what they told Catalyst was pretty amazing.

Business Outcomes Driven By OpenStack

Despite the range of industries represented in the study, and the even wider range of specific use-cases explored, interviewees turned out to have prioritized a surprisingly-uniform and conceptually-simple set of critical success metrics and KPIs. The biggest game-changers, they said, were:

• Faster time to market by speeding-up provisioning time, which led to quicker development release cycles.
• Getting code-changes tested and deployed to production faster while maintaining high-availability for applications and good user experience. The interviewees on average claimed that the overall time to develop software was up to 4x faster.
• Faster time to market through quicker and easier deployment.
• Lower infrastructure costs through increased utilization and greater vendor independence.
• Lower operational costs through improved productivity of both developer and operations personnel.
• More flexibility to respond to peak demands through faster provisioning and better capacity management.
• Improved risk management through faster problem resolution, improved error and up-time rates, and reduced vendor dependence.

Here are some of the superusers’ reported results:

• Quicker provisioning times: OpenStack superusers all told stories of massive gains— from half-year bare metal and two-month VM provisioning times, to delivery of VM platforms in minutes—with the added benefit of being able to locate that compute capacity in a desired geographic location.
• Faster deployment times: Aggressive adoption of DevOps automation, coupled with open standards, enables superusers' to have multi-site and hybrid public/private application hosting strategies, as well as decrease app deployment times from months to minutes.
• More releases per year: In some cases, superusers had already been able to speed-up software development but were being held back by long platform provisioning times. At one Fortune 500 financial services company, developers had established a two-week cadence for new releases of their fast-growing app but a required new infrastructure for each release was taking six or eight weeks to deploy, thus extending their effective release cycle to 42 days. By moving to OpenStack, the company was able to decouple underlying infrastructure scaling from release-required provisioning and brought their provisioning time per release down to only ten minutes. Instead of six releases per year, they now do 24 or more—a 400% improvement!
• More features per release: By simplifying dev/test and accelerating deployment, OpenStack frees developers to create better, more feature-rich releases. Agile/Scrum teams tend to measure this richness in terms of “story points”—the comparative difficulty of providing a requested feature (and not related directly to actual hours worked). Superusers credited OpenStack with helping them enrich releases by an average of 20 to 30 percent more story points and as much as 60 percent in some cases.
• Lower cost per VM: A simple yet dramatic metric used by many respondents in the study. Total VM cost per month includes costs of hosting hardware, OS license, upstream network ports, cost of storage, and allocation of engineering time for network and data center provisioning and management. On average, respondents quoted a 95 percent reduction in cost-per-VM averages (from $600 per month to $30 per month with OpenStack).
• Improved capacity utilization: OpenStack's accelerated provisioning lets superusers scale up and down as needed, eliminating the incentive to reserve bare-metal servers or overprovision and hoard complex virtual infrastructure against the possibility of traffic spikes (thus underutilizing capacity). OpenStack adoption has improved utilization for some superusers from an average of around 4 percent to as high as 60 percent.
• Pivoting sysops, release and reliability engineers to creative functions: Superusers agreed that OpenStack, driven by automated provisioning and deployment, requires fewer dedicated operators and specialists overall to architect and manage critical releases. One respondent had been able to pivot all their dedicated sysops—from two to ten people supporting each product team—into other roles, reducing dedicated sysop headcount by 98 percent.

More to Learn from Superusers

As these superusers explained, gaining organizational agility and enjoying big cost savings—winning with open cloud—is possible and worthwhile for almost any organization. There are several additional benefits that the study discovered, including secrets of success in terms of the criteria and ways of approaching the culture change that worked for many of these companies.

I'll be revealing much more detail from this landmark study in "How Four Superusers Measure the Business Value of their OpenStack Cloud," a talk at OpenStack Summit Barcelona. Click the link for details.

See you there!