OpenStack Project Technical Lead Interview Series #4: Thierry Carrez, Chair of the OpenStack Technical Committee & Release Manager
This post is the fourth of a continuing series of interviews with OpenStack Project Technical Leads on our Mirantis blog. Our goal is to educate the broader tech community and help people understand how they can contribute to and benefit from OpenStack. Naturally, these are the opinions of the interviewee, not of Mirantis.
Here the interview is with Thierry Carrez, Chair of the OpenStack Technical Committee and Release Manager.
Mirantis: Can you please introduce yourself?
Thierry Carrez: Sure! I turned 40 this year. I'm based in a small village in the center of France, where I live with my wife and two daughters.
Q: Lovely! Do you see any cultural differences when engaging with developers from all other the world?
A: Yes, those cultural differences are pretty apparent, and it's one of the challenges of a global community to form a united whole that goes beyond those differences. That said, there are also cultural differences due to various professional backgrounds, so it's not purely a geographical issue.
Q: Can you give a few examples? In Eastern Europe feedback on code contributions, for example, is perceived as criticism and disapproval - while it's actually meant to help developers improve their code.
A: Taking a "-1" in a code review as personal criticism is sometimes also an issue in Japan, for example. You also sometimes see with developers coming from big companies a fear in having their names directly (and permanently) attached to code commits. It prevents them from fully engaging with the developer community. This is really not an issue with developers with an open source or startup development background, who see this transparency as an opportunity.
Q: What is your history with OpenStack? Why do you engage?
A: I was the Technical Lead for Ubuntu Server at Canonical, working on the Ubuntu Enterprise Cloud product. When OpenStack was formed, I followed a few colleagues to Rackspace to help organize the project and fill release management duties.
I'm particularly interested in promoting the "open innovation" development model: creating a level technical workspace that lets developers coming from all over the industry collaborate to openly design and openly develop OpenStack. My goal in the project is to ensure we stay true to this model, to help in maintaining and improving it, and to prove that it can be wildly successful.
Q: How do companies adopt the "open innovation" development model?
A: In an open innovation setting, companies employ developers to work on the project, but they can't control the project itself. They ask their developers to channel their priorities and interests within the project, like ambassadors. But ultimately, the body of contributors and the technical meritocracy derived from it is in control. Companies have to accept to trade control for influence.
Q: Many companies are using OpenStack but they do not share their experiences. Is that something that needs to change? If so, how? Or is that just common and tolerable practice within an open source project?
A: The Apache license allows companies to take the code and not contribute back, so this is definitely acceptable practice. That doesn't make it a good idea, though. By not participating to the project you drastically reduce your influence on the technical side. You also face the risk of having new code and added features break your secret sauce. In the end it's just not worth it.
Q: Who determines how OpenStack is being built - what does that decision process look like?
A: Our feature development process is extremely open. Anyone can propose a change and it will get reviewed by our core reviewers. That said (and especially for disruptive features), you increase your chances of successful review by engaging early on the design on the development mailing-list and participating in our Design Summits. That way the feature can be blessed by the community before too much work is spent on implementation details. The decision process works by lazy consensus among the developers of the project. For each project we have an elected Project Technical Lead who can make the final call if one needs to be made, but it's been extremely rarely needed.
Q: What are your responsibilities as the Chair of the Technical Committee and Release Manager?
A: The Technical Committee is the final expression of the technical meritocracy in OpenStack. It is elected by the contributors to the various projects. Its role is to look into new projects as they are proposed for incubation into OpenStack, and to address potential cross-project issues. As the chair of the committee I'm responsible for keeping track of the items that are brought to the attention of the committee, running our IRC meetings, and communicating the outcome of those meetings to the rest of the world.
The release management has two aspects. One is (obviously) to get the coordinated releases of the various OpenStack projects out, but with the tooling and automation we now have, that part is quite simple. Most of the release management work is more about handling our 6-month release cycles, keeping track of what's being done and trying to prevent cross-project disruption.
Q: Apropos of IRC, we hear from some people interested in OpenStack, "Why do you guys use IRC? It's so old school. Why don't you use Twitter, like everyone else does today?"
A: Eh. Not everyone likes to be tied to a proprietary messaging platform! I've been using IRC for the last 20 years, and it is perfect for quick online discussions and meetings. We have topic-oriented channels and team-oriented channels. It's our public watercooler area which works 24/7. We can log it so everyone can access that data forever. We can use proxies to stay connected. It's very common in our open source developers target audience, and most of us use it outside of OpenStack anyway.
Q: You mentioned the 6-month release cycles. What is the value of release cycles?
A: The most obvious value of release cycles is to help producing a release. It allows us to shift focus from feature development to release-critical bugfixing, which results in better quality in the end release. But to me, the most important value of release cycles is that they create a common rhythm of contribution, a common pulse, which is essential for our virtual and global community to feel part of the same project.
Q: Can you explain what an OpenStack release cycle looks like, from brainstorming, through implementation, to integration?
A: We start at the Design Summit, where developers who want to work on something specific during the next cycle will come to present their ideas and discuss them with the rest of the developers. Then comes implementation time, which we artificially split into three sprint-like "milestones" to encourage people to land their work as early as possible.
After the third development milestone we switch to feature-frozen mode, where the focus is on fixing a list of release-critical bugs. When we bring that list down to zero, we build a release candidate. Unless new release-critical bugs are found in it (in which case we fix those bugs and respin a new release candidate), those candidates are assembled at release date to form the final integrated release of OpenStack projects.
Q: What determines OpenStack’s release cycle frequency?
A: We strive to have an always-runnable master branch. "Releases" are actually just points in time, tags in our git repositories that we special-case. Our release cycle is about reducing the risk that a critical regression finds its way in our "releases", so they are less risky than a random master commit. What's interesting is what comes next: we maintain stable branches where we backport critical bug fixes and vulnerability fixes, and those stable branches are created from those release points.
The current OpenStack release cycle frequency is a trade-off between our will to iterate fast, the frequency at which we can have our Design Summits, the need to give time for documentation to catch up after we go feature-frozen, how often we can afford to freeze development, and the resources we can dedicate to stable branch maintenance.
Q: What would speak in favor and against a 3-month release cycle?
A: You may remember that we used to have 3-month cycles at the beginning of OpenStack. Currently, it takes us about 4 weeks after we stop adding features to come up with a valid release candidate. In a 6-month release cycle, it's acceptable to be feature-frozen for one month. In a 3-month release cycle, less so. Releasing every 3 months also means maintaining twice as many stable branches. So if more people addressed critical bugs during the rest of the cycle (when we are not feature-frozen) and more people helped with stable branch maintenance and security updates, we could definitely consider going for 3-month cycles. I like to have a Design Summit at the beginning of each cycle (I think it helps us deliver better results), so we'd probably also have to convince the Foundation to pay for twice as many developer events!
Q: How can developers participate if they are not able to attend, or cannot afford to travel to the Design Summits?
A: Participating to the Design Summit is by no means necessary to get your code into OpenStack. Discussions can also happen on the mailing-lists and our code review system. The Design Summit is just a convenient way to go faster through the brainstorming and early implementation stages, by receiving early feedback in lively in-person discussions.
Q: Has the OpenStack community considered following a similar path the the Ubuntu community, with an "online only" design summit? Would that work for OpenStack?
A: Ubuntu Developer Summits turned mostly into a feedback gathering exercise where Canonical developers present what they will work on for the next months. For that use case a virtual setting works well and saves you a lot of money. OpenStack Design Summits are used for other things: to motivate our global and virtual development community for the next 6 months, brainstorm design ideas, or reduce duplication of effort by engaging with various groups of developers. But the most important reason to keep in-person contributors gatherings is that they help maintain our community's sanity. Meeting in-person from time to time is necessary to settle any resentment that may have built up over 6 months of virtual interactions.
Q: We just recently interviewed Monty Taylor, Project Technical Lead for CI. In your opinion, why does Continuous Integration matter for OpenStack?
A: Continuous Integration lets us trust that what we have in the master branch is working and that no regression was inserted. OpenStack is a complex beast, and it's really easy to inadvertently break things. In the early days, I used to deploy locally and run a few manual tests from time to time just to make sure we didn't break basic functionality.
Today's infrastructure runs thousands more tests than I used to run, and it does so for every proposed change. Our continuous integration makes me sleep better at nights. There is no way we would be where we are today without it.
Q: Are there still any areas of the development process that are not automated yet - but should be in the future?
A: No, it's all automated in some way, though that automation can always be perfected. Some projects could use more integration tests: your continuous integration is only as good as the tests you run.
Q: What should developers bring to the table who want to participate in writing OpenStack?
A: A good sense of humor. OpenStack developers are a bunch of pretty nice folks, and it would be great if we could keep it that way forever !
Q: What advice would you give to newbies -- both developers willing to contribute, and companies that want to build their businesses around OpenStack?
A: OpenStack is what we all make of it. Dive in, you won't regret it.
Q: Thank you very much, Thierry.