Glance API V3 and the OpenStack Community App Catalog
The Community App Catalog is a convenient way for users to find applications they can deploy to their OpenStack clouds, but what happens behind the scenes can be just as important to both users and developers. I am writing this article to clarify our vision of what Glance V3 is and how its features may be used to provide the REST API for the Community App Catalog, and what that can mean for OpenStack.
1. Versioned schema
First of all, it's important to understand that the Glance V3 API operates on entities called “artifacts”, and that these artifacts perfectly map to the Data Assets of the Community App Catalog. Artifacts are strongly typed: there are artifact types for murano packages, glance images, and heat templates - and there may be (and will be) more. Each artifact type is represented by a plugin defining the schema of the objects’ data and metadata and - optionally - any custom logic. This structure makes the catalog extensible: when a new type of asset needs to be added to a catalog it can be done really quickly by just defining the schema and putting that schema into a plugin. Also, these plugins are versioned, so the possible changes in the schema are handled properly.
2. Generic properties
Next, all the artifact types in Glance V3 have some generic metadata properties (i.e. part of the schema which is common for all the types), including the name, the version, description, authorship information and so on. This also corresponds to the data schema of the Community App Catalog. Because the catalog and Glance were developed independently for slightly different purposes, the mapping is not 1:1, but the two project teams can easily work together on this to make sure that these generic properties match the expectations of the catalog. (In fact, we plan to discuss it at the OpenStack Summit next week in Tokyo.)
3. Versioning
Versions are very important for catalogs of objects, so when we designed Glance V3 initially, we kept versioning questions in mind: each artifact has a semver-based version assigned, so artifacts having the same name but different versions are grouped into the proper sequences. The API is able to query artifacts based on their version spec. For example, it is possible to fetch the latest artifact with the name “foo” having a version greater than 2.1 and less than 3.5.7 -- or any other version spec, similar to the way this situation is handled by pip or other tools. As far as I know, the Community App Catalog does not have these capabilities right now - and I strongly believe that it is really a must-have feature for a catalog to be successful. (At least it is absolutely mandatory for Murano packages, which are the only “real apps” among the asset types right now.)
4. Cross-artifact dependencies
Glance V3 has also had dependency relations from the very beginning, so they may be defined as part of an artifact's type schema. As a result, an artifact may “reference” any number of other artifacts. For example, a murano package may define a set of references to other murano packages and call it “requires” - and this will behave in a way similar to the requirements of a python package. Similar properties may be defined for heat templates and glance images - they may reference each other with various semantics.
Of course, these dependency definitions may be done internally inside the packages, so they may be resolved locally by the service using the app, but letting the catalog know about them enables us to do the import-export operations for any given artifact and its dependencies automatically, using only the catalog.
5. Search and filtering API
So far we've been talking about functionality that exists. Right now the Glance V3 API is in the "experimental" state (we plan to stabilize it during the Mitaka cycle), but it already provides quite good capabilities for discovering things. It can search artifacts by their type, name and (optionally) aforementioned version specs, by tag, or even by an arbitrary set of metadata properties. We have plans to integrate Glance V3 with the Searchlight project to have even more index and search capabilities using Searchlight's (currently) Elasticsearch-based engine.
6. Data storage
As you probably know, Glance does not own the binary data of its images. Instead, it provides an abstraction of the backend storage, which may be Swift, Ceph, S3, or even something else entirely. Glance V3 uses the same approach for artifacts data, but with more per-type control; particular artifact types may be configured independently to store their blobs in different backends. This capability can come in handy, because different types of assets (and usage patterns) are best optimized on different types of storage. This aspect of asset storage could be of particular importance for a large public catalog such the OpenStack Community App Catalog, which operates on different types of storage for its assets. For example, a Heat template is just a text file, but a VM is a large blog of data that may be best served out of a CDN.
7. Sharing and access control
Glance V3 inherits the same access mechanics present in Glance V2: an artifact may be only be visible to its owner project, it may be public (that is, visible to all projects), or it may be directly shared by the owner to a specific project. Also, Glance can act in "anonymous mode" (that is, without an access token), thus providing access to public artifacts even to unauthenticated users.
You can easily apply this idea to a public web service such as the Community App Catalog: regular unauthenticated users use anonymous mode to access only the public assets (this is the current behavior of http://apps.openstack.org), while registered users will have their own private spaces (“projects”) with various access restrictions.
8. The federation
The ultimate goal for the Glance Artifact Repository is to have the ability to build trees of artifact repos in different clouds. The top node of that tree is some Global Application Catalog (and the Community App Catalog at http://apps.openstack.org may be this global catalog - if it is glance-based or at least supports Glance V3 federation), then there are repositories of particular openstack vendors or distributors, and then finally, the catalogs of enterprises operating different openstack clouds.
In this situation the particular glance deployments in those clouds are the leaves of that tree, being able to search for data assets in all the upstream repositories, download them from there or - if permitted - submit their local assets back upstream. This will be the ultimate network for application delivery and exchange in the OpenStack world - and this is one of the main reasons we’ve begun the Artifacts initiative in Glance.
Unlike other aforementioned features this one is not implemented yet, but we are planning to add it as soon as we have stabilized the API itself.
Other things we still need
There are many other features which are present in V3’s roadmap and may be useful for the app catalog, such as the ability to sign artifacts with their developers’ key and verify that key on usage to ensure the authenticity of the artifact.
Another thing we don’t have right now is the ability to associate ratings (“stars”) and comments to the artifact, as well as to aggregate different usage and download statistics: these features are really needed only for a public website such as http://apps.openstack.org, and are not strictly required for Glance’s artifacts in particular clouds. But we may still find it helpful to find some way to solve this, either by wrapping the Glance API with additional middleware which would add appropriate information from a different data source, or by having custom plugins, or in some other way. I am sure we can find a solution for this particular problem.
In conclusion...
This was just a brief description of what Glance v3 has to offer as a backend for the App Catalog API. It also worth reiterating that this API is in the “EXPERIMENTAL” state right now, which means that it is not yet fixed; we may modify it significantly if there is a need to do so. Far from being a problem, however, it makes it that much easier for us to work closely with the Community App Catalog team to adapt it for their needs.
Ultimately, I would really prefer to not create any overlaps between Glance v3 and the Community App Catalog: if the app catalog builds its own incompatible implementation of an asset discovery and distribution API, then we’ll have a huge duplication of effort for developers and lots of confusion for end-users who will get two entirely different ways to do the same task.
So, I’d propose to discuss these potential overlaps, look at the features need by the Community App Catalog and see how Glance V3 may be of use here. I’ll be more than happy to help with that. We will dive deeper into the details in Tokyo, and we would love for you to join us there!
