How do I write tooling for the OpenStack APIs? What are RESTful web services?
- May 11, 2016
In order to tackle this question, we need to first split this larger question into two focused questions: What is a RESTful Web Service, and how should I use the OpenStack API to develop my own tools? We will first define a RESTful Web Service and then provide you with the main two ways to write OpenStack tools, through Heat and using the OpenStack CLI.
In recent years REST has overtaken other styles of Web service development, such SOAP or WSDL, primarily because of its ability to provide reliable, scalable, secure, and easy to use and integrate Web services. Understanding the details of REST is important if you are an OpenStack developer so you can create such a service, but not so important if you want to use the OpenStack services. As a user you want to find the best way to consume the OpenStack API, and leave the complexity of REST to the producers of the API.
The main two ways to write OpenStack tools/scripts are Heat and OpenStack CLI. As a comparison of the two, think of Heat as a way to write important, sometimes complex, applications that are tested, and then released, and will be used to deploy critical applications. The CLI can be used to write quick and dirty scripts to automate repetitive, but not overly complex tasks. CLI is easier to learn and use - Heat has somewhat of a learning curve. If you are new to OpenStack I would start with the CLI. OpenStack CLI is going through a major overhaul in the past couple of cycles. The old, per program CLI are being replaced with the new, comprehensive ‘openstack’ CLI, which in time will replace the legacy CLI. For an example of how to use the legacy CLI you can start here. To use the new comprehensive CLI you can read the docs here.
If you are a python programmer, and are not put off by the complexity of the language, there is a third option available to you that fits somewhere in between the CLI and Heat in term of complexity and flexibility - and that’s the OpenStack Client Libraries. The python libraries allow you to make native python calls to the services, rather than invoking them via another CLI process. Sometimes the CLI has a bug, or it may not expose the full feature set of the RESTful API on its command line options, while the underlying libraries may be more general, exposing all the API options.
Each OpenStack program has its own python client library which you can use to consume the services of the program. For example, here is a link to get started with the nova client library.
What is a RESTful Web Service?
REST (or Representational State Transfer) is a way of writing stateless client/server applications - in other words, a software architecture style. There are many references available on the Web on what REST is or how to write a RESTful Web service. You can start with the wikipedia. In short, the most common way to implement a RESTful service is to use HTTP as the underlying transport, and if you do, every service request is an HTTP method, such as HTTP GET/PUT/POST/DELETE/PATCH etc.In recent years REST has overtaken other styles of Web service development, such SOAP or WSDL, primarily because of its ability to provide reliable, scalable, secure, and easy to use and integrate Web services. Understanding the details of REST is important if you are an OpenStack developer so you can create such a service, but not so important if you want to use the OpenStack services. As a user you want to find the best way to consume the OpenStack API, and leave the complexity of REST to the producers of the API.
How should I use the OpenStack API to develop my own tools?
OpenStack is a collection of independent programs. Each program has a RESTful API. The definitive API of OpenStack can be found here. Although this API can be consumed to interface to OpenStack to accomplish your task, using as simple a tool as curl, this is not the ideal way to write your tools. To use an analogy, all Linux devices are controlled using open()/close()/read()/write()/ioctl() system calls. However, to read or write a file you typically use higher semantic abstractions, such as command line utilities, or as a programmer, perl/python language abstractions, or standard I/O libraries of C/C++. OpenStack also offers higher abstraction layers for implementing your tool.The main two ways to write OpenStack tools/scripts are Heat and OpenStack CLI. As a comparison of the two, think of Heat as a way to write important, sometimes complex, applications that are tested, and then released, and will be used to deploy critical applications. The CLI can be used to write quick and dirty scripts to automate repetitive, but not overly complex tasks. CLI is easier to learn and use - Heat has somewhat of a learning curve. If you are new to OpenStack I would start with the CLI. OpenStack CLI is going through a major overhaul in the past couple of cycles. The old, per program CLI are being replaced with the new, comprehensive ‘openstack’ CLI, which in time will replace the legacy CLI. For an example of how to use the legacy CLI you can start here. To use the new comprehensive CLI you can read the docs here.
If you are a python programmer, and are not put off by the complexity of the language, there is a third option available to you that fits somewhere in between the CLI and Heat in term of complexity and flexibility - and that’s the OpenStack Client Libraries. The python libraries allow you to make native python calls to the services, rather than invoking them via another CLI process. Sometimes the CLI has a bug, or it may not expose the full feature set of the RESTful API on its command line options, while the underlying libraries may be more general, exposing all the API options.
Each OpenStack program has its own python client library which you can use to consume the services of the program. For example, here is a link to get started with the nova client library.
In other words...
To write tools for OpenStack you do not need to understand REST. For a user of OpenStack REST is simply a decorated HTTP URL, such as http(s)://<some-openstack-service>/<object>?<params>. But one hardly ever uses the raw API of OpenStack. To be a proficient consumer of OpenStack API, you must become a CLI or HEAT user.Getting our feet wet
To get started with the OpenStack CLI, API, and Heat, let's go through some simple examples. This is by no means comprehensive, but should serve as a starting point on your journey down the path of programming your OpenStack tools. The examples are based on a Liberty release devstack installation.Using OpenStack Command Line Interface
OpenStack has its legacy CLI, but I'm going to use the new 'openstack' client or OSC. Let's start with:openstack --os-auth-url http://127.0.0.1:5000/v3 --os-username admin --os-password nova --os-user-domain-name Default --os-project-name admin --os-project-domain-name Default configuration showThe --os-* optional parameters are required, but can be provided through other mechanisms, such as environment variables. My favorite way is to provide them via clouds.yaml. So create a file in ~/.config/openstack/clouds.yaml with the following content:
~/.config/openstack/clouds.yaml ---------------------------------------------------- Clouds:Let's boot an instance for devstack3 cloud - user demo, project demo:
devstack3:
auth:
auth_url: http://10.20.0.8:5000/v3
password: nova
project_name: demo
username: demo
domain_name: Default
identity_api_version: '3'
region_name: RegionOne
devstack-admin3:
auth:
auth_url: http://10.20.0.8:5000/v3
password: nova
project_name: admin
username: admin
domain_name: Default
identity_api_version: '3'
region_name: RegionOne
openstack --os-cloud devstack3 server create --image cirros-0.3.4-x86_64-uec --flavor 1 test-vm
+--------------------------------------+--------------------------------------------------------------+
| Field | Value |
+--------------------------------------+--------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | sBGu9QgmFmwL |
| config_drive | |
| created | 2016-04-23T15:35:21Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | a8a5d76d-e3fe-4e66-a0a5-9844ce96be80 |
| image | cirros-0.3.4-x86_64-uec (4a206785-3d8b-4781-89e2-7be3de4ef84b) |
| key_name | None |
| name | test-vm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | 2a7bcd9bedfb4102b940752d04c15a50 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2016-04-23T15:35:22Z |
| user_id | 64831d6f859c4368aa4d3d8be11bafe2 |
+--------------------------------------+--------------------------------------------------------------+
Using the OpenStack raw HTTP API
Doing the same thing using the raw API requires a utility like curl, which allows HTTP requests to be sent to any web service. To do the above we must refer to the OpenStack API documentation and formulate a call to keystone to get an authentication token, and then a call to nova to create a new server. When calling keystone we must pass the -i option to curl because the authentication code is returned in the HTTP header:curl -si -X POST -H "Content-Type: application/json" -d '{ "auth": { "identity": { "methods": [ "password" ], "password": { "user": { "domain": { "name": "Default" }, "name": "demo", "password": "nova" } } }, "scope": { "project": { "domain": { "name": "Default" }, "name": "demo" } } } }' http://10.20.0.8:5000/v3/auth/tokens?nocatalogNext, we need to extract the authentication token id from the previous response and pass it to the API endpoint of nova - which can be queried by removing the "?nocatalog" from the previous curl call. Nova API endpoint for the demo project is http://10.20.0.8:8774/v2/2a7bcd9bedfb4102b940752d04c15a50. To launch a new instance all we have to do is:
HTTP/1.1 201 Created
Date: Sat, 23 Apr 2016 16:24:52 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Subject-Token: b0cb3edfa73542069d81538f5ef9609c
Vary: X-Auth-Token
x-openstack-request-id: req-2a9f2d1c-b924-43df-be7e-79286248b480
Content-Length: 555
Content-Type: application/json
{"token": {"methods": ["password"], "roles": [{"id": "96b87338d6eb41e9b0236cda11a25d05", "name": "anotherrole"}, {"id": "b5f7a3fb454441bc9caad77df7b799cd", "name": "Member"}], "expires_at": "2016-04-23T17:24:52.385761Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "2a7bcd9bedfb4102b940752d04c15a50", "name": "demo"}, "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "64831d6f859c4368aa4d3d8be11bafe2", "name": "demo"}, "audit_ids": ["dZK3w3h8T2axRX POST http://10.20.0.8:8774/v2/2a7bcd9bedfb4102b940752d04c15a50/servers -H "X-Auth-Token: b0cb3edfa73542069d81538f5ef9609c" -d '{"server": {"min_count": 1, "flavorRef": "1", "name": "foobar", "imageRef": "f56098bb-2002-435a-bb93-48ea8e46d05d", "max_count": 1}}'
curl -X POST http://10.20.0.8:8774/v2/2a7bcd9bedfb4102b940752d04c15a50/servers -H "X-Auth-Token: b0cb3edfa73542069d81538f5ef9609c" -H "Content-Type: application/json" -d '{"server": {"min_count": 1, "flavorRef": "1", "name": "foobar", "imageRef": "4a206785-3d8b-4781-89e2-7be3de4ef84b", "max_count": 1}}'The authentication token is good for an hour, so doing additional curl commands doesn't require a call to keystone until the authentication token expires. For example, we can list our servers:curl -g -X GET http://10.20.0.8:8774/v2.1/2a7bcd9bedfb4102vers/detail -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token:b0cb3edfa73542069d81538f5ef9609c" | python -m json.toolDelete the second server by copying the id in the url below:
{
"servers": [
{
"OS-DCF:diskConfig": "MANUAL",
"OS-EXT-AZ:availability_zone": "nova",
"OS-EXT-STS:power_state": 1,
"OS-EXT-STS:task_state": null,
"OS-EXT-STS:vm_state": "active",
"OS-SRV-USG:launched_at": "2016-04-23T16:28:28.000000",
"OS-SRV-USG:terminated_at": null,
"accessIPv4": "",
"accessIPv6": "",
"addresses": {
"private": [
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e5:1a:06",
"OS-EXT-IPS:type": "fixed",
"addr": "10.0.0.5",
"version": 4
},
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:e5:1a:06",
"OS-EXT-IPS:type": "fixed",
"addr": "fd72:af8a:f919:0:f816:3eff:fee5:1a06",
"version": 6
}
]
},
"config_drive": "True",
"created": "2016-04-23T16:28:19Z",
"flavor": {
"id": "1",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/flavors/1",
"rel": "bookmark"
}
]
},
"hostId": "f6e30b1bd0d3f4f342df6c093140cdcc912ba228bc2f6a41e396d2b1",
"id": "345a468a-a64e-4340-a06a-2cfad118f6f1",
"image": {
"id": "4a206785-3d8b-4781-89e2-7be3de4ef84b",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/images/4a206785-3d8b-4781-89e2-7be3de4ef84b",
"rel": "bookmark"
}
]
},
"key_name": null,
"links": [
{
"href": "http://10.20.0.8:8774/v2.1/2a7bcd9bedfb4102b940752d04c15a50/servers/345a468a-a64e-4340-a06a-2cfad118f6f1",
"rel": "self"
},
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/servers/345a468a-a64e-4340-a06a-2cfad118f6f1",
"rel": "bookmark"
}
],
"metadata": {},
"name": "foobar",
"os-extended-volumes:volumes_attached": [],
"progress": 0,
"security_groups": [
{
"name": "default"
}
],
"status": "ACTIVE",
"tenant_id": "2a7bcd9bedfb4102b940752d04c15a50",
"updated": "2016-04-23T16:28:29Z",
"user_id": "64831d6f859c4368aa4d3d8be11bafe2"
},
{
"OS-DCF:diskConfig": "MANUAL",
"OS-EXT-AZ:availability_zone": "nova",
"OS-EXT-STS:power_state": 1,
"OS-EXT-STS:task_state": null,
"OS-EXT-STS:vm_state": "active",
"OS-SRV-USG:launched_at": "2016-04-23T15:35:30.000000",
"OS-SRV-USG:terminated_at": null,
"accessIPv4": "",
"accessIPv6": "",
"addresses": {
"private": [
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:53:8a:31",
"OS-EXT-IPS:type": "fixed",
"addr": "10.0.0.4",
"version": 4
},
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:53:8a:31",
"OS-EXT-IPS:type": "fixed",
"addr": "fd72:af8a:f919:0:f816:3eff:fe53:8a31",
"version": 6
}
]
},
"config_drive": "True",
"created": "2016-04-23T15:35:21Z",
"flavor": {
"id": "1",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/flavors/1",
"rel": "bookmark"
}
]
},
"hostId": "f6e30b1bd0d3f4f342df6c093140cdcc912ba228bc2f6a41e396d2b1",
"id": "a8a5d76d-e3fe-4e66-a0a5-9844ce96be80",
"image": {
"id": "4a206785-3d8b-4781-89e2-7be3de4ef84b",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/images/4a206785-3d8b-4781-89e2-7be3de4ef84b",
"rel": "bookmark"
}
]
},
"key_name": null,
"links": [
{
"href": "http://10.20.0.8:8774/v2.1/2a7bcd9bedfb4102b940752d04c15a50/servers/a8a5d76d-e3fe-4e66-a0a5-9844ce96be80",
"rel": "self"
},
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/servers/a8a5d76d-e3fe-4e66-a0a5-9844ce96be80",
"rel": "bookmark"
}
],
"metadata": {},
"name": "test-vm",
"os-extended-volumes:volumes_attached": [],
"progress": 0,
"security_groups": [
{
"name": "default"
}
],
"status": "ACTIVE",
"tenant_id": "2a7bcd9bedfb4102b940752d04c15a50",
"updated": "2016-04-23T15:35:31Z",
"user_id": "64831d6f859c4368aa4d3d8be11bafe2"
}
]
}
curl -gi -X DELETE -H "Accept: application/json" -H "X-Auth-Token: f74e903fbc2b4c68832291eb9bc66196" http://10.20.0.8:8774/v2.1/2a7bcd9bedfb4102b940752d04c15a50/servers/a8a5d76d-e3fe-4e66-a0a5-9844ce96be80
Using Python Client Libraries of OpenStack
Like everything else in OpenStack, there are multiple client libraries, and different ways of doing things. I like the os_client_config for authentication, but prefer the easy to use interface of novaclient for creating instances, even though os_client_config sessions can be used to do the same thing in a generic fashion, similar to how curl calls are used. In either case, I need to pick a cloud configuration using os_client_config. So start your python shell and:import os_client_configAt this point we know that we can find the ‘devstack’ cloud in our clouds.yaml. There are multiple ways to proceed here, for example we can ask os_client_config for a ‘compute’ session and use the ‘put’ method to create our server. But I’m going to ask for an ‘identity’ session and then extract the session object in it, and pass that to a novaclient object:
>>> cc=os_client_config.OpenStackConfig().get_one_cloud('devstack3')
>>> type(cc)
<class 'os_client_config.cloud_config.CloudConfig'>
>>> ks=os_client_config.make_client('identity', cloud='devstack3')Now, let’s use novaclient:
>>> type(ks)
<class 'keystoneclient.v3.client.Client'>
>>> type(ks.session)
<class 'keystoneauth1.session.Session'>
>>> from novaclient import clientLet’s list our servers to see if our instance was created successfully:
>>> nova = client.Client(‘2.1’, session=ks.session)
>>> type(nova)
<class 'novaclient.v2.client.Client'>
>>> nova.servers.list()
[]
>>> nova.flavors.list()
[<Flavor: m1.tiny>, <Flavor: m1.small>, <Flavor: m1.medium>, <Flavor: m1.large>, <Flavor: m1.xlarge>]
>>> nova.images.list()
[<Image: cirros-0.3.4-x86_64-uec>, <Image: cirros-0.3.4-x86_64-uec-ramdisk>, <Image: cirros-0.3.4-x86_64-uec-kernel>]
>>> f=nova.flavors.list()[0]
>>> m=nova.images.find(name='cirros-0.3.4-x86_64-uec')
>>> nova.servers.create('client-lib', m, f)
<Server: client-lib>
>>> s= nova.servers.list()[0]Now delete the server:
>>> import json.tool
>>> print json.dumps(s._info, sort_keys=True, indent=4, separators=(',', ': '))
{
"OS-DCF:diskConfig": "MANUAL",
"OS-EXT-AZ:availability_zone": "nova",
"OS-EXT-STS:power_state": 1,
"OS-EXT-STS:task_state": null,
"OS-EXT-STS:vm_state": "active",
"OS-SRV-USG:launched_at": "2016-04-23T20:19:34.000000",
"OS-SRV-USG:terminated_at": null,
"accessIPv4": "",
"accessIPv6": "",
"addresses": {
"private": [
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:65:68:c7",
"OS-EXT-IPS:type": "fixed",
"addr": "10.0.0.6",
"version": 4
},
{
"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:65:68:c7",
"OS-EXT-IPS:type": "fixed",
"addr": "fd72:af8a:f919:0:f816:3eff:fe65:68c7",
"version": 6
}
]
},
"config_drive": "True",
"created": "2016-04-23T20:19:24Z",
"flavor": {
"id": "1",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/flavors/1",
"rel": "bookmark"
}
]
},
"hostId": "f6e30b1bd0d3f4f342df6c093140cdcc912ba228bc2f6a41e396d2b1",
"id": "3663622c-ce4e-477e-8b83-b0930a86869d",
"image": {
"id": "4a206785-3d8b-4781-89e2-7be3de4ef84b",
"links": [
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/images/4a206785-3d8b-4781-89e2-7be3de4ef84b",
"rel": "bookmark"
}
]
},
"key_name": null,
"links": [
{
"href": "http://10.20.0.8:8774/v2.1/2a7bcd9bedfb4102b940752d04c15a50/servers/3663622c-ce4e-477e-8b83-b0930a86869d",
"rel": "self"
},
{
"href": "http://10.20.0.8:8774/2a7bcd9bedfb4102b940752d04c15a50/servers/3663622c-ce4e-477e-8b83-b0930a86869d",
"rel": "bookmark"
}
],
"metadata": {},
"name": "client-lib",
"os-extended-volumes:volumes_attached": [],
"progress": 0,
"security_groups": [
{
"name": "default"
}
],
"status": "ACTIVE",
"tenant_id": "2a7bcd9bedfb4102b940752d04c15a50",
"updated": "2016-04-23T20:19:34Z",
"user_id": "64831d6f859c4368aa4d3d8be11bafe2"
}
>>> nova.servers.delete(s)By now you should have a good understanding of what a RESTful Web Service is and how to use the OpenStack API to develop tools via Heat and OpenStack CLI. If you have additional questions, feel free to post a comment below. For all questions regarding OpenStack, take a look at the OpenStack courses that Mirantis Training offers.
()
