Swift Deployment with OpenStack Keystone authentication
Today I would like to share with you my story about installation OpenStack service Swift with Keystone authentication. Swift is a highly available, distributed, eventually consistent object/blob store. More info at http://docs.openstack.org/developer/swift/
My mission was to test the features of Swift with our open solution tcp cloud Mk.20. Then I decided to write this blog post because of little bit tricky steps in deployment especially during Keystone integration. I spent some time on that and realized that community solves similar issues.
Swift enables to run several modes of authentication:
- TempAuth
- Keystone Auth - the most important mode for OpenStack integration. Lots of applications have issues with Swift and Keystone. Therefore this is critical.
- Extending Auth - Creating Your Own Auth Server and Middleware
Lab Infrastructure
For testing I used OpenStack Heat Template built on tcp cloud Mk.20. It is the first open source tcp cloud release built on Kilo. Our release supports KVM virtualization with SDN OpenContrail 2.2, runs on Ubuntu 14.04 operating system and other open source projects. Automation of deployment and installation is done by configuration management tool Salt. Detailed information you can find at http://www.opentcpcloud.org/en/documentation/openstack-lab/.
I launched 5 other instances VMs
- 2x Proxy Nodes with Neutron LBaaS (OpenContrail HAProxy instances)
- 3x Storage Nodes to ensure HA.
Deploy Procedures
I decided to describe complete procedure how I tested and identified issues/resolutions. Finally, there were verified and tested more than 4 guides including official documentation.
1) OpenStack Cloud Computing Cookbook
When I deployed OpenStack through Heat, I continued with configuration according to OpenStack Cloud Computing Cookbook (http://openstackcookbook.com/). Despite adherence to the instructions, Swift didn’t work. This book describes almost similar steps like official documentation extended to administration tips.
All problems with Keystone integration are related to Swift Proxy proxy-server.conf. I had following config
https://github.com/sfinga/swift-testing/blob/master/proxy-server-CookBook.conf
Error Output
I hit following error message with swift stat command:
root@ctl01:~# swift -V 3 stat
Traceback (most recent call last):
File "/usr/bin/swift", line 10, in
sys.exit(main())
File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 1287, in main
globals()['st_%s' % args[0]](parser, argv[1:], output)
File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 492, in st_stat
stat_result = swift.stat()
File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 443, in stat
raise SwiftError('Account not found', exc=err)
swiftclient.service.SwiftError: 'Account not found'
After that I tested it with –debug parameter.
root@ctl01:~# swift --debug -V 3 stat
DEBUG:keystoneclient.auth.identity.v3:Making authentication request to http://172.10.20.254:35357/v3/auth/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.254
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"POST /v3/auth/tokens HTTP/1.1" 201 6728
DEBUG:iso8601.iso8601:Parsed 2015-11-30T06:07:29.066102Z into {'tz_sign': None, 'second_fraction': u'066102', 'hour': u'06', 'daydash': u'30', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'29', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'11', 'day': None, 'minute': u'07'} with default timezone
DEBUG:iso8601.iso8601:Got u'2015' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'11' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 11 for 'month' with default 11
DEBUG:iso8601.iso8601:Got u'30' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 30 for 'day' with default 30
DEBUG:iso8601.iso8601:Got u'06' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'07' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'29' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.114
DEBUG:urllib3.connectionpool:Setting read timeout to 2) OpenStack documentation
Then I tried to search another information and tips, but without any success. Therefore I decided to follow the official OpenStack documentation for Kilo release. But even after the configuration of proxy and storage nodes according to official OpenStack documentation, Swift didn’t work.
Still the same output: “Accout not found”.
Following proxy-server.conf:
https://github.com/sfinga/swift-testing/blob/master/proxy-server-OSDocumentation.conf
Error Output
Again I got the same output as above.
root@ctl01:~# swift -V 3 stat
Traceback (most recent call last):
File "/usr/bin/swift", line 10, in
sys.exit(main())
File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 1287, in main
globals()['st_%s' % args[0]](parser, argv[1:], output)
File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 492, in st_stat
stat_result = swift.stat()
File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 443, in stat
raise SwiftError('Account not found', exc=err)
swiftclient.service.SwiftError: 'Account not found'
3) Vendors configuration
At this moment I was a little bit desperate. The next step was combining configuration from vendors like e.g. Mirantis or SwifStack.
I used the proxy-server.conf file from OpenStack Documentation and tried to set different variable, but it did help neither.
4) Launchpad
I did not give up and I kept looking for some solution to this problem. I managed to find on Launchpad one proposed solution.
Changing the [pipeline:main] to
pipeline = catch_errors healthcheck cache authtoken proxy-logging proxy-server
The most important step was determining keystoneauth in [pipeline:main] section.
After this step I restarted proxy-server service and Swift worked. But during the verifying functionality of Swift, I came to the problem with Keystone. There was no authentication and without authentication with Keystone I was able to get any file from any tenant.
In this way is Swift almost unusable, because it must be multi-tenant environment with correct user rights. User cannot access files of anyone else.
Error Output
I tested swift with swift stat command and it works for first time.
root@ctl01:~# swift -V 3 stat
Account: AUTH_910432f8e6f64ef0bed54271de5e7e81
Containers: 1
Objects: 1
Bytes: 13
Containers in policy "policy-0": 1
Objects in policy "policy-0": 1
Bytes in policy "policy-0": 13
X-Timestamp: 1448019739.62868
X-Trans-Id: tx7541eb08c30049bcadec2-005658814c
Content-Type: text/plain; charset=utf-8
Accept-Ranges: bytes
Then I tested it with other swift commands.
root@ctl01:~# swift -V 3 list
root@ctl01:~# swift -V 3 upload test keystonerc
keystonerc
root@ctl01:~# swift -V 3 list
test
container-test
And then I tested it with –debug parameter.
root@ctl01:~# swift --debug -V 3 list
DEBUG:keystoneclient.auth.identity.v3:Making authentication request to http://172.10.20.254:35357/v3/auth/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.254
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"POST /v3/auth/tokens HTTP/1.1" 201 6728
DEBUG:iso8601.iso8601:Parsed 2015-11-28T03:02:09.114035Z into {'tz_sign': None, 'second_fraction': u'114035', 'hour': u'03', 'daydash': u'28', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'09', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'11', 'day': None, 'minute': u'02'} with default timezone
DEBUG:iso8601.iso8601:Got u'2015' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'11' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 11 for 'month' with default 11
DEBUG:iso8601.iso8601:Got u'28' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 28 for 'day' with default 28
DEBUG:iso8601.iso8601:Got u'03' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'02' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'09' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.114
DEBUG:urllib3.connectionpool:Setting read timeout to If I test curl from swift –debug -V 3 list, I got this output:
root@ctl01:~# curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json -X GET -H "X-Auth-Token: 0a39abe8670a4f88bfa218a281d0af4a"
HTTP/1.1 200 OK
Content-Length: 53
X-Account-Object-Count: 1
X-Account-Storage-Policy-Policy-0-Bytes-Used: 13
X-Account-Storage-Policy-Policy-0-Container-Count: 1
X-Timestamp: 1448019739.62868
X-Account-Storage-Policy-Policy-0-Object-Count: 1
X-Account-Bytes-Used: 13
X-Account-Container-Count: 1
Content-Type: application/json; charset=utf-8
Accept-Ranges: bytes
X-Trans-Id: txd306df98ec2a44089a3ea-0056587138
Date: Fri, 27 Nov 2015 15:05:28 GMT
Then I tested it with token of different user and get the same output, so I am able to get all of it’s files.
root@ctl01:~# curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json -X GET -H "X-Auth-Token: 2608e9468f254b45be47eda3391c29c7"
HTTP/1.1 200 OK
Content-Length: 53
X-Account-Object-Count: 1
X-Account-Storage-Policy-Policy-0-Bytes-Used: 13
X-Account-Storage-Policy-Policy-0-Container-Count: 1
X-Timestamp: 1448019739.62868
X-Account-Storage-Policy-Policy-0-Object-Count: 1
X-Account-Bytes-Used: 13
X-Account-Container-Count: 1
Content-Type: application/json; charset=utf-8
Accept-Ranges: bytes
X-Trans-Id: txcc74fbe00dd04ea78ca97-00565874bc
Date: Fri, 27 Nov 2015 15:20:28 GMT
[{"count": 1, "bytes": 13, "name": "container-test"}]
5) DevStack
My last idea was deploy DevStack with Swift service and compare their configuration files with my existing one.
It was good idea. Their proxy-server.conf file was different in [pipeline:main] section and in order of filters. So I adjust my existing proxy-server.conf according to proxy-server.conf file in DevStack. After that I restarted proxy-server service and tested Swift.
Swift worked with Keystone authentication!
Correct form of the proxy-server.conf:
https://github.com/sfinga/swift-testing/blob/master/proxy-server-devstack.conf
Correct Output
I tested Swift with swift list command and it worked.
root@ctl01:~# source keystonercv3
root@ctl01:~# swift --debug -V 3 list
DEBUG:keystoneclient.auth.identity.v3:Making authentication request to http://172.10.20.254:35357/v3/auth/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.254
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"POST /v3/auth/tokens HTTP/1.1" 201 6728
DEBUG:iso8601.iso8601:Parsed 2015-11-28T01:20:23.233637Z into {'tz_sign': None, 'second_fraction': u'233637', 'hour': u'01', 'daydash': u'28', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'23', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'11', 'day': None, 'minute': u'20'} with default timezone
DEBUG:iso8601.iso8601:Got u'2015' for 'year' with default None
DEBUG:iso8601.iso8601:Got u'11' for 'monthdash' with default 1
DEBUG:iso8601.iso8601:Got 11 for 'month' with default 11
DEBUG:iso8601.iso8601:Got u'28' for 'daydash' with default 1
DEBUG:iso8601.iso8601:Got 28 for 'day' with default 28
DEBUG:iso8601.iso8601:Got u'01' for 'hour' with default None
DEBUG:iso8601.iso8601:Got u'20' for 'minute' with default None
DEBUG:iso8601.iso8601:Got u'23' for 'second' with default None
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.114
DEBUG:urllib3.connectionpool:Setting read timeout to Again I tried curl command:
root@ctl01:~# curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json -X GET -H "X-Auth-Token: d1a575a7600740578a670b0c3cca8c7c"
HTTP/1.1 200 OK
Content-Length: 53
X-Account-Object-Count: 1
X-Account-Storage-Policy-Policy-0-Bytes-Used: 13
X-Account-Storage-Policy-Policy-0-Container-Count: 1
X-Timestamp: 1448019739.62868
X-Account-Storage-Policy-Policy-0-Object-Count: 1
X-Account-Bytes-Used: 13
X-Account-Container-Count: 1
Content-Type: application/json; charset=utf-8
Accept-Ranges: bytes
X-Trans-Id: tx28713ec2734545cf9d37d-00565858de
Date: Fri, 27 Nov 2015 13:21:34 GMT
[{"count": 1, "bytes": 13, "name": "container-test"}]
And then I tested it with token of different user and access was denied.
root@ctl01:~# curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json -X GET -H "X-Auth-Token: f46baff150b2496b8b006ef79a6ded14"
HTTP/1.1 403 Forbidden
Content-Length: 73
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx0bb1063fc588458eb74da-0056585904
Date: Fri, 27 Nov 2015 13:22:12 GMT
Forbidden
Access was denied to this resource.
Conclusion
I tried both procedures of installation - OpenStack Cook Book and OpenStack Documentation. Actually both of them was working. The only one problem lies in proxy-server.conf file. So I recommend to follow official OpenStack Documentation and adjust only proxy-server.conf as is shown above. I will try to contribute official documentation for next releases.
Now is swift working and finally is also secure. User without without relevant approach is unable get files from any tenants.
Alena Holanova & tcp cloud team
Cloud Platform Engineer Junior
)
)
)
