Swift Deployment with OpenStack Keystone authentication
Jim Phillips - November 30, 2015
Today I would like to share with you my story about installation OpenStack service Swift with Keystone authentication. Swift is a highly available, distributed, eventually consistent object/blob store. More info at http://docs.openstack.org/developer/swift/
My mission was to test the features of Swift with our open solution tcp cloud Mk.20. Then I decided to write this blog post because of little bit tricky steps in deployment especially during Keystone integration. I spent some time on that and realized that community solves similar issues.
Swift enables to run several modes of authentication:
TempAuth
Keystone Auth - the most important mode for OpenStack integration. Lots of applications have issues with Swift and Keystone. Therefore this is critical.
Extending Auth - Creating Your Own Auth Server and Middleware
Lab Infrastructure
For testing I used OpenStack Heat Template built on tcp cloud Mk.20. It is the first open source tcp cloud release built on Kilo. Our release supports KVM virtualization with SDN OpenContrail 2.2, runs on Ubuntu 14.04 operating system and other open source projects. Automation of deployment and installation is done by configuration management tool Salt. Detailed information you can find at http://www.opentcpcloud.org/en/documentation/openstack-lab/.
I launched 5 other instances VMs
2x Proxy Nodes with Neutron LBaaS (OpenContrail HAProxy instances)
3x Storage Nodes to ensure HA.
Deploy Procedures
I decided to describe complete procedure how I tested and identified issues/resolutions. Finally, there were verified and tested more than 4 guides including official documentation.
1) OpenStack Cloud Computing Cookbook
When I deployed OpenStack through Heat, I continued with configuration according to OpenStack Cloud Computing Cookbook (http://openstackcookbook.com/). Despite adherence to the instructions, Swift didn’t work. This book describes almost similar steps like official documentation extended to administration tips.
All problems with Keystone integration are related to Swift Proxy proxy-server.conf. I had following config
I hit following error message with swift stat command:
root@ctl01:~# swift -V 3 stat Traceback (most recent call last): File "/usr/bin/swift", line 10, in sys.exit(main()) File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 1287, in main globals()['st_%s' % args[0]](parser, argv[1:], output) File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 492, in st_stat stat_result = swift.stat() File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 443, in stat raise SwiftError('Account not found', exc=err) swiftclient.service.SwiftError: 'Account not found'
After that I tested it with –debug parameter.
root@ctl01:~# swift --debug -V 3 stat DEBUG:keystoneclient.auth.identity.v3:Making authentication request to http://172.10.20.254:35357/v3/auth/tokens INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.254 DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"POST /v3/auth/tokens HTTP/1.1" 201 6728 DEBUG:iso8601.iso8601:Parsed 2015-11-30T06:07:29.066102Z into {'tz_sign': None, 'second_fraction': u'066102', 'hour': u'06', 'daydash': u'30', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'29', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'11', 'day': None, 'minute': u'07'} with default timezone DEBUG:iso8601.iso8601:Got u'2015' for 'year' with default None DEBUG:iso8601.iso8601:Got u'11' for 'monthdash' with default 1 DEBUG:iso8601.iso8601:Got 11 for 'month' with default 11 DEBUG:iso8601.iso8601:Got u'30' for 'daydash' with default 1 DEBUG:iso8601.iso8601:Got 30 for 'day' with default 30 DEBUG:iso8601.iso8601:Got u'06' for 'hour' with default None DEBUG:iso8601.iso8601:Got u'07' for 'minute' with default None DEBUG:iso8601.iso8601:Got u'29' for 'second' with default None INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.114 DEBUG:urllib3.connectionpool:Setting read timeout to
2) OpenStack documentation
Then I tried to search another information and tips, but without any success. Therefore I decided to follow the official OpenStack documentation for Kilo release. But even after the configuration of proxy and storage nodes according to official OpenStack documentation, Swift didn’t work. Still the same output: “Accout not found”.
root@ctl01:~# swift -V 3 stat Traceback (most recent call last): File "/usr/bin/swift", line 10, in sys.exit(main()) File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 1287, in main globals()['st_%s' % args[0]](parser, argv[1:], output) File "/usr/lib/python2.7/dist-packages/swiftclient/shell.py", line 492, in st_stat stat_result = swift.stat() File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 443, in stat raise SwiftError('Account not found', exc=err) swiftclient.service.SwiftError: 'Account not found'
3) Vendors configuration
At this moment I was a little bit desperate. The next step was combining configuration from vendors like e.g. Mirantis or SwifStack. I used the proxy-server.conf file from OpenStack Documentation and tried to set different variable, but it did help neither.
4) Launchpad
I did not give up and I kept looking for some solution to this problem. I managed to find on Launchpad one proposed solution.
The most important step was determining keystoneauth in [pipeline:main] section.
After this step I restarted proxy-server service and Swift worked. But during the verifying functionality of Swift, I came to the problem with Keystone. There was no authentication and without authentication with Keystone I was able to get any file from any tenant.
In this way is Swift almost unusable, because it must be multi-tenant environment with correct user rights. User cannot access files of anyone else.
Error Output
I tested swift with swift stat command and it works for first time.
root@ctl01:~# swift -V 3 stat Account: AUTH_910432f8e6f64ef0bed54271de5e7e81 Containers: 1 Objects: 1 Bytes: 13 Containers in policy "policy-0": 1 Objects in policy "policy-0": 1 Bytes in policy "policy-0": 13 X-Timestamp: 1448019739.62868 X-Trans-Id: tx7541eb08c30049bcadec2-005658814c Content-Type: text/plain; charset=utf-8 Accept-Ranges: bytes
Then I tested it with other swift commands.
root@ctl01:~# swift -V 3 list root@ctl01:~# swift -V 3 upload test keystonerc keystonerc root@ctl01:~# swift -V 3 list test container-test
And then I tested it with –debug parameter.
root@ctl01:~# swift --debug -V 3 list DEBUG:keystoneclient.auth.identity.v3:Making authentication request to http://172.10.20.254:35357/v3/auth/tokens INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.254 DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"POST /v3/auth/tokens HTTP/1.1" 201 6728 DEBUG:iso8601.iso8601:Parsed 2015-11-28T03:02:09.114035Z into {'tz_sign': None, 'second_fraction': u'114035', 'hour': u'03', 'daydash': u'28', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'09', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'11', 'day': None, 'minute': u'02'} with default timezone DEBUG:iso8601.iso8601:Got u'2015' for 'year' with default None DEBUG:iso8601.iso8601:Got u'11' for 'monthdash' with default 1 DEBUG:iso8601.iso8601:Got 11 for 'month' with default 11 DEBUG:iso8601.iso8601:Got u'28' for 'daydash' with default 1 DEBUG:iso8601.iso8601:Got 28 for 'day' with default 28 DEBUG:iso8601.iso8601:Got u'03' for 'hour' with default None DEBUG:iso8601.iso8601:Got u'02' for 'minute' with default None DEBUG:iso8601.iso8601:Got u'09' for 'second' with default None INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.10.20.114 DEBUG:urllib3.connectionpool:Setting read timeout to DEBUG:urllib3.connectionpool:"GET /v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json HTTP/1.1" 200 53 DEBUG:swiftclient:REQ: curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json -X GET -H "X-Auth-Token: 0a39abe8670a4f88bfa218a281d0af4a" DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: [('content-length', '53'), ('x-account-object-count', '1'), ('x-account-storage-policy-policy-0-bytes-used', '13'), ('x-account-storage-policy-policy-0-container-count', '1'), ('x-timestamp', '1448019739.62868'), ('x-account-storage-policy-policy-0-object-count', '1'), ('x-trans-id', 'tx4720c01cc47c43b785d1e-0056587071'), ('date', 'Fri, 27 Nov 2015 15:02:09 GMT'), ('x-account-bytes-used', '13'), ('x-account-container-count', '1'), ('content-type', 'application/json; charset=utf-8'), ('accept-ranges', 'bytes')] DEBUG:swiftclient:RESP BODY: [{"count": 1, "bytes": 13, "name": "container-test"}] DEBUG:urllib3.connectionpool:Setting read timeout to DEBUG:urllib3.connectionpool:"GET /v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json&marker=container-test HTTP/1.1" 200 2 DEBUG:swiftclient:REQ: curl -i http://172.10.20.114:8080/v1/AUTH_910432f8e6f64ef0bed54271de5e7e81?format=json&marker=container-test -X GET -H "X-Auth-Token: 0a39abe8670a4f88bfa218a281d0af4a" DEBUG:swiftclient:RESP STATUS: 200 OK DEBUG:swiftclient:RESP HEADERS: [('content-length', '2'), ('x-account-object-count', '1'), ('x-account-storage-policy-policy-0-bytes-used', '13'), ('x-account-storage-policy-policy-0-container-count', '1'), ('x-timestamp', '1448019739.62868'), ('x-account-storage-policy-policy-0-object-count', '1'), ('x-trans-id', 'tx303c11ad532e4452873b6-0056587071'), ('date', 'Fri, 27 Nov 2015 15:02:09 GMT'), ('x-account-bytes-used', '13'), ('x-account-container-count', '1'), ('content-type', 'application/json; charset=utf-8'), ('accept-ranges', 'bytes')] DEBUG:swiftclient:RESP BODY: [] container-test X-Account-Storage-Policy-Policy-0-Object-Count: 1 X-Account-Bytes-Used: 13 X-Account-Container-Count: 1 Content-Type: application/json; charset=utf-8 Accept-Ranges: bytes X-Trans-Id: tx351ca253b43e4ae49a65b-005658707d Date: Fri, 27 Nov 2015 15:02:21 GMT
My last idea was deploy DevStack with Swift service and compare their configuration files with my existing one. It was good idea. Their proxy-server.conf file was different in [pipeline:main] section and in order of filters. So I adjust my existing proxy-server.conf according to proxy-server.conf file in DevStack. After that I restarted proxy-server service and tested Swift. Swift worked with Keystone authentication!
I tried both procedures of installation - OpenStack Cook Book and OpenStack Documentation. Actually both of them was working. The only one problem lies in proxy-server.conf file. So I recommend to follow official OpenStack Documentation and adjust only proxy-server.conf as is shown above. I will try to contribute official documentation for next releases.
Now is swift working and finally is also secure. User without without relevant approach is unable get files from any tenants.
Alena Holanova & tcp cloud team
Cloud Platform Engineer Junior
Choose your cloud native journey.
Whatever your role, we’re here to help with open source tools and world-class support.