From a19b2825992f23efc98bd6ec7428427fc6bb1f1b Mon Sep 17 00:00:00 2001 From: Monty Taylor Date: Wed, 18 Oct 2017 13:32:54 +0200 Subject: Add slides from OpenStack Days Nordic --- src/everything-you-need-to-know.rst | 1153 +++++++++++++++++++++++++++++++++++ 1 file changed, 1153 insertions(+) create mode 100644 src/everything-you-need-to-know.rst diff --git a/src/everything-you-need-to-know.rst b/src/everything-you-need-to-know.rst new file mode 100644 index 0000000..f478d55 --- /dev/null +++ b/src/everything-you-need-to-know.rst @@ -0,0 +1,1153 @@ +========================================================= +Everything you need to know about using the OpenStack API +========================================================= + +This document contains a presentation in `presentty`_ format. If you want to +walk through it like a presentation, install `presentty` and run: + +.. code:: bash + + presentty doc/source/user/multi-cloud-demo.rst + +The content is hopefully helpful even if it's not being narrated, so it's being +included in the `shade` docs. + +.. _presentty: https://pypi.python.org/pypi/presentty + +Everything you need to know about using the OpenStack API +========================================================= + +Who am I? +========= + +Monty Taylor + +* OpenStack Infra Core +* irc: mordred +* twitter: @e_monty + +This Talk is Free Software +========================== + +* http://git.inaugust.com/cgit/inaugust.com/tree/src/everything-you-need-to-know.rst + +What are we going to talk about? +================================ + +* Overview of some terminology +* Discussion of SDKs +* 'Simple' example +* Step by step through in much more detail + + * Auth + * The Catalog + * Service Types + * Version Discovery + * Actually doing things + * Microversions + +Expected Ranting +================ + +* Fixed vs. Floating IPs +* Build your own images +* Boot-time scripting with cloud-init + +Relevant Docs +============= + +* https://specs.openstack.org/openstack/api-wg/ +* https://service-types.openstack.org/ +* https://developer.openstack.org/api-ref/identity/ +* https://developer.openstack.org/api-ref/compute/ +* https://docs.openstack.org/os-client-config/latest/ +* https://docs.openstack.org/keystoneauth/latest/using-sessions.html + +Cloud Terminology +================= + +Let's define a few terms, so that we can use them with ease: + +* `cloud` - logically related collection of services +* `region` - completely independent subset of a given cloud +* `patron` - human who has an account +* `user` - account on a cloud +* `project` - logical collection of cloud resources +* `domain` - collection of users and projects + +Cloud Terminology Relationships +=============================== + +* A `cloud` has one or more `regions` +* A `patron` has one or more `users` +* A `patron` has one or more `projects` +* A `cloud` has one or more `domains` +* In a `cloud` with one `domain` it is named "default" +* Each `patron` may have their own `domain` +* Each `user` is in one `domain` +* Each `project` is in one `domain` +* A `user` has one or more `roles` on one or more `projects` + +HTTP Sessions +============= + +* HTTP interactions are authenticated via keystone +* Authenticating returns a `token` +* An authenticated HTTP Session is shared across a `region` + +Cloud Regions +============= + +A `cloud region` is the basic unit of REST interaction. + +* A `cloud` has a `service catalog` +* The `service catalog` is returned in the `token` +* The `service catalog` lists `endpoint` for each `service` in each `region` +* A `region` is completely autonomous + +Availability Zones +================== + +* If you have exposure to AWS - OpenStack Availability Zones are not what you + think they are. +* OpenStack Availability Zones are tagged groupings of hypervisor hosts, + storage nodes and network nodes. + +Users, Projects and Domains +=========================== + +In clouds with multiple domains, project and user names are +only unique within a region. + +* Names require `domain` information for uniqueness. IDs do not. +* Providing `domain` information when not needed is fine. +* `project_name` requires `project_domain_name` or `project_domain_id` +* `project_id` does not +* `username` requires `user_domain_name` or `user_domain_id` +* `user_id` does not + +Auth vs. Identity +================= + +* Both provided by the ``keystone`` service +* ``identity`` refers to CRUD operations on users, projects, etc. +* ``auth`` refers to a user getting a token and catalog + +Keystone v2 vs. Keystone v3 +=========================== + +* Two major versions of identity API +* v2 is deprecated/legacy +* ``project`` was called ``tenant`` in v2 +* ``domain`` did not exist in v2 + +Confused Yet? +============= + +Awesome + +SDKs +==== + +REST +==== + +* You can always do direct REST +* Auth and discovery are a bit of a pain +* Once you have those, it's all pretty easy + +python +====== + +First: + +* python-openstackclient ... CLI tool ``openstack`` + +Five main libraries to be aware of: + +* keystoneauth +* os-client-config +* shade +* python-openstacksdk +* python-*client + +python-*client +============== + +* Per-service client libraries +* Avoid them. They're not written for you. They make things harder. + +keystoneauth +============ + +* wrapper around python-requests that handles auth, catalog endpoint and + version discovery +* "REST" library + +os-client-config +================ + +* Library to handle client configuration +* Can create python REST clients for you +* Used under the covers by shade, python-openstacksdk + and python-openstackclient +* Handles clouds.yaml files, environment variables and command line arguments + +shade +===== + +* a task and end-user oriented Python library +* abstracts deployment differences +* designed for multi-cloud +* simple to use +* massive scale + + * optional advanced features to handle 20k servers a day + +* Initial logic/design extracted from nodepool +* Librified to re-use in Ansible + +python-openstacksdk +=================== + +* Object oriented interface to OpenStack REST APIs +* Just got adopted by the shade team +* Interface is going to break soon +* shade, python-openstacksdk and os-client-config are going to merge +* shade is safe to use - there will always be backwards compat for shade + +Multi-Cloud Shade Example +========================== + +.. code:: python + + import shade + + # Initialize and turn on debug logging + shade.simple_logging(debug=True) + + for cloud_name, region_name in [ + ('my-vexxhost', 'ca-ymq-1'), + ('my-citycloud', 'Buf1'), + ('my-internap', 'ams01')]: + # Initialize cloud + cloud = shade.openstack_cloud(cloud=cloud_name, region_name=region_name) + + # Upload an image to the cloud + image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]') + + # Find a flavor with at least 512M of RAM + flavor = cloud.get_flavor_by_ram(512) + + # Boot a server, wait for it to boot, and then do whatever is needed + # to get a public ip for it. + cloud.create_server( + 'my-server', image=image, flavor=flavor, wait=True, auto_ip=True) + +It's Not Possible To Cover Everything That Does +=============================================== + +Simpler Example +=============== + +It still may not be possible to cover it all. + +.. code-block:: python + + import shade + + # Initialize and turn on debug logging + shade.simple_logging(debug=True) + + # Initialize cloud + cloud = shade.openstack_cloud(cloud='vexxhost') + + # Upload an image to the cloud + image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]') + + # Find a flavor with at least 512M of RAM + flavor = cloud.get_flavor_by_ram(512) + + # Boot a server, wait for it to boot, and then do whatever is needed + # to get a public ip for it. + cloud.create_server( + 'my-server', image=image, flavor=flavor, wait=True, auto_ip=True) + +Get Some Stuff Installed +======================== + +.. code-block:: bash + + virtualenv venv + source venv/bin/activate + pip install shade + +Essential pieces of Auth Information +==================================== + +* auth_url +* password +* (username + (user_domain_name or user_domain_id)) or user_id) +* (project_name + (project_domain_name or project_domain_id)) or project_id + +Note: It's important to know if you have a name or id for your user and + project. + +I Lied +====== + +Essential pieces of Auth Information +==================================== + +* auth_url +* auth API version +* auth_type + +If nobody tells you an auth_type, assume ``password`` + +If nobody tells you an auth_url, look at + +https://docs.openstack.org/os-client-config/latest/user/vendor-support.html + +Exercise - Find Auth URL for known cloud +=========================================== + +The cloud is Vexxhost. + +Answer +====== + +https://auth.vexxhost.net + +Version Discovery +================= + +* Each OpenStack service has a 'Version Discovery' document at its unversioned + root URL. We call this the 'Unversioned Discovery Document' + + * Except for swift. (pending some work with discovery caching) + +* Each OpenStack service ALSO has a 'Version Discovery' document at the root + of each Versioned URL. We call this the 'Versioned Discovery Document' + +* Version Discovery documents should not require Authentication + + * If they do on your cloud, your cloud is broken, please file a bug + +Exercise #2 - Find Auth API Version from Auth URL +================================================= + +:: + + curl -i https://auth.vexxhost.net + +Auth Version Discovery Response +=============================== + +:: + + { + "versions": { + "values": [ + { + "status": "stable", + "updated": "2017-02-22T00:00:00Z", + "media-types": [ + { + "base": "application/json", + "type": "application/vnd.openstack.identity-v3+json" + } + ], + "id": "v3.8", + "links": [ + { + "href": "https://auth.vexxhost.net/v3/", + "rel": "self" + } + ] + }, + +Auth Version Discovery Response Part 2 +====================================== + +:: + { + "status": "deprecated", + "updated": "2016-08-04T00:00:00Z", + "media-types": [ + { + "base": "application/json", + "type": "application/vnd.openstack.identity-v2.0+json" + } + ], + "id": "v2.0", + "links": [ + { + "href": "https://auth.vexxhost.net/v2.0/", + "rel": "self" + }, + { + "href": "https://docs.openstack.org/", + "type": "text/html", + "rel": "describedby" + } + ] + } + ] + } + } + +Versioned Discovery Document +============================ + +Sometimes a cloud gives you a versioned auth_url. + +:: + + curl -i https://auth.vexxhost.net/v3 + +Auth Version Discovery Resonse +============================== + +:: + + { + "version": { + "status": "stable", + "updated": "2017-02-22T00:00:00Z", + "media-types": [ + { + "base": "application/json", + "type": "application/vnd.openstack.identity-v3+json" + } + ], + "id": "v3.8", + "links": [ + { + "href": "https://auth.vexxhost.net/v3/", + "rel": "self" + } + ] + } + } + +Finding a Better Version +======================== + +Sometimes a cloud gives you a v2 versioned auth_url. + +:: + + curl -i https://auth.vexxhost.net/v2.0 + +**ALWAYS** prefer v3 auth if it's available. + +Old Auth Version Discovery Resonse +================================== + +:: + + { + "version": { + "status": "deprecated", + "updated": "2016-08-04T00:00:00Z", + "media-types": [ + { + "base": "application/json", + "type": "application/vnd.openstack.identity-v2.0+json" + } + ], + "id": "v2.0", + "links": [ + { + "href": "https://auth.vexxhost.net/v2.0/", + "rel": "self" + }, + { + "href": "https://docs.openstack.org/", + "type": "text/html", + "rel": "describedby" + } + ] + } + } + +Status Deprecated +================= + +Strip the trailing ``v.*`` portion from the URL and try again. + +:: + + { + "version": { + "status": "deprecated", + +Strip the trailing ``v.*`` portion from the URL and try again. + +If ``v3`` is not available, complain to someone. Keystone v3 api is considered +stable as of February 20, 2013. + +Isn't this fun? +=============== + +To Domain or Not To Domain +========================== + +In ``v3``: + +* Project Names and User Names are only unique within a domain +* If you don't have domain info, assume ``domain_id`` is ``default`` + +In ``v2``: + +* No such thing as domains +* Projects are called "tenant" + +Essential pieces of Auth Information +==================================== + +* auth_url - https://auth.vexxhost.net +* auth version - ``3`` +* versioned auth endpoint - https://auth.vexxhost.net/v3 + +* username - osdn-user-1 +* user domain id - ``default`` +* project_name - osdn-project-1 +* project domain id = ``default`` +* password - XXXXXXXX + +Now we can authenticate!!! +========================== + +:: + + curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens -H "Content-Type: application/json" -H "Accept: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": {"user": {"domain": {"id": "default"}, "name": "osdn-user-1", "password": "XXXXXXXX"}}},"scope": {"project": {"domain": {"id": "default"}, "name": "osdn-project-1"}}}}' + +What, you can't read that? +========================== + +:: + + curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens \ + -H "Content-Type: application/json" \ + -H "Accept: application/json" -d '{ + "auth": { + "identity": { + "methods": ["password"], + "password": { + "user": { + "domain": {"id": "default"}, + "name": "osdn-user-1", + "password": "XXXXXXXX"}}}, + "scope": { + "project": { + "domain": {"id": "default"}, + "name": "osdn-project-1"}}}}' + +Authentication Response +======================= + +Readable version of payload: http://paste.openstack.org/raw/623947/ + +:: + + HTTP/1.1 201 Created + Date: Tue, 17 Oct 2017 23:16:05 GMT + Server: Apache/2.4.6 (CentOS) + X-Subject-Token: gAAAAABZ5o81tyDsyJMXl6hVcVOqHgzKQ2WmpvepCSr2NWYM1D97VoaBy-i5OLT8-BbkWBZJq_aqrt03eutp2eEpORI-V4CYpYcWBDkR5jLNfq2gD-TDB9lzvRo7rHKu-PWSLIxEF7whsFJ8bVGDPMN4yc_ZwgfWcR13GxAKutU9_rzR9Gvewjg + Vary: X-Auth-Token + x-openstack-request-id: req-29d8c4e6-1924-42e9-8efa-4f154e813ee6 + Content-Length: 11696 + Content-Type: application/json + + {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "fd6d2130b7784b82aa6b49fd5901f70b", "name": "Member"}], "expires_at": "2017-10-18T23:16:05.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "bbb1e479a9d04f34941470f37c7a0ecb", "name": "osdn-project-1"}, "catalog": [{"endpoints": [{"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "522041f846934310b928a6afab47241e"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "660a742372554b5581c1f4870d4d42cf"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c1c32be0de8441eea4306b8c22442ba3"}], "type": "key-manager", "id": "023de54c09654931856403e708b60a22", "name": "barbican"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "289f7dcf096743fb9a17924bd598e716"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "61b6d283ef4b419a8ad23fc53dea70df"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "e1e7c79ce7834ec0aa74308326e2f96c"}], "type": "container-infra", "id": "041eac2489bb495b8a900c40b93b2065", "name": "magnum"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "203fd4e466b44569aa9ab8c78ef55bad"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "da3d087e0c724338ba12c9a1168ef80c"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "f0a311cb5dbf4ae788670107a3433ac2"}], "type": "orchestration", "id": "0e852aec60e6412b88b066e62c2cd940", "name": "heat"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "4f1fafb25e58421e91a2139dc0382c53"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "5808ab12037f4ab49c37cba0ac9cd58e"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "a680b7674b3b430c9f5e008f72f3dfe4"}], "type": "s3", "id": "13f4cf29c7394a7a801d8aeacca24378", "name": "swift_s3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "internal", "id": "0bba70ede2ef4bc0a998ed73110406cb"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "public", "id": "25ee42ace1ad4707b485494baa5d4693"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "admin", "id": "cf817bfde96d4aa6807cdb4be21eca3b"}], "type": "dns", "id": "1b6455f4855e44bd85be449258340a02", "name": "designate"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://identity-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "8df18f47fcdc4c348d521d4724a5b7ac"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c0dd9f5f8db248b093d6735b167e1af6"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "c8505f07c349413aa7cd61d42337af99"}], "type": "identity", "id": "1ca1ba37600a436185c317bef6d88c36", "name": "keystone"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "299df431ac7942c18e2137613d8e03d8"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "bec896cde16646d6a3b56cff997d08cc"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "ce70c7029b6a45b78e008bc833187559"}], "type": "placement", "id": "37077d282f2b4961ad65f9fb87f58789", "name": "placement"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "4babd2f2b8d842e4b2303468fbf5d3ac"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "6b3ddcf427e04182a7aa4e24d4649581"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "e8e1eb90f7394f5999aec5c8f8c75c88"}], "type": "volume", "id": "40686c5446374c2b92771e67249521b3", "name": "cinder"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "218d9460bd6e4eabb32b980fc580a26a"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "21a2898a0fcf4ec783974cf75d774916"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "8842c03d2c51449ebf9ff36778cf17c1"}], "type": "image", "id": "49e5dc4cfe294aa0971e797d3bb1db20", "name": "glance"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "83fbacf0d0014a27955384fb8632de4c"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "9308a40f3a6f4dff9d20afa2dfa6b186"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "e5cb7e4e6f3440a386d397ec7e00ca44"}], "type": "cloudformation", "id": "4e77820be3494d26a0c39301082da4d7", "name": "heat-cfn"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5fdfd118a3d6460ab45c1995ca21b7f1"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ab9e70b4fde943e49d92c41d2f0892d9"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "cfa16dc27b82420a8cf7db5ea79aa3dd"}], "type": "data-processing", "id": "523b8e9f4bd44e51aec65c524c953c32", "name": "sahara"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "37860b492dd947daa738f461b9084d2a"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "7a095734e4984cc7b8ac581aa6131f23"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "96357df3d6694477b0ad17fef6091210"}], "type": "network", "id": "98f33898880441ddb91522bac15410b5", "name": "neutron"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "01fdd8e07ca74c9daf80a8b66dcc8bf6"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "a25efaf48347441a8d36ce302f31d527"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ceea6017e14e4b549882b6fe359d07e0"}], "type": "volumev2", "id": "9ec8ed2478e44b368f51dd71a6f2759c", "name": "cinderv2"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "2f582f99db974766af7548dda56c3b50"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "39e28fa99b69466195aad1fbfb610323"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "4d38fa91197e4712a2f2d3f89fcd7dad"}], "type": "compute", "id": "a30d9b8a170144f0a7bca3ce028c9ed3", "name": "nova"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5d9aa667f2fb4c83bf14ee8ae52b541a"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "96bcaf12b54d4060bc9973ded5ce4093"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "ac9138596d3942babe1f9241f93a5e04"}], "type": "volumev3", "id": "a5ae83af15b146559711905c6531c721", "name": "cinderv3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "39830aba2eea44a3a432ea0bd219c1ab"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "5883ae0fb85842c5aca05ed51327ac35"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "80bd3d21df274c43ac8457aa2230598f"}], "type": "octavia", "id": "b6a1520b90a146f4a96321960e7a243a", "name": "octavia"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://secure.vexxhost.com/console/api", "region": "ca-ymq-1", "interface": "public", "id": "156bf3730b834d2bacf7eff6c542f394"}], "type": "dashboard", "id": "b7bb9d5476c649f483dfbcb49359c17b", "name": "cloudconsole"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "20b24181722b49a3983d17d42147a22c"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "79fa33ff42ec45118ae8b36789fcb8ae"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "b073b767f10d44f895d9d14fbc3e3d6b"}], "type": "object-store", "id": "c1efec5454ec49b1b016721938401182", "name": "swift"}], "user": {"password_expires_at": null, "domain": {"id": "default", "name": "Default"}, "id": "bc2260d5b7cb40ff8270043fd7af3a08", "name": "osdn-user-1"}, "audit_ids": ["6bu2KObmSwGfO2gsIcmPNw"], "issued_at": "2017-10-17T23:16:05.000000Z"}} + +Authentication Response Readable Version +======================================== + +http://paste.openstack.org/raw/623947/ + +Listing Images +============== + +* Find 'image' Endpoint from the Catalog +* Find image api version +* Make API call + +Consuming Service Catalog +========================= + +https://specs.openstack.org/openstack/api-wg/guidelines/consuming-catalog.html + +Docs on version discovery and cloud profile: + +* https://review.openstack.org/#/c/459869 Add guideline describing a cloud profile document +* https://review.openstack.org/#/c/459710 Add guidelines on Version Discovery +* https://review.openstack.org/#/c/459405 Add document describing consuming version discovery + +Find Endpoint +============= + +* Most important piece of info: ``service_type`` (``type`` in catalog) +* ``service_name`` or name carries no meaning + +Service Types Authority +======================= + +https://service-types.openstack.org + +* Registry of known service types +* Listing of historical aliases for those service types +* https://service-types.openstack.org/service-types.json +* Not release versoined - most current file is always accurate +* Python library: os-service-types + +Find Image Endpoint +=================== + +:: + + {'endpoints': [ + {'id': '218d9460bd6e4eabb32b980fc580a26a', + 'interface': 'internal', + 'region': 'ca-ymq-1', + 'region_id': 'ca-ymq-1', + 'url': 'https://image-ca-ymq-1.vexxhost.net'}, + {'id': '21a2898a0fcf4ec783974cf75d774916', + 'interface': 'admin', + 'region': 'ca-ymq-1', + 'region_id': 'ca-ymq-1', + 'url': 'https://image-ca-ymq-1.vexxhost.net'}, + {'id': '8842c03d2c51449ebf9ff36778cf17c1', + 'interface': 'public', + 'region': 'ca-ymq-1', + 'region_id': 'ca-ymq-1', + 'url': 'https://image-ca-ymq-1.vexxhost.net'}], + 'id': '49e5dc4cfe294aa0971e797d3bb1db20', + 'name': 'glance', + 'type': 'image'}, + +Endpoint Interface +================== + +* public - Public/External API access - default value +* internal - Internal network API access +* admin - Historical - was used for keystone - not used anymore + +Endpoint Region Name +==================== + +* If there is more than one region, region_name must be specified +* If there is only one, it can be omitted + +Example with more than one region: + +http://paste.openstack.org/raw/623900/ + +Endpoints Found +=============== + +* cloud=vexxhost1,service_type=image + +https://image-ca-ymq-1.vexxhost.net + +* cloud=citycloud1,service_type=image,region_name=Sto2 + +https://sto2.citycloud.com:9292 + +Finding Image API Version +========================= + +:: + + curl -i https://image-ca-ymq-1.vexxhost.net + +http://paste.openstack.org/raw/623901/ + +* CURRENT - Best current version + +https://image-ca-ymq-1.vexxhost.net/v2/ + +Listing Images +============== + +* Send value of X-Subject-Token from auth request in X-Auth-Token + +:: + + curl -i -X GET https://image-ca-ymq-1.vexxhost.net/v2/images \ + -H "Accept: application/json" \ + -H "X-Auth-Token: gAAAAABZ5qpnytYif9wk-4fsqvgm3Ik81WQl_L4tdlfka7Vr4b-EG0w7QGF-JU_T4zkJsuql4XYSU_9epUnx-aE_UeKMMdaKAyecGG9om6TR3CA6yeeHbMJBTVFfdxvW-QFA76_JXoibCvXGIcKlpotK01BBl3DG5qYNxUFkSquk5cIQh-w63AY" + +:: + + HTTP/1.1 200 OK + Content-Length: 24153 + Content-Type: application/json + X-Openstack-Request-Id: req-8a602280-1bba-46b4-9406-97addcdd9b6c + Date: Wed, 18 Oct 2017 01:12:43 GMT + +* Content: http://paste.openstack.org/raw/623902/ +* Note pagination links + +Getting Auth information +======================== + +* Download a clouds.yaml file from horizon +* Download an legacy openrc file from horizon +* Investigate dashboard +* Carrier pigeon + +Sometimes the files are wrong and you have to apply the earlier rules. + +Incorrect openrc file (horizon bug) +=================================== + +This is downloaded from my account. Who can spot the problems? + +:: + + export OS_AUTH_URL=https://auth.vexxhost.net/v2.0 + export OS_PROJECT_ID=bbb1e479a9d04f34941470f37c7a0ecb + export OS_PROJECT_NAME="osdn-project-1" + export OS_USER_DOMAIN_NAME="None" + export OS_USERNAME="osdn-user-1" + export OS_REGION_NAME="ca-ymq-1" + export OS_INTERFACE=public + export OS_IDENTITY_API_VERSION=3 + +Put the info in a clouds.yaml file +================================== + +clouds.yaml +=========== + +Information about the clouds you want to connect to is stored in a file +called `clouds.yaml`. + +`clouds.yaml` can be in your homedir: `~/.config/openstack/clouds.yaml` +or system-wide: `/etc/openstack/clouds.yaml`. + +Information in your homedir, if it exists, takes precedence. + +Full docs on `clouds.yaml` are at +https://docs.openstack.org/developer/os-client-config/ + +What about Mac and Windows? +=========================== + +`USER_CONFIG_DIR` is different on Linux, OSX and Windows. + +* Linux: `~/.config/openstack` +* OSX: `~/Library/Application Support/openstack` +* Windows: `C:\\Users\\USERNAME\\AppData\\Local\\OpenStack\\openstack` + +`SITE_CONFIG_DIR` is different on Linux, OSX and Windows. + +* Linux: `/etc/openstack` +* OSX: `/Library/Application Support/openstack` +* Windows: `C:\\ProgramData\\OpenStack\\openstack` + +Config Terminology +================== + +For multi-cloud, think of two types: + +* `profile` - Facts about the `cloud` that are true for everyone +* `cloud` - Information specific to a given `user` + +Apologies for the use of `cloud` twice. + +Example clouds.yaml file +======================== + +.. code-block:: yaml + + clouds: + vexxhost1: + identity_api_version: 3 + auth: + auth_url: https://auth.vexxhost.net + user_domain_id: default + project_domain_id: default + project_name: osdn-project-1 + username: osdn-user-1 + password: XXXXXXXX + +Vexxhost is known +================= + +.. code-block:: yaml + + clouds: + vexxhost1: + identity_api_version: 3 + profile: vexxhost + auth: + user_domain_id: default + project_domain_id: default + project_name: osdn-project-1 + username: osdn-user-1 + password: XXXXXXXX + +More than one cloud +=================== + +Yes- I created the same user with the same password on CityCloud: + +.. code-block:: yaml + + clouds: + citycloud1: + profile: citycloud + auth: + username: osdn-user-1 + password: XXXXXXXX + project_name: osdn-project-1 + user_domain_id: d0919bd5e8d74e49adf0e145807ffc38 + project_domain_id: d0919bd5e8d74e49adf0e145807ffc38 + vexxhost1: + profile: vexxhost + auth: + user_domain_id: default + project_domain_id: default + project_name: osdn-project-1 + username: osdn-user-1 + password: XXXXXXXX + +Quick note about SSL +==================== + +You can disable SSL verification:: + + unitedstack: + profile: unitedstack + verify: False + +You can set specific SSL arguments:: + + openstackci-infracloud-vanilla: + region_name: RegionOne + cacert: /home/nodepool/.config/openstack/infracloud_vanilla_cacert.pem + auth: + auth_url: https://controller00.vanilla.ic.openstack.org:5000 + username: openstackci + +Auth per cloud, select per region +================================= + +In general, the thing you need to know is: + +* Configure authentication per `cloud` +* Select config to use by `cloud` and `region` + +Environment Variables and Simple Usage +====================================== + +* Environment variables starting with `OS_` go into a cloud called `envvars` +* If you only have one cloud, you don't have to specify it +* `OS_CLOUD` and `OS_REGION_NAME` are default values for + `cloud` and `region_name` + +Easier version of the above REST +================================ + +* What you should be able to do + +.. code-block:: python + + import os_client_config + client = os_client_config.make_rest_client('image', cloud='vexxhost1') + print(client.get('/images').json()) + +* There's a bug with make_rest_client and glance - I'll get it fixed this week. + Workaround: + +.. code-block:: python + + import os_client_config + from keystoneauth1 import adapter + config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') + client = adapter.Adapter( + config.get_session(), service_type='image', version='latest') + print(client.get('/images').json()) + +Sessions and Adapters +===================== + +* A Session is an Authenticated HTTP connection to a cloud + +.. code-block:: python + + config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') + session = config.get_session() + session.get('https://image-ca-ymq-1.vexxhost.net/v2/images') + +* An "Adapter" is a Session that is 'mounted' on a base endpoint + +.. code-block:: python + + config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') + client = adapter.Adapter( + config.get_session(), service_type='image', version='latest') + session.get('/images') + +* ``os_client_config.make_rest_client`` returns a mounted Adapter +* Latest keystoneauth library handles version discovery properly in Adapter + +Creating keystoneauth session directly +====================================== + +.. code-block:: python + + from keystoneauth1 import adapter, loading, session + loader = loading.get_plugin_loader('password') + auth = loader.load_from_options( + auth_url='https://auth.vexxhost.net', + username='osdn-user-1', + user_domain_id='default', + password='XXXXXXXX', + project_name='osdn-project-1', + project_domain_id='default') + # verify and ssl cert info go here too + session = session.Session(auth) + # interface, region_name also go here + client = adapter.Adapter(session, service_type='image', version='latest') + client.get('/images') + +Command Line +============ + +.. code-bock:: bash + + openstack --os-cloud=vexxhost1 image list + +Back to shade +============= + +.. code-block:: python + + import shade + cloud = shade.openstack_cloud(cloud='vexxhost') + cloud.pprint(cloud.list_images()) + +Shade Logging +============= + +* `shade` uses standard python logging +* `simple_logging` does easy defaults +* Squelches some meaningless warnings + + * `debug` + + * Logs shade loggers at debug level + + * `http_debug` Implies `debug`, turns on HTTP tracing + +.. code:: python + + # Initialize and turn on debug logging + shade.simple_logging(debug=True) + +Find out what REST calls shade is making +======================================== + +.. code-block:: python + + import shade + shade.simple_logging(http_debug=True) + cloud = shade.openstack_cloud(cloud='vexxhost1') + cloud.pprint(cloud.list_images()) + +http://paste.openstack.org/raw/623906/ + +Real tokens are not logged, only a sha of the token. Reusing the printed +curl commands reuquires replacing the token with a real one. + +:: + + "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" + +Microversions +============= + +* Per-HTTP-request behavior differences. +* Services declare microversion support ranges in Version Discovery Document + +.. code-block:: bash + + curl -i https://compute-ca-ymq-1.vexxhost.net + +Nova Compute Version Document Part One +====================================== + +:: + + HTTP/1.1 200 OK + Date: Wed, 18 Oct 2017 01:58:51 GMT + Server: Apache/2.4.6 (CentOS) + Content-Length: 399 + Vary: Accept-Encoding + Content-Type: application/json + + { + "versions": [ + { + "status": "SUPPORTED", + "version": "", + "updated": "2011-01-21T11:33:21Z", + "id": "v2.0", + "links": [ + { + "href": "https://compute-ca-ymq-1.vexxhost.net/v2/", + "rel": "self" + } + ], + "min_version": "" + }, + +Nova Compute Version Document Part Two +====================================== + +Note 'version' and 'min_version'. + +:: + + { + "status": "CURRENT", + "version": "2.53", + "updated": "2013-07-23T11:33:21Z", + "id": "v2.1", + "links": [ + { + "href": "https://compute-ca-ymq-1.vexxhost.net/v2.1/", + "rel": "self" + } + ], + "min_version": "2.1" + } + +Microversion Field Names +======================== + +The min and max available microversion for the given Major API version. + +In some services they are: + +* min_version +* version + +In other services they are: + +* min_version +* max_version + +Request a Microversion +====================== + +* Add a OpenStack-API-Version header with service-type and version:: + + OpenStack-API-Version: compute 2.13 + +* Response will contain the same header specifying which version was used:: + + OpenStack-API-Version: compute 2.13 + +* Older versions of Nova (pre Newton - or versions that do not support a max + version of 2.27 or higher), use a different form:: + + X-OpenStack-Nova-API-Version: 2.13 + +* It doesn't hurt anything to send both - which is what keystoneauth does + under the covers. + +Documentation on Nova Microversion Changes +========================================== + +https://docs.openstack.org/nova/latest/reference/api-microversion-history.html + +Simple Microversion Example with Default +======================================== + +Microversion 2.36 removed some calls. Before 2.36 nova had a ``/images`` API +that would proxy to glance. It is removed in 2.36 and later. + +Making a GET call defaults to the minimum (2.1 in this case) + +.. code-bock:: python + + import os_client_config + + client = os_client_config.make_rest_client('compute', cloud='vexxhost') + client.get('/images') + +:: + + curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" + [200] Date: Wed, 18 Oct 2017 02:10:49 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac x-compute-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac Content-Encoding: gzip Content-Length: 2411 Content-Type: application/json + +Simple Microversion Example with Microversion +============================================= + +Sending microversion 2.36 for the same call tells Nova to use the 2.36 +behavior, which results in a 404. + +.. code-bock:: python + + import os_client_config + + client = os_client_config.make_rest_client('compute', cloud='vexxhost') + client.get('/images', microversion='2.36') + +:: + + curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "OpenStack-API-Version: compute 2.36" -H "X-OpenStack-Nova-API-Version: 2.36" + [404] Date: Wed, 18 Oct 2017 02:11:00 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.36 X-OpenStack-Nova-API-Version: 2.36 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version x-openstack-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 x-compute-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 Content-Length: 78 Content-Type: application/json; charset=UTF-8 + {"itemNotFound": {"message": "The resource could not be found.", "code": 404}} + + +Create Server Group +=================== + +The server group doesn't matter - this is just to have an object to show. + +.. code-bock:: python + + import os_client_config + + client = os_client_config.make_rest_client('compute', cloud='vexxhost') + client.post( + '/os-server-groups', + json=dict(server_group=dict(name='test', policies=['anti-affinity']))) + +Example of No Microversion Content Behavior +=========================================== + +No microversion specified. + +.. code-bock:: python + + client.get('/os-server-groups') + +:: + + curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" + [200] Date: Wed, 18 Oct 2017 02:18:30 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-85e061c0-d474-497c-bd17-96b629086169 x-compute-request-id: req-85e061c0-d474-497c-bd17-96b629086169 Content-Encoding: gzip Content-Length: 137 Content-Type: application/json + {"server_groups": [{"members": [], "metadata": {}, "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "policies": ["anti-affinity"], "name": "test"}]} + +Example of Microversion Content Behavior +======================================== + +Request microversion 2.13, which adds user_id and project_id. + +.. code-bock:: python + + client.get('/os-server-groups', microversion='2.13').json() + +:: + + curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "OpenStack-API-Version: compute 2.13" -H "X-OpenStack-Nova-API-Version: 2.13" + [200] Date: Wed, 18 Oct 2017 02:20:13 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.13 X-OpenStack-Nova-API-Version: 2.13 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 x-compute-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 Content-Encoding: gzip Content-Length: 195 Content-Type: application/json + {"server_groups": [{"user_id": "bc2260d5b7cb40ff8270043fd7af3a08", "policies": ["anti-affinity"], "name": "test", "members": [], "project_id": "bbb1e479a9d04f34941470f37c7a0ecb", "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "metadata": {}}]} + -- cgit v1.2.3