========================================================= Everything you need to know about using the OpenStack API ========================================================= This document contains a presentation in `presentty`_ format. If you want to walk through it like a presentation, install `presentty` and run: .. code:: bash presentty doc/source/user/multi-cloud-demo.rst The content is hopefully helpful even if it's not being narrated, so it's being included in the `shade` docs. .. _presentty: https://pypi.python.org/pypi/presentty Everything you need to know about using the OpenStack API ========================================================= Who am I? ========= Monty Taylor * OpenStack Infra Core * irc: mordred * twitter: @e_monty This Talk is Free Software ========================== * http://git.inaugust.com/cgit/inaugust.com/tree/src/everything-you-need-to-know.rst What are we going to talk about? ================================ * Overview of some terminology * Discussion of SDKs * 'Simple' example * Step by step through in much more detail * Auth * The Catalog * Service Types * Version Discovery * Actually doing things * Microversions Expected Ranting ================ * Fixed vs. Floating IPs * Build your own images * Boot-time scripting with cloud-init Relevant Docs ============= * https://specs.openstack.org/openstack/api-wg/ * https://service-types.openstack.org/ * https://developer.openstack.org/api-ref/identity/ * https://developer.openstack.org/api-ref/compute/ * https://docs.openstack.org/os-client-config/latest/ * https://docs.openstack.org/keystoneauth/latest/using-sessions.html Cloud Terminology ================= Let's define a few terms, so that we can use them with ease: * `cloud` - logically related collection of services * `region` - completely independent subset of a given cloud * `patron` - human who has an account * `user` - account on a cloud * `project` - logical collection of cloud resources * `domain` - collection of users and projects Cloud Terminology Relationships =============================== * A `cloud` has one or more `regions` * A `patron` has one or more `users` * A `patron` has one or more `projects` * A `cloud` has one or more `domains` * In a `cloud` with one `domain` it is named "default" * Each `patron` may have their own `domain` * Each `user` is in one `domain` * Each `project` is in one `domain` * A `user` has one or more `roles` on one or more `projects` HTTP Sessions ============= * HTTP interactions are authenticated via keystone * Authenticating returns a `token` * An authenticated HTTP Session is shared across a `region` Cloud Regions ============= A `cloud region` is the basic unit of REST interaction. * A `cloud` has a `service catalog` * The `service catalog` is returned in the `token` * The `service catalog` lists `endpoint` for each `service` in each `region` * A `region` is completely autonomous Availability Zones ================== * If you have exposure to AWS - OpenStack Availability Zones are not what you think they are. * OpenStack Availability Zones are tagged groupings of hypervisor hosts, storage nodes and network nodes. Users, Projects and Domains =========================== In clouds with multiple domains, project and user names are only unique within a region. * Names require `domain` information for uniqueness. IDs do not. * Providing `domain` information when not needed is fine. * `project_name` requires `project_domain_name` or `project_domain_id` * `project_id` does not * `username` requires `user_domain_name` or `user_domain_id` * `user_id` does not Auth vs. Identity ================= * Both provided by the ``keystone`` service * ``identity`` refers to CRUD operations on users, projects, etc. * ``auth`` refers to a user getting a token and catalog Keystone v2 vs. Keystone v3 =========================== * Two major versions of identity API * v2 is deprecated/legacy * ``project`` was called ``tenant`` in v2 * ``domain`` did not exist in v2 Confused Yet? ============= Awesome SDKs ==== REST ==== * You can always do direct REST * Auth and discovery are a bit of a pain * Once you have those, it's all pretty easy python ====== First: * python-openstackclient ... CLI tool ``openstack`` Five main libraries to be aware of: * keystoneauth * os-client-config * shade * python-openstacksdk * python-*client python-*client ============== * Per-service client libraries * Avoid them. They're not written for you. They make things harder. keystoneauth ============ * wrapper around python-requests that handles auth, catalog endpoint and version discovery * "REST" library os-client-config ================ * Library to handle client configuration * Can create python REST clients for you * Used under the covers by shade, python-openstacksdk and python-openstackclient * Handles clouds.yaml files, environment variables and command line arguments shade ===== * a task and end-user oriented Python library * abstracts deployment differences * designed for multi-cloud * simple to use * massive scale * optional advanced features to handle 20k servers a day * Initial logic/design extracted from nodepool * Librified to re-use in Ansible python-openstacksdk =================== * Object oriented interface to OpenStack REST APIs * Just got adopted by the shade team * Interface is going to break soon * shade, python-openstacksdk and os-client-config are going to merge * shade is safe to use - there will always be backwards compat for shade Multi-Cloud Shade Example ========================== .. code:: python import shade # Initialize and turn on debug logging shade.simple_logging(debug=True) for cloud_name, region_name in [ ('my-vexxhost', 'ca-ymq-1'), ('my-citycloud', 'Buf1'), ('my-internap', 'ams01')]: # Initialize cloud cloud = shade.openstack_cloud(cloud=cloud_name, region_name=region_name) # Upload an image to the cloud image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]') # Find a flavor with at least 512M of RAM flavor = cloud.get_flavor_by_ram(512) # Boot a server, wait for it to boot, and then do whatever is needed # to get a public ip for it. cloud.create_server( 'my-server', image=image, flavor=flavor, wait=True, auto_ip=True) It's Not Possible To Cover Everything That Does =============================================== Simpler Example =============== It still may not be possible to cover it all. .. code-block:: python import shade # Initialize and turn on debug logging shade.simple_logging(debug=True) # Initialize cloud cloud = shade.openstack_cloud(cloud='vexxhost') # Upload an image to the cloud image = cloud.get_image('Fedora 24 (Cloud Edition) [2016-09-23]') # Find a flavor with at least 512M of RAM flavor = cloud.get_flavor_by_ram(512) # Boot a server, wait for it to boot, and then do whatever is needed # to get a public ip for it. cloud.create_server( 'my-server', image=image, flavor=flavor, wait=True, auto_ip=True) Get Some Stuff Installed ======================== .. code-block:: bash virtualenv venv source venv/bin/activate pip install shade Essential pieces of Auth Information ==================================== * auth_url * password * (username + (user_domain_name or user_domain_id)) or user_id) * (project_name + (project_domain_name or project_domain_id)) or project_id Note: It's important to know if you have a name or id for your user and project. I Lied ====== Essential pieces of Auth Information ==================================== * auth_url * auth API version * auth_type If nobody tells you an auth_type, assume ``password`` If nobody tells you an auth_url, look at https://docs.openstack.org/os-client-config/latest/user/vendor-support.html Exercise - Find Auth URL for known cloud =========================================== The cloud is Vexxhost. Answer ====== https://auth.vexxhost.net Version Discovery ================= * Each OpenStack service has a 'Version Discovery' document at its unversioned root URL. We call this the 'Unversioned Discovery Document' * Except for swift. (pending some work with discovery caching) * Each OpenStack service ALSO has a 'Version Discovery' document at the root of each Versioned URL. We call this the 'Versioned Discovery Document' * Version Discovery documents should not require Authentication * If they do on your cloud, your cloud is broken, please file a bug Exercise #2 - Find Auth API Version from Auth URL ================================================= :: curl -i https://auth.vexxhost.net Auth Version Discovery Response =============================== :: { "versions": { "values": [ { "status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v3+json" } ], "id": "v3.8", "links": [ { "href": "https://auth.vexxhost.net/v3/", "rel": "self" } ] }, Auth Version Discovery Response Part 2 ====================================== :: { "status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json" } ], "id": "v2.0", "links": [ { "href": "https://auth.vexxhost.net/v2.0/", "rel": "self" }, { "href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby" } ] } ] } } Versioned Discovery Document ============================ Sometimes a cloud gives you a versioned auth_url. :: curl -i https://auth.vexxhost.net/v3 Auth Version Discovery Resonse ============================== :: { "version": { "status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v3+json" } ], "id": "v3.8", "links": [ { "href": "https://auth.vexxhost.net/v3/", "rel": "self" } ] } } Finding a Better Version ======================== Sometimes a cloud gives you a v2 versioned auth_url. :: curl -i https://auth.vexxhost.net/v2.0 **ALWAYS** prefer v3 auth if it's available. Old Auth Version Discovery Resonse ================================== :: { "version": { "status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json" } ], "id": "v2.0", "links": [ { "href": "https://auth.vexxhost.net/v2.0/", "rel": "self" }, { "href": "https://docs.openstack.org/", "type": "text/html", "rel": "describedby" } ] } } Status Deprecated ================= Strip the trailing ``v.*`` portion from the URL and try again. :: { "version": { "status": "deprecated", Strip the trailing ``v.*`` portion from the URL and try again. If ``v3`` is not available, complain to someone. Keystone v3 api is considered stable as of February 20, 2013. Isn't this fun? =============== To Domain or Not To Domain ========================== In ``v3``: * Project Names and User Names are only unique within a domain * If you don't have domain info, assume ``domain_id`` is ``default`` In ``v2``: * No such thing as domains * Projects are called "tenant" Essential pieces of Auth Information ==================================== * auth_url - https://auth.vexxhost.net * auth version - ``3`` * versioned auth endpoint - https://auth.vexxhost.net/v3 * username - osdn-user-1 * user domain id - ``default`` * project_name - osdn-project-1 * project domain id = ``default`` * password - XXXXXXXX Now we can authenticate!!! ========================== :: curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens -H "Content-Type: application/json" -H "Accept: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": {"user": {"domain": {"id": "default"}, "name": "osdn-user-1", "password": "XXXXXXXX"}}},"scope": {"project": {"domain": {"id": "default"}, "name": "osdn-project-1"}}}}' What, you can't read that? ========================== :: curl -g -i -X POST https://auth.vexxhost.net/v3/auth/tokens \ -H "Content-Type: application/json" \ -H "Accept: application/json" -d '{ "auth": { "identity": { "methods": ["password"], "password": { "user": { "domain": {"id": "default"}, "name": "osdn-user-1", "password": "XXXXXXXX"}}}, "scope": { "project": { "domain": {"id": "default"}, "name": "osdn-project-1"}}}}' Authentication Response ======================= Readable version of payload: http://paste.openstack.org/raw/623947/ :: HTTP/1.1 201 Created Date: Tue, 17 Oct 2017 23:16:05 GMT Server: Apache/2.4.6 (CentOS) X-Subject-Token: gAAAAABZ5o81tyDsyJMXl6hVcVOqHgzKQ2WmpvepCSr2NWYM1D97VoaBy-i5OLT8-BbkWBZJq_aqrt03eutp2eEpORI-V4CYpYcWBDkR5jLNfq2gD-TDB9lzvRo7rHKu-PWSLIxEF7whsFJ8bVGDPMN4yc_ZwgfWcR13GxAKutU9_rzR9Gvewjg Vary: X-Auth-Token x-openstack-request-id: req-29d8c4e6-1924-42e9-8efa-4f154e813ee6 Content-Length: 11696 Content-Type: application/json {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "fd6d2130b7784b82aa6b49fd5901f70b", "name": "Member"}], "expires_at": "2017-10-18T23:16:05.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "bbb1e479a9d04f34941470f37c7a0ecb", "name": "osdn-project-1"}, "catalog": [{"endpoints": [{"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "522041f846934310b928a6afab47241e"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "660a742372554b5581c1f4870d4d42cf"}, {"region_id": "ca-ymq-1", "url": "https://key-manager-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c1c32be0de8441eea4306b8c22442ba3"}], "type": "key-manager", "id": "023de54c09654931856403e708b60a22", "name": "barbican"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "289f7dcf096743fb9a17924bd598e716"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "61b6d283ef4b419a8ad23fc53dea70df"}, {"region_id": "ca-ymq-1", "url": "https://container-infra-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "e1e7c79ce7834ec0aa74308326e2f96c"}], "type": "container-infra", "id": "041eac2489bb495b8a900c40b93b2065", "name": "magnum"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "203fd4e466b44569aa9ab8c78ef55bad"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "da3d087e0c724338ba12c9a1168ef80c"}, {"region_id": "ca-ymq-1", "url": "https://orchestration-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "f0a311cb5dbf4ae788670107a3433ac2"}], "type": "orchestration", "id": "0e852aec60e6412b88b066e62c2cd940", "name": "heat"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "4f1fafb25e58421e91a2139dc0382c53"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "5808ab12037f4ab49c37cba0ac9cd58e"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "a680b7674b3b430c9f5e008f72f3dfe4"}], "type": "s3", "id": "13f4cf29c7394a7a801d8aeacca24378", "name": "swift_s3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "internal", "id": "0bba70ede2ef4bc0a998ed73110406cb"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "public", "id": "25ee42ace1ad4707b485494baa5d4693"}, {"region_id": "ca-ymq-1", "url": "http://dns.vexxhost.net/", "region": "ca-ymq-1", "interface": "admin", "id": "cf817bfde96d4aa6807cdb4be21eca3b"}], "type": "dns", "id": "1b6455f4855e44bd85be449258340a02", "name": "designate"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://identity-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "8df18f47fcdc4c348d521d4724a5b7ac"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "c0dd9f5f8db248b093d6735b167e1af6"}, {"region_id": "ca-ymq-1", "url": "https://auth.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "c8505f07c349413aa7cd61d42337af99"}], "type": "identity", "id": "1ca1ba37600a436185c317bef6d88c36", "name": "keystone"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "299df431ac7942c18e2137613d8e03d8"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "bec896cde16646d6a3b56cff997d08cc"}, {"region_id": "ca-ymq-1", "url": "https://placement-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "ce70c7029b6a45b78e008bc833187559"}], "type": "placement", "id": "37077d282f2b4961ad65f9fb87f58789", "name": "placement"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "4babd2f2b8d842e4b2303468fbf5d3ac"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "6b3ddcf427e04182a7aa4e24d4649581"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "e8e1eb90f7394f5999aec5c8f8c75c88"}], "type": "volume", "id": "40686c5446374c2b92771e67249521b3", "name": "cinder"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "218d9460bd6e4eabb32b980fc580a26a"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "21a2898a0fcf4ec783974cf75d774916"}, {"region_id": "ca-ymq-1", "url": "https://image-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "8842c03d2c51449ebf9ff36778cf17c1"}], "type": "image", "id": "49e5dc4cfe294aa0971e797d3bb1db20", "name": "glance"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "admin", "id": "83fbacf0d0014a27955384fb8632de4c"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "internal", "id": "9308a40f3a6f4dff9d20afa2dfa6b186"}, {"region_id": "ca-ymq-1", "url": "https://cloudformation-ca-ymq-1.vexxhost.net/v1", "region": "ca-ymq-1", "interface": "public", "id": "e5cb7e4e6f3440a386d397ec7e00ca44"}], "type": "cloudformation", "id": "4e77820be3494d26a0c39301082da4d7", "name": "heat-cfn"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5fdfd118a3d6460ab45c1995ca21b7f1"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ab9e70b4fde943e49d92c41d2f0892d9"}, {"region_id": "ca-ymq-1", "url": "https://data-processing-ca-ymq-1.vexxhost.net/v1.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "cfa16dc27b82420a8cf7db5ea79aa3dd"}], "type": "data-processing", "id": "523b8e9f4bd44e51aec65c524c953c32", "name": "sahara"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "37860b492dd947daa738f461b9084d2a"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "7a095734e4984cc7b8ac581aa6131f23"}, {"region_id": "ca-ymq-1", "url": "https://network-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "96357df3d6694477b0ad17fef6091210"}], "type": "network", "id": "98f33898880441ddb91522bac15410b5", "name": "neutron"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "01fdd8e07ca74c9daf80a8b66dcc8bf6"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "a25efaf48347441a8d36ce302f31d527"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v2/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "ceea6017e14e4b549882b6fe359d07e0"}], "type": "volumev2", "id": "9ec8ed2478e44b368f51dd71a6f2759c", "name": "cinderv2"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "2f582f99db974766af7548dda56c3b50"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "39e28fa99b69466195aad1fbfb610323"}, {"region_id": "ca-ymq-1", "url": "https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "4d38fa91197e4712a2f2d3f89fcd7dad"}], "type": "compute", "id": "a30d9b8a170144f0a7bca3ce028c9ed3", "name": "nova"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "5d9aa667f2fb4c83bf14ee8ae52b541a"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "96bcaf12b54d4060bc9973ded5ce4093"}, {"region_id": "ca-ymq-1", "url": "https://block-storage-ca-ymq-1.vexxhost.net/v3/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "ac9138596d3942babe1f9241f93a5e04"}], "type": "volumev3", "id": "a5ae83af15b146559711905c6531c721", "name": "cinderv3"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "internal", "id": "39830aba2eea44a3a432ea0bd219c1ab"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "public", "id": "5883ae0fb85842c5aca05ed51327ac35"}, {"region_id": "ca-ymq-1", "url": "https://octavia-ca-ymq-1.vexxhost.net", "region": "ca-ymq-1", "interface": "admin", "id": "80bd3d21df274c43ac8457aa2230598f"}], "type": "octavia", "id": "b6a1520b90a146f4a96321960e7a243a", "name": "octavia"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "http://secure.vexxhost.com/console/api", "region": "ca-ymq-1", "interface": "public", "id": "156bf3730b834d2bacf7eff6c542f394"}], "type": "dashboard", "id": "b7bb9d5476c649f483dfbcb49359c17b", "name": "cloudconsole"}, {"endpoints": [{"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "admin", "id": "20b24181722b49a3983d17d42147a22c"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "public", "id": "79fa33ff42ec45118ae8b36789fcb8ae"}, {"region_id": "ca-ymq-1", "url": "https://object-storage-ca-ymq-1.vexxhost.net/v1/bbb1e479a9d04f34941470f37c7a0ecb", "region": "ca-ymq-1", "interface": "internal", "id": "b073b767f10d44f895d9d14fbc3e3d6b"}], "type": "object-store", "id": "c1efec5454ec49b1b016721938401182", "name": "swift"}], "user": {"password_expires_at": null, "domain": {"id": "default", "name": "Default"}, "id": "bc2260d5b7cb40ff8270043fd7af3a08", "name": "osdn-user-1"}, "audit_ids": ["6bu2KObmSwGfO2gsIcmPNw"], "issued_at": "2017-10-17T23:16:05.000000Z"}} Authentication Response Readable Version ======================================== http://paste.openstack.org/raw/623947/ Listing Images ============== * Find 'image' Endpoint from the Catalog * Find image api version * Make API call Consuming Service Catalog ========================= https://specs.openstack.org/openstack/api-wg/guidelines/consuming-catalog.html Docs on version discovery and cloud profile: * https://review.openstack.org/#/c/459869 Add guideline describing a cloud profile document * https://review.openstack.org/#/c/459710 Add guidelines on Version Discovery * https://review.openstack.org/#/c/459405 Add document describing consuming version discovery Find Endpoint ============= * Most important piece of info: ``service_type`` (``type`` in catalog) * ``service_name`` or name carries no meaning Service Types Authority ======================= https://service-types.openstack.org * Registry of known service types * Listing of historical aliases for those service types * https://service-types.openstack.org/service-types.json * Not release versoined - most current file is always accurate * Python library: os-service-types Find Image Endpoint =================== :: {'endpoints': [ {'id': '218d9460bd6e4eabb32b980fc580a26a', 'interface': 'internal', 'region': 'ca-ymq-1', 'region_id': 'ca-ymq-1', 'url': 'https://image-ca-ymq-1.vexxhost.net'}, {'id': '21a2898a0fcf4ec783974cf75d774916', 'interface': 'admin', 'region': 'ca-ymq-1', 'region_id': 'ca-ymq-1', 'url': 'https://image-ca-ymq-1.vexxhost.net'}, {'id': '8842c03d2c51449ebf9ff36778cf17c1', 'interface': 'public', 'region': 'ca-ymq-1', 'region_id': 'ca-ymq-1', 'url': 'https://image-ca-ymq-1.vexxhost.net'}], 'id': '49e5dc4cfe294aa0971e797d3bb1db20', 'name': 'glance', 'type': 'image'}, Endpoint Interface ================== * public - Public/External API access - default value * internal - Internal network API access * admin - Historical - was used for keystone - not used anymore Endpoint Region Name ==================== * If there is more than one region, region_name must be specified * If there is only one, it can be omitted Example with more than one region: http://paste.openstack.org/raw/623900/ Endpoints Found =============== * cloud=vexxhost1,service_type=image https://image-ca-ymq-1.vexxhost.net * cloud=citycloud1,service_type=image,region_name=Sto2 https://sto2.citycloud.com:9292 Finding Image API Version ========================= :: curl -i https://image-ca-ymq-1.vexxhost.net http://paste.openstack.org/raw/623901/ * CURRENT - Best current version https://image-ca-ymq-1.vexxhost.net/v2/ Listing Images ============== * Send value of X-Subject-Token from auth request in X-Auth-Token :: curl -i -X GET https://image-ca-ymq-1.vexxhost.net/v2/images \ -H "Accept: application/json" \ -H "X-Auth-Token: gAAAAABZ5qpnytYif9wk-4fsqvgm3Ik81WQl_L4tdlfka7Vr4b-EG0w7QGF-JU_T4zkJsuql4XYSU_9epUnx-aE_UeKMMdaKAyecGG9om6TR3CA6yeeHbMJBTVFfdxvW-QFA76_JXoibCvXGIcKlpotK01BBl3DG5qYNxUFkSquk5cIQh-w63AY" :: HTTP/1.1 200 OK Content-Length: 24153 Content-Type: application/json X-Openstack-Request-Id: req-8a602280-1bba-46b4-9406-97addcdd9b6c Date: Wed, 18 Oct 2017 01:12:43 GMT * Content: http://paste.openstack.org/raw/623902/ * Note pagination links Getting Auth information ======================== * Download a clouds.yaml file from horizon * Download an legacy openrc file from horizon * Investigate dashboard * Carrier pigeon Sometimes the files are wrong and you have to apply the earlier rules. Incorrect openrc file (horizon bug) =================================== This is downloaded from my account. Who can spot the problems? :: export OS_AUTH_URL=https://auth.vexxhost.net/v2.0 export OS_PROJECT_ID=bbb1e479a9d04f34941470f37c7a0ecb export OS_PROJECT_NAME="osdn-project-1" export OS_USER_DOMAIN_NAME="None" export OS_USERNAME="osdn-user-1" export OS_REGION_NAME="ca-ymq-1" export OS_INTERFACE=public export OS_IDENTITY_API_VERSION=3 Put the info in a clouds.yaml file ================================== clouds.yaml =========== Information about the clouds you want to connect to is stored in a file called `clouds.yaml`. `clouds.yaml` can be in your homedir: `~/.config/openstack/clouds.yaml` or system-wide: `/etc/openstack/clouds.yaml`. Information in your homedir, if it exists, takes precedence. Full docs on `clouds.yaml` are at https://docs.openstack.org/developer/os-client-config/ What about Mac and Windows? =========================== `USER_CONFIG_DIR` is different on Linux, OSX and Windows. * Linux: `~/.config/openstack` * OSX: `~/Library/Application Support/openstack` * Windows: `C:\\Users\\USERNAME\\AppData\\Local\\OpenStack\\openstack` `SITE_CONFIG_DIR` is different on Linux, OSX and Windows. * Linux: `/etc/openstack` * OSX: `/Library/Application Support/openstack` * Windows: `C:\\ProgramData\\OpenStack\\openstack` Config Terminology ================== For multi-cloud, think of two types: * `profile` - Facts about the `cloud` that are true for everyone * `cloud` - Information specific to a given `user` Apologies for the use of `cloud` twice. Example clouds.yaml file ======================== .. code-block:: yaml clouds: vexxhost1: identity_api_version: 3 auth: auth_url: https://auth.vexxhost.net user_domain_id: default project_domain_id: default project_name: osdn-project-1 username: osdn-user-1 password: XXXXXXXX Vexxhost is known ================= .. code-block:: yaml clouds: vexxhost1: identity_api_version: 3 profile: vexxhost auth: user_domain_id: default project_domain_id: default project_name: osdn-project-1 username: osdn-user-1 password: XXXXXXXX More than one cloud =================== Yes- I created the same user with the same password on CityCloud: .. code-block:: yaml clouds: citycloud1: profile: citycloud auth: username: osdn-user-1 password: XXXXXXXX project_name: osdn-project-1 user_domain_id: d0919bd5e8d74e49adf0e145807ffc38 project_domain_id: d0919bd5e8d74e49adf0e145807ffc38 vexxhost1: profile: vexxhost auth: user_domain_id: default project_domain_id: default project_name: osdn-project-1 username: osdn-user-1 password: XXXXXXXX Quick note about SSL ==================== You can disable SSL verification:: unitedstack: profile: unitedstack verify: False You can set specific SSL arguments:: openstackci-infracloud-vanilla: region_name: RegionOne cacert: /home/nodepool/.config/openstack/infracloud_vanilla_cacert.pem auth: auth_url: https://controller00.vanilla.ic.openstack.org:5000 username: openstackci Auth per cloud, select per region ================================= In general, the thing you need to know is: * Configure authentication per `cloud` * Select config to use by `cloud` and `region` Environment Variables and Simple Usage ====================================== * Environment variables starting with `OS_` go into a cloud called `envvars` * If you only have one cloud, you don't have to specify it * `OS_CLOUD` and `OS_REGION_NAME` are default values for `cloud` and `region_name` Easier version of the above REST ================================ * What you should be able to do .. code-block:: python import os_client_config client = os_client_config.make_rest_client('image', cloud='vexxhost1') print(client.get('/images').json()) * There's a bug with make_rest_client and glance - I'll get it fixed this week. Workaround: .. code-block:: python import os_client_config from keystoneauth1 import adapter config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') client = adapter.Adapter( config.get_session(), service_type='image', version='latest') print(client.get('/images').json()) Sessions and Adapters ===================== * A Session is an Authenticated HTTP connection to a cloud .. code-block:: python config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') session = config.get_session() session.get('https://image-ca-ymq-1.vexxhost.net/v2/images') * An "Adapter" is a Session that is 'mounted' on a base endpoint .. code-block:: python config = os_client_config.OpenStackConfig().get_one_cloud(cloud='vexxhost1') client = adapter.Adapter( config.get_session(), service_type='image', version='latest') session.get('/images') * ``os_client_config.make_rest_client`` returns a mounted Adapter * Latest keystoneauth library handles version discovery properly in Adapter Creating keystoneauth session directly ====================================== .. code-block:: python from keystoneauth1 import adapter, loading, session loader = loading.get_plugin_loader('password') auth = loader.load_from_options( auth_url='https://auth.vexxhost.net', username='osdn-user-1', user_domain_id='default', password='XXXXXXXX', project_name='osdn-project-1', project_domain_id='default') # verify and ssl cert info go here too session = session.Session(auth) # interface, region_name also go here client = adapter.Adapter(session, service_type='image', version='latest') client.get('/images') Command Line ============ .. code-block:: bash openstack --os-cloud=vexxhost1 image list Back to shade ============= .. code-block:: python import shade cloud = shade.openstack_cloud(cloud='vexxhost') cloud.pprint(cloud.list_images()) Shade Logging ============= * `shade` uses standard python logging * `simple_logging` does easy defaults * Squelches some meaningless warnings * `debug` * Logs shade loggers at debug level * `http_debug` Implies `debug`, turns on HTTP tracing .. code:: python # Initialize and turn on debug logging shade.simple_logging(debug=True) Find out what REST calls shade is making ======================================== .. code-block:: python import shade shade.simple_logging(http_debug=True) cloud = shade.openstack_cloud(cloud='vexxhost1') cloud.pprint(cloud.list_images()) http://paste.openstack.org/raw/623906/ Real tokens are not logged, only a sha of the token. Reusing the printed curl commands reuquires replacing the token with a real one. :: "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" Microversions ============= * Per-HTTP-request behavior differences. * Services declare microversion support ranges in Version Discovery Document .. code-block:: bash curl -i https://compute-ca-ymq-1.vexxhost.net Nova Compute Version Document Part One ====================================== :: HTTP/1.1 200 OK Date: Wed, 18 Oct 2017 01:58:51 GMT Server: Apache/2.4.6 (CentOS) Content-Length: 399 Vary: Accept-Encoding Content-Type: application/json { "versions": [ { "status": "SUPPORTED", "version": "", "updated": "2011-01-21T11:33:21Z", "id": "v2.0", "links": [ { "href": "https://compute-ca-ymq-1.vexxhost.net/v2/", "rel": "self" } ], "min_version": "" }, Nova Compute Version Document Part Two ====================================== Note 'version' and 'min_version'. :: { "status": "CURRENT", "version": "2.53", "updated": "2013-07-23T11:33:21Z", "id": "v2.1", "links": [ { "href": "https://compute-ca-ymq-1.vexxhost.net/v2.1/", "rel": "self" } ], "min_version": "2.1" } Microversion Field Names ======================== The min and max available microversion for the given Major API version. In some services they are: * min_version * version In other services they are: * min_version * max_version Request a Microversion ====================== * Add a OpenStack-API-Version header with service-type and version:: OpenStack-API-Version: compute 2.13 * Response will contain the same header specifying which version was used:: OpenStack-API-Version: compute 2.13 * Older versions of Nova (pre Newton - or versions that do not support a max version of 2.27 or higher), use a different form:: X-OpenStack-Nova-API-Version: 2.13 * It doesn't hurt anything to send both - which is what keystoneauth does under the covers. Documentation on Nova Microversion Changes ========================================== https://docs.openstack.org/nova/latest/reference/api-microversion-history.html Simple Microversion Example with Default ======================================== Microversion 2.36 removed some calls. Before 2.36 nova had a ``/images`` API that would proxy to glance. It is removed in 2.36 and later. Making a GET call defaults to the minimum (2.1 in this case) .. code-block:: python import os_client_config client = os_client_config.make_rest_client('compute', cloud='vexxhost') client.get('/images') :: curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" [200] Date: Wed, 18 Oct 2017 02:10:49 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac x-compute-request-id: req-3e6bc318-dfd6-493b-9675-8bb1f9285cac Content-Encoding: gzip Content-Length: 2411 Content-Type: application/json Simple Microversion Example with Microversion ============================================= Sending microversion 2.36 for the same call tells Nova to use the 2.36 behavior, which results in a 404. .. code-block:: python import os_client_config client = os_client_config.make_rest_client('compute', cloud='vexxhost') client.get('/images', microversion='2.36') :: curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/db92b20496ae4fbda850a689ea9d563f/images -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}23ffd369ff0c0e7673c17bb97e6a77783d789e5e" -H "OpenStack-API-Version: compute 2.36" -H "X-OpenStack-Nova-API-Version: 2.36" [404] Date: Wed, 18 Oct 2017 02:11:00 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.36 X-OpenStack-Nova-API-Version: 2.36 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version x-openstack-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 x-compute-request-id: req-d733583e-d2c3-4ca4-bdfb-5e658e194028 Content-Length: 78 Content-Type: application/json; charset=UTF-8 {"itemNotFound": {"message": "The resource could not be found.", "code": 404}} Create Server Group =================== The server group doesn't matter - this is just to have an object to show. .. code-block:: python import os_client_config client = os_client_config.make_rest_client('compute', cloud='vexxhost') client.post( '/os-server-groups', json=dict(server_group=dict(name='test', policies=['anti-affinity']))) Example of No Microversion Content Behavior =========================================== No microversion specified. .. code-block:: python client.get('/os-server-groups') :: curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" [200] Date: Wed, 18 Oct 2017 02:18:30 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.1 X-OpenStack-Nova-API-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-85e061c0-d474-497c-bd17-96b629086169 x-compute-request-id: req-85e061c0-d474-497c-bd17-96b629086169 Content-Encoding: gzip Content-Length: 137 Content-Type: application/json {"server_groups": [{"members": [], "metadata": {}, "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "policies": ["anti-affinity"], "name": "test"}]} Example of Microversion Content Behavior ======================================== Request microversion 2.13, which adds user_id and project_id. .. code-block:: python client.get('/os-server-groups', microversion='2.13').json() :: curl -g -i -X GET https://compute-ca-ymq-1.vexxhost.net/v2.1/bbb1e479a9d04f34941470f37c7a0ecb/os-server-groups -H "User-Agent: os-client-config/1.28.1 keystoneauth1/3.2.1 python-requests/2.18.3 CPython/3.5.2" -H "X-Auth-Token: {SHA1}fe541768b55bbe23f79f97595b137fed3326166a" -H "OpenStack-API-Version: compute 2.13" -H "X-OpenStack-Nova-API-Version: 2.13" [200] Date: Wed, 18 Oct 2017 02:20:13 GMT Server: Apache/2.4.6 (CentOS) OpenStack-API-Version: compute 2.13 X-OpenStack-Nova-API-Version: 2.13 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding x-openstack-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 x-compute-request-id: req-6c71b008-185a-46c1-8762-5c2718be1741 Content-Encoding: gzip Content-Length: 195 Content-Type: application/json {"server_groups": [{"user_id": "bc2260d5b7cb40ff8270043fd7af3a08", "policies": ["anti-affinity"], "name": "test", "members": [], "project_id": "bbb1e479a9d04f34941470f37c7a0ecb", "id": "f88c7aef-b34a-4b0e-b0c7-9076eab88de2", "metadata": {}}]}