Date: Tue, 9 May 2017 08:11:51 -0400 Subject: Add shade: Everything you ever wanted as a cloud user --- src/talks/everything-you-ever-wanted.hbs | 1335 ++++++++++++++++++++++++++++++ 1 file changed, 1335 insertions(+) create mode 100644 src/talks/everything-you-ever-wanted.hbs (limited to 'src/talks') diff --git a/src/talks/everything-you-ever-wanted.hbs b/src/talks/everything-you-ever-wanted.hbs new file mode 100644 index 0000000..a5c79fc --- /dev/null +++ b/src/talks/everything-you-ever-wanted.hbs @@ -0,0 +1,1335 @@ + + + + + + + shade: Everything You Ever Wanted As a Cloud User + + + +

Who am I?

Office of Technology

Zuul

Ansible

+ +

Who am I?

Technical Committee

Developer Infrastructure Core Team

+ +

What are we going to talk about?

What
Why
Configuration
Basics
Advanced things +
- Caching
- Task Management
+

+ +

shade: a Python library to wrap business logic around + OpenStack resources and operations

+ +

Design Principles

Expose a single API that works on all clouds
Hide all vendor or deployer differences
Support multi-cloud (write once, run anywhere)
Simple to use (sane defaults)
No plugins *
Efficient at scale
API always backwards compatible

+ +

Current Status

https://git.openstack.org/cgit/openstack-infra/shade

https://pypi.python.org/pypi/shade

Used in Ansible
Used in Infra Nodepool
Official OpenStack Project
Backend conversion to pure REST almost done

+ +

Why?

+ +

+ Brand experts insist that success comes from promoting your unique + attributes, but in practice differentiation is less profitable than + consolidation. +

+ +

OpenStack Leaks Abstractions

+ +

OpenStack Breaks APIs

+ +

Basic concepts are needlessly complex

+ +

Client libraries are really for server-server communication

+ +

Infra solved these problems

Infra runs across 14 clouds at massive scale (20k servers per day)

Why not share what we've learned with other people?

+ +

simplicity

This is what using a cloud should look like


+cloud = openstack_cloud('vexxhost')
+image = cloud.create_image(
+    'image-name', filename='image-filename.qcow2', wait=True)
+flavor = cloud.get_flavor_by_ram(512)
+cloud.create_server(
+    'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)
+

+ +

existence of shade is a bug

+ +

existence of shade is a feature

+ +

Using shade

Step One: Configuration

+ +

os-client-config

http://git.openstack.org/cgit/openstack/os-client-config

A library to handle config information for openstack clients
Tracks differences in vendors that can't be discovered
Also used for ansible, python-openstackclient and openstacksdk
Reads clouds.yaml config file, environment vars and argparse

+ +

clouds.yaml - simple


+clouds:
+  my-citycloud:
+    profile: citycloud
+    auth:
+      username: mordred
+      password: XXXXXXXXXXXXX
+      project_id: 65222a4d09ea4c68934fa1028c77f394
+      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
+      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38
+

+ +

clouds.yaml - mildly complex


+  my-vexxhost:
+    volume_api_version: 3
+    profile: vexxhost
+    auth:
+      user_domain_id: default
+      project_domain_id: default
+      project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
+      username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1
+      password: XXXXXXXXXXXXX
+    regions:
+    - ca-ymq-1
+    image_endpoint_override: https://image-ca-ymq-1.vexxhost.net/v2
+

+ +

clouds.yaml - complex


+  my-internap:
+    auth:
+      auth_url: https://identity.api.cloud.iweb.com
+      username: api-55f9a00fb2619
+      project_name: inap-17037
+      password: XXXXXXXXXXXXX
+    identity_api_version: 3
+    floating_ip_source: None
+    regions:
+    - name: ams01
+      values:
+        networks:
+        - name: inap-17037-WAN1654
+          routes_externally: true
+          default_interface: true
+        - name: inap-17037-LAN3631
+          routes_externally: false
+

+ +

Environment Variables

os-client-config also processes OS_ environment variables
If present, they go into a cloud named "envvars"

+ +

auth_type

keystone has pluggable authentication

+ +

auth dict contents vary based on plugin
Defaults to 'password' which autodetects based on parameters
Other options: admin_token, v3oidcpassword, v3oidcauthcode
This is where I said "*" on "no plugins" earlier

+ +

auth_type - not part of auth_dict


+  my-vexxhost:
+    profile: vexxhost
+    auth_type: v3password
+    auth:
+      user_domain_id: default
+      project_domain_id: default
+      project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
+      username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1
+      password: XXXXXXXXXXXXX
+    regions:
+    - ca-ymq-1
+

+ +

python-openstackclient

+openstack --os-cloud=my-vexxhost servers list
+

+ Or +

+export OS_CLOUD=my-vexxhost
+openstack servers list
+

+ +

shade inventory

Code behind ansible OpenStack dynamic inventory plugin
All resources in all of your clouds

+mordred@camelot:~$ shade-inventory --list --yaml
+


+- accessIPv4: 199.204.45.50
+  accessIPv6: 2604:e100:1:0:f816:3eff:feb5:ce98
+  addresses:
+    public:
+    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:b5:ce:98
+      OS-EXT-IPS:type: fixed
+      addr: 2604:e100:1:0:f816:3eff:feb5:ce98
+      version: 6
+    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:b5:ce:98
+      OS-EXT-IPS:type: fixed
+      addr: 199.204.45.50
+      version: 4
+  adminPass: null
+  created: '2017-04-02T17:04:49Z'
+  disk_config: AUTO
+  flavor:
+    id: bbcb7eb5-5c8d-498f-9d7e-307c575d3566
+    name: v1-standard-1
+  has_config_drive: false
+  host_id: 5b303fdae29a6b2c39c90a2a1c6dc1191297840d8c395e09c4e319b7
+  id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
+  image:
+    id: dd33a4c8-5a86-40ec-9fe0-8578d6a6dc10
+    name: Ubuntu 16.04.1 LTS [2017-03-03]
+  interface_ip: 199.204.45.50
+  key_name: mordred
+  launched_at: '2017-04-02T17:05:02.000000'
+  location:
+    cloud: my-vexxhost
+    project:
+      domain_id: default
+      domain_name: null
+      id: db92b20496ae4fbda850a689ea9d563f
+      name: d8af8a8f-a573-48e6-898a-af333b970a2d
+    region_name: ca-ymq-1
+    zone: ca-ymq-2
+  metadata: {}
+  name: irc-bouncer
+  networks:
+    public:
+    - 2604:e100:1:0:f816:3eff:feb5:ce98
+    - 199.204.45.50
+  power_state: 1
+  private_v4: ''
+  progress: 0
+  properties: {}
+  public_v4: 199.204.45.50
+  public_v6: 2604:e100:1:0:f816:3eff:feb5:ce98
+  security_groups:
+  - description: default
+    id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
+    location:
+      cloud: my-vexxhost
+      project:
+        domain_id: default
+        domain_name: null
+        id: db92b20496ae4fbda850a689ea9d563f
+        name: d8af8a8f-a573-48e6-898a-af333b970a2d
+      region_name: ca-ymq-1
+      zone: null
+    name: default
+    properties: {}
+    security_group_rules:
+    - direction: ingress
+      ethertype: IPv4
+      id: 2172daaa-e402-4cf0-b353-1a7cb7dae184
+      location:
+        cloud: my-vexxhost
+        project:
+          domain_id: default
+          domain_name: null
+          id: db92b20496ae4fbda850a689ea9d563f
+          name: d8af8a8f-a573-48e6-898a-af333b970a2d
+        region_name: ca-ymq-1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group: {}
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: 0.0.0.0/0
+      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
+    - direction: ingress
+      ethertype: IPv4
+      id: 47fb6b4c-2f9b-4279-a77c-a645f7fc964f
+      location:
+        cloud: my-vexxhost
+        project:
+          domain_id: default
+          domain_name: null
+          id: db92b20496ae4fbda850a689ea9d563f
+          name: d8af8a8f-a573-48e6-898a-af333b970a2d
+        region_name: ca-ymq-1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: db92b20496ae4fbda850a689ea9d563f
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
+    - direction: ingress
+      ethertype: IPv4
+      id: 8287f9be-614b-4145-9f00-62fa9c856362
+      location:
+        cloud: my-vexxhost
+        project:
+          domain_id: default
+          domain_name: null
+          id: db92b20496ae4fbda850a689ea9d563f
+          name: d8af8a8f-a573-48e6-898a-af333b970a2d
+        region_name: ca-ymq-1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: db92b20496ae4fbda850a689ea9d563f
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
+    - direction: ingress
+      ethertype: IPv4
+      id: fdd91863-c19b-4b4b-a1bc-822cb8b0adf7
+      location:
+        cloud: my-vexxhost
+        project:
+          domain_id: default
+          domain_name: null
+          id: db92b20496ae4fbda850a689ea9d563f
+          name: d8af8a8f-a573-48e6-898a-af333b970a2d
+        region_name: ca-ymq-1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group: {}
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: 0.0.0.0/0
+      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
+  status: ACTIVE
+  task_state: null
+  terminated_at: null
+  updated: '2017-04-02T17:05:02Z'
+  user_id: e9b21dc437d149858faee0898fb08e92
+  vm_state: active
+  volumes:
+  - attachments:
+    - attached_at: '2017-04-11T19:11:09.000000'
+      attachment_id: 60f81620-4714-438b-b481-cd8fd7a4dd3e
+      device: /dev/vdc
+      host_name: null
+      id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
+      server_id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
+      volume_id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
+    can_multiattach: false
+    consistencygroup_id: null
+    created_at: '2017-04-11T19:10:20.000000'
+    description: ''
+    device: /dev/vdc
+    host: null
+    id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
+    is_bootable: false
+    is_encrypted: false
+    location:
+      cloud: my-vexxhost
+      project:
+        domain_id: default
+        domain_name: null
+        id: db92b20496ae4fbda850a689ea9d563f
+        name: d8af8a8f-a573-48e6-898a-af333b970a2d
+      region_name: ca-ymq-1
+      zone: ca-ymq-2
+    metadata:
+      attached_mode: rw
+      readonly: 'False'
+    migration_status: null
+    name: foo
+    properties:
+      volume_image_metadata: {}
+    replication_driver: null
+    replication_extended_status: null
+    replication_status: disabled
+    size: 1
+    snapshot_id: null
+    source_volume_id: null
+    status: in-use
+    updated_at: '2017-04-11T19:11:09.000000'
+    volume_type: null
+  - attachments:
+    - attached_at: '2017-04-02T17:06:27.000000'
+      attachment_id: 4f24b276-ea5c-4a79-824f-434183c3c518
+      device: /dev/vdb
+      host_name: null
+      id: 9b6110df-cd97-4e37-9596-af371653a791
+      server_id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
+      volume_id: 9b6110df-cd97-4e37-9596-af371653a791
+    can_multiattach: false
+    consistencygroup_id: null
+    created_at: '2017-04-02T16:34:41.000000'
+    description: ''
+    device: /dev/vdb
+    host: null
+    id: 9b6110df-cd97-4e37-9596-af371653a791
+    is_bootable: true
+    is_encrypted: false
+    location:
+      cloud: my-vexxhost
+      project:
+        domain_id: default
+        domain_name: null
+        id: db92b20496ae4fbda850a689ea9d563f
+        name: d8af8a8f-a573-48e6-898a-af333b970a2d
+      region_name: ca-ymq-1
+      zone: ca-ymq-2
+    metadata:
+      attached_mode: rw
+      readonly: 'False'
+    migration_status: null
+    name: mordred-irc-save
+    properties:
+      volume_image_metadata:
+        base_image_ref: 69c99b45-cd53-49de-afdc-f24789eb8f83
+        checksum: dddc88434b12b5b61b6c7a3a5a880c35
+        clean_attempts: '7'
+        container_format: bare
+        disk_format: raw
+        hw_rng_model: virtio
+        image_id: 1659da1f-7650-4b91-b316-c3d69623e3fc
+        image_location: snapshot
+        image_name: mordred-irc-save
+        image_state: available
+        image_type: snapshot
+        instance_uuid: 811c5197-dba7-4d3a-a3f6-68ca5328b9a7
+        libvirt_use_agent: 'True'
+        min_disk: '40'
+        min_ram: '0'
+        network_allocated: 'True'
+        os_type: linux
+        owner_id: db92b20496ae4fbda850a689ea9d563f
+        size: '42949672960'
+        user_id: e9b21dc437d149858faee0898fb08e92
+    replication_driver: null
+    replication_extended_status: null
+    replication_status: disabled
+    size: 40
+    snapshot_id: null
+    source_volume_id: null
+    status: in-use
+    updated_at: '2017-04-02T17:06:27.000000'
+    volume_type: null
+- accessIPv4: 89.40.216.229
+  accessIPv6: ''
+  addresses:
+    private:
+    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:29:6d:20
+      OS-EXT-IPS:type: fixed
+      addr: 10.4.0.22
+      version: 4
+    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:29:6d:20
+      OS-EXT-IPS:type: floating
+      addr: 89.40.216.229
+      version: 4
+  adminPass: null
+  created: '2017-05-08T18:48:34Z'
+  disk_config: MANUAL
+  flavor:
+    id: 10973fe6-81ba-480c-bc36-bec93ee03d5c
+    name: 4C-4GB-100GB
+  has_config_drive: false
+  host_id: 8ed21447e12616c0c24f6e246f0ec423d6be1532716869a3617fbe80
+  id: a72fd3ea-9357-482d-b33a-fe61d7c37bfc
+  image:
+    id: 95e4c449-8abf-486e-97d9-dc3f82417d2d
+    name: Ubuntu 16.04 Xenial Xerus
+  interface_ip: 89.40.216.229
+  key_name: null
+  launched_at: '2017-05-08T18:48:39.000000'
+  location:
+    cloud: my-citycloud
+    project:
+      domain_id: d0919bd5e8d74e49adf0e145807ffc38
+      domain_name: null
+      id: 65222a4d09ea4c68934fa1028c77f394
+      name: null
+    region_name: Buf1
+    zone: nova
+  metadata: {}
+  name: my-server
+  networks:
+    private:
+    - 10.4.0.22
+    - 89.40.216.229
+  power_state: 1
+  private_v4: 10.4.0.22
+  progress: 0
+  properties: {}
+  public_v4: 89.40.216.229
+  public_v6: ''
+  security_groups:
+  - description: Default security group
+    id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
+    location:
+      cloud: my-citycloud
+      project:
+        domain_id: d0919bd5e8d74e49adf0e145807ffc38
+        domain_name: null
+        id: 65222a4d09ea4c68934fa1028c77f394
+        name: null
+      region_name: Buf1
+      zone: null
+    name: default
+    properties: {}
+    security_group_rules:
+    - direction: ingress
+      ethertype: IPv4
+      id: 67cbd716-ace1-4822-93e3-1a14a2763efa
+      location:
+        cloud: my-citycloud
+        project:
+          domain_id: d0919bd5e8d74e49adf0e145807ffc38
+          domain_name: null
+          id: 65222a4d09ea4c68934fa1028c77f394
+          name: null
+        region_name: Buf1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: 65222a4d09ea4c68934fa1028c77f394
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
+    - direction: ingress
+      ethertype: IPv4
+      id: f5e80c4c-fe7a-4bfc-a8c0-0105b2c272d5
+      location:
+        cloud: my-citycloud
+        project:
+          domain_id: d0919bd5e8d74e49adf0e145807ffc38
+          domain_name: null
+          id: 65222a4d09ea4c68934fa1028c77f394
+          name: null
+        region_name: Buf1
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: 65222a4d09ea4c68934fa1028c77f394
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
+  status: ACTIVE
+  task_state: null
+  terminated_at: null
+  updated: '2017-05-08T18:48:39Z'
+  user_id: c17534835f8f42bf98fc367e0bf35e09
+  vm_state: active
+  volumes: []
+- accessIPv4: 173.231.180.251
+  accessIPv6: ''
+  addresses:
+    inap-17037-WAN1654:
+    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:6b:97:9c
+      OS-EXT-IPS:type: fixed
+      addr: 173.231.180.251
+      version: 4
+  adminPass: null
+  created: '2017-05-08T18:48:56Z'
+  disk_config: null
+  flavor:
+    id: A1.4
+    name: A1.4
+  has_config_drive: true
+  host_id: 6d0c22f3176c3bcbb78a418a93809ba6bfa16267a4c38bd614a892d9
+  id: ba864796-b56a-4154-b220-e1b96daf39c0
+  image:
+    id: a6fefc9b-c208-4cd0-acc3-2ab09a38054b
+    name: Ubuntu 16.04 LTS (Xenial Xerus)
+  interface_ip: 173.231.180.251
+  key_name: null
+  launched_at: null
+  location:
+    cloud: my-internap
+    project:
+      domain_id: null
+      domain_name: null
+      id: 760e6c137b3840d78472d313dfa3df45
+      name: inap-17037
+    region_name: ams01
+    zone: null
+  metadata: {}
+  name: my-server
+  networks:
+    inap-17037-WAN1654:
+    - 173.231.180.251
+  power_state: 1
+  private_v4: ''
+  progress: 0
+  properties: {}
+  public_v4: 173.231.180.251
+  public_v6: ''
+  security_groups:
+  - description: default
+    id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+    location:
+      cloud: my-internap
+      project:
+        domain_id: null
+        domain_name: null
+        id: 760e6c137b3840d78472d313dfa3df45
+        name: inap-17037
+      region_name: ams01
+      zone: null
+    name: default
+    properties: {}
+    security_group_rules:
+    - direction: ingress
+      ethertype: IPv4
+      id: 24b2ce3a-bf29-44fb-849f-f958e3efb736
+      location:
+        cloud: my-internap
+        project:
+          domain_id: null
+          domain_name: null
+          id: 760e6c137b3840d78472d313dfa3df45
+          name: inap-17037
+        region_name: ams01
+        zone: null
+      port_range_max: 65535
+      port_range_min: 1
+      properties:
+        group: {}
+      protocol: tcp
+      remote_group_id: null
+      remote_ip_prefix: 0.0.0.0/0
+      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+    - direction: ingress
+      ethertype: IPv4
+      id: 374aa520-c101-4de3-be88-f0025396bdb7
+      location:
+        cloud: my-internap
+        project:
+          domain_id: null
+          domain_name: null
+          id: 760e6c137b3840d78472d313dfa3df45
+          name: inap-17037
+        region_name: ams01
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: 760e6c137b3840d78472d313dfa3df45
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+    - direction: ingress
+      ethertype: IPv4
+      id: 376b67c2-284b-4a81-a989-e7b8f6d44a07
+      location:
+        cloud: my-internap
+        project:
+          domain_id: null
+          domain_name: null
+          id: 760e6c137b3840d78472d313dfa3df45
+          name: inap-17037
+        region_name: ams01
+        zone: null
+      port_range_max: 65535
+      port_range_min: 1
+      properties:
+        group: {}
+      protocol: udp
+      remote_group_id: null
+      remote_ip_prefix: 0.0.0.0/0
+      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+    - direction: ingress
+      ethertype: IPv4
+      id: 481f9402-753d-4286-8079-be45a414023c
+      location:
+        cloud: my-internap
+        project:
+          domain_id: null
+          domain_name: null
+          id: 760e6c137b3840d78472d313dfa3df45
+          name: inap-17037
+        region_name: ams01
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group: {}
+      protocol: icmp
+      remote_group_id: null
+      remote_ip_prefix: 0.0.0.0/0
+      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+    - direction: ingress
+      ethertype: IPv4
+      id: 832aaf3b-b0fb-4103-b2fa-def955077667
+      location:
+        cloud: my-internap
+        project:
+          domain_id: null
+          domain_name: null
+          id: 760e6c137b3840d78472d313dfa3df45
+          name: inap-17037
+        region_name: ams01
+        zone: null
+      port_range_max: null
+      port_range_min: null
+      properties:
+        group:
+          name: default
+          tenant_id: 760e6c137b3840d78472d313dfa3df45
+      protocol: null
+      remote_group_id: null
+      remote_ip_prefix: null
+      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
+  status: ACTIVE
+  task_state: null
+  terminated_at: null
+  updated: '2017-05-08T18:49:05Z'
+  user_id: c18bd37c27d14b1ba4c424e6d42f2ccd
+  vm_state: active
+  volumes: []
+

+ +

Cloud-Region

Each OpenStackCloud object represents one region of one cloud

+ +

Simplest Cloud Construction

Works if you only have one cloud

+ +


+import shade
+cloud = shade.openstack_cloud()
+

+ +

Simplest Cloud Construction - More than one


+clients:
+  default:
+    cloud: vexxhost
+

+ +


+import shade
+cloud = shade.openstack_cloud()
+

+ +

Simple Cloud Construction


+import shade
+cloud = shade.openstack_cloud(cloud='ustack', region_name='bj1')
+

+ +

Complex Cloud Construction


+import os_client_config
+import shade
+
+config = os_client_config.OpenStackConfig()
+cloud_config = config.get_one_cloud(
+    cloud='ustack', region_name='bj1',
+    argparse=my_args, **other_arguments)
+cloud = shade.OpenStackCloud(cloud_config=cloud_config)
+

+ +

logging

Python logging
shade.simple_logging() helper function
Turns off annoying warnings
debug=True - turn on debug logging and request_id logging
http_debug=True - debug=True + http request tracing

+ +

A note on Exceptions

All (remaining) python*client exceptions are hidden and re-raised
Hiding exceptions is evil and I'm a bad person
Exceptions are part of the interface
Can't hide vendor choice if shade user catches underlying exception
See also: Restification later

+ +

Exceptions

All Exceptions are subclasses of OpenStackCloudException
Direct REST calls throw OpenStackCloudHTTPError
OpenStackCloudHTTPError subclasses OpenStackCloudException + and requests.exceptions.HTTPError
OpenStackCloudURINotFound for 404
OpenStackCloudBadRequest for 400

+ +

Putting it all together


+import shade
+
+shade.simple_logging(debug=True)
+cloud = shade.openstack_cloud(cloud='vexxhost')
+image = cloud.create_image(
+    'ubuntu-trusty', filename='ubuntu-trusty.qcow2', wait=True)
+flavor = cloud.get_flavor_by_ram(512)
+cloud.create_server(
+    'my-server', image=image['id'], flavor=flavor['id'],
+     wait=True, auto_ip=True)
+

+ +

Problem: Image API version

+ v1 PUT: Catalyst, Datacentred, Internap +
+ v2 PUT: Auro, City Cloud, Dreamhost, Elastx, Enter Cloud Suite, OVH, RunAbove, Vexxhost, Ultimum, UnitedStack +
+ v2 Tasks: Switch Engines, Rackspace +

+ +

Image Upload Code


+cloud = shade.openstack_cloud(cloud='vexxhost', region_name='ca-ymq-1')
+cloud.create_image('my-image', filename='my-image.qcow2', wait=True)
+

+ +

Image Upload PUT v2 Log


+Calculating hashes for my-image.qcow2
+Image file my-image.vhd md5:b28fa44077b94ba635842a73eecc1d8c sha256:9a3da71e37ef1579657c4a77ff2caf936dd9be8361d1a02595cc116f0ffd328d
+Manager my-vexxhost:ca-ymq-1 running task image.GET.images
+Manager my-vexxhost:ca-ymq-1 ran task image.GET.images in 1.38961982727s
+GET call to image for https://image-ca-ymq-1.vexxhost.net/v2/images used request id req-2bbc8c10-6c4c-4aa3-8bb7-6ba425cb0ea7
+Manager my-vexxhost:ca-ymq-1 running task image.GET.images
+Manager my-vexxhost:ca-ymq-1 ran task image.GET.images in 0.245140790939s
+GET call to image for https://image-ca-ymq-1.vexxhost.net/v2/images?marker=5ebb74ba-c9c5-4006-8c77-ceebf9833428 used request id req-da204dc3-eb07-460d-b9f5-74b6ea5fe944
+Manager my-vexxhost:ca-ymq-1 running task image.POST.images
+Manager my-vexxhost:ca-ymq-1 ran task image.POST.images in 0.114197969437s
+POST call to image for https://image-ca-ymq-1.vexxhost.net/v2/images used request id req-02a5fd04-ca9a-4a71-bfe1-4ff80e49cdea returning object 28d3e4b2-4607-4778-84bd-e84ef217d21c
+Manager my-vexxhost:ca-ymq-1 running task image.PUT.images.file
+Manager my-vexxhost:ca-ymq-1 ran task image.PUT.images.file in 2.10132312775s
+PUT call to image for https://image-ca-ymq-1.vexxhost.net/v2/images/28d3e4b2-4607-4778-84bd-e84ef217d21c/file used request id req-71e3ed5e-77dc-4062-9ba8-13d495713e13
+

+ +

Image Tasks


+cloud = shade.openstack_cloud(cloud='rax', region_name='DFW')
+cloud.create_image('my-image', filename='my-image.vhd', wait=True)
+

+ + +

Image Upload Tasks Log


+Calculating hashes for my-image.vhd
+Image file my-image.vhd md5:b28fa44077b94ba635842a73eecc1d8c sha256:9a3da71e37ef1579657c4a77ff2caf936dd9be8361d1a02595cc116f0ffd328d
+Manager rax:DFW running task image.GET.discovery
+Manager rax:DFW ran task image.GET.discovery in 1.65571808815s
+Glance version discovery failed, assuming endpoint in the catalog is already versioned. (404) Client Error: Not Found for https://dfw.images.api.rackspacecloud.com/v2/
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.637840032578s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/images used request id req-b716bd86-121f-4cdc-b386-2f50546e8d88
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.491132974625s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=e389ab51-1327-488a-867c-4d7c14e6c916 used request id req-90746cbf-610a-4c3d-aab8-f81e673b315a
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.371850967407s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=3eba4fbb-51da-4233-b699-8a4030561add used request id req-00937f32-f573-4ac3-86d5-0aeada7432d1
+Manager rax:DFW running task object-store.GET.info
+Manager rax:DFW ran task object-store.GET.info in 0.335763931274s
+Manager rax:DFW running task object-store.HEAD.images
+Manager rax:DFW ran task object-store.HEAD.images in 0.393274784088s
+Manager rax:DFW running task object-store.HEAD.images
+Manager rax:DFW ran task object-store.HEAD.images in 3.68135595322s
+swift stale check, no object: images/my-image
+swift uploading my-image.vhd to images/my-image
+Manager rax:DFW running task object-store.PUT.images
+Manager rax:DFW ran task object-store.PUT.images in 3.57636904716s
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.616628885269s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/images used request id req-ee7091ae-88a2-4cdb-ac8e-35159ef4cda0
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.527029037476s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=e389ab51-1327-488a-867c-4d7c14e6c916 used request id req-d2caefd2-d69f-46d2-86c1-a062f039ce74
+Manager rax:DFW running task image.GET.images
+Manager rax:DFW ran task image.GET.images in 0.317733049393s
+GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=3eba4fbb-51da-4233-b699-8a4030561add used request id req-56411197-65ce-4ac9-847d-0a7e7041766d
+Manager rax:DFW running task image.POST.tasks
+Manager rax:DFW ran task image.POST.tasks in 0.796432971954s
+POST call to image for https://dfw.images.api.rackspacecloud.com/v2/tasks used request id req-edc7842f-5fca-4a08-96b3-55a67c840d16 returning object 45df1ffa-3837-4fbc-a8fb-f213f845b18c
+

+ +

Problem: Version Discovery

OpenStack Services have version discovery documents
Deployers register versioned endpoints in catalog
Some services (cinder, mistral) have versioned service types (volumev2, workflowv2)
Users can't (sanely) do version discovery
shade fixes this for glance and cinder today

+ +

Version Discovery Solutions

shade will fix for everything by end of the month
OpenStack Service Types Authority
API-WG specs on Full Service Type and Version Discovery Process
Work with other languages to implement API-WG specs
User requests:
- image version 2
- workflow LATEST
- compute
- volume versions 2 or 3
Later today we'll be discussing moving this forward (Room 102 at 4:40)

+ +

Problem: Networking choices

Cloud has externally routable IP from neutron (OVH)
Cloud has externally routable IP neutron AND supports optional private tenant networks (vexxhost)
Cloud has private tenant network provided by neutron and requires floating IP (citycloud)
Cloud has private tenant network provided by nova-network and requires floating-ip for external routing (auro)
Cloud has externally routable IP from neutron but no neutron APIs (Rackspace)

+ +

The Floating IP Case


+import shade
+
+shade.simple_logging(debug=True)
+cloud = shade.openstack_cloud(cloud='my-citycloud')
+cloud.create_server(
+    'my-server',
+    image='Ubuntu 16.04 Xenial Xerus',
+    flavor=dict(id='0dab10b5-42a2-438e-be7b-505741a7ffcc'),
+    wait=True, auto_ip=True)
+

+ +

The Floating IP Case Log


+Manager my-citycloud:Buf1 running task network.GET.networks
+Manager my-citycloud:Buf1 ran task network.GET.networks in 1.29657912254s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/networks.json used request id req-a65496e6-984f-4323-98d5-3adb339875c6
+Manager my-citycloud:Buf1 running task network.GET.subnets
+Manager my-citycloud:Buf1 ran task network.GET.subnets in 0.713509082794s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/subnets.json used request id req-66ccbfce-ceb6-4e0a-a969-0eaf62605d92
+Manager my-citycloud:Buf1 running task image.GET.discovery
+Manager my-citycloud:Buf1 ran task image.GET.discovery in 0.474808931351s
+Manager my-citycloud:Buf1 running task image.GET.images
+Manager my-citycloud:Buf1 ran task image.GET.images in 0.714179039001s
+GET call to image for https://buf1.citycloud.com:9292/v2/images used request id req-9a7756c3-2ee4-416d-a6ad-804027330e5f
+Manager my-citycloud:Buf1 running task ServerCreate
+Manager my-citycloud:Buf1 ran task ServerCreate in 3.38534998894s
+Manager my-citycloud:Buf1 running task ServerList
+Manager my-citycloud:Buf1 ran task ServerList in 0.959898948669s
+Waiting 2.0 seconds
+Manager my-citycloud:Buf1 running task ServerList
+Manager my-citycloud:Buf1 ran task ServerList in 1.21962594986s
+Manager my-citycloud:Buf1 running task network.GET.ports
+Manager my-citycloud:Buf1 ran task network.GET.ports in 0.287118911743s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-fd5acf95-21bd-40bb-ad08-e15ac7107874
+Manager my-citycloud:Buf1 running task network.GET.floatingips
+Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.228282928467s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json?port_id=af554254-ac9f-4440-b93a-e4ec64783758 used request id req-ed52ca15-9924-488e-a3c0-3a805348eca3
+Manager my-citycloud:Buf1 running task network.GET.ports
+Manager my-citycloud:Buf1 ran task network.GET.ports in 0.200226068497s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-f808b9e0-d136-4bd7-a51b-b08d3c6eddb3
+Manager my-citycloud:Buf1 running task network.GET.floatingips
+Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.140635967255s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json used request id req-880919b7-4af3-42a3-8e07-4b80a6ae5bac
+Manager my-citycloud:Buf1 running task network.GET.ports
+Manager my-citycloud:Buf1 ran task network.GET.ports in 0.317744970322s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-af4bb447-e157-44c1-a727-2cecfb595c0e
+Manager my-citycloud:Buf1 running task network.PUT.floatingips
+Manager my-citycloud:Buf1 ran task network.PUT.floatingips in 0.425180912018s
+PUT call to network for https://buf1.citycloud.com:9696/v2.0/floatingips/e69179dc-a904-4c9a-a4c9-891e2ecb984c.json used request id req-2241d8b6-bf50-41c1-9129-c8cfca2c73bc returning object e69179dc-a904-4c9a-a4c9-891e2ecb984c
+Manager my-citycloud:Buf1 running task ServerList
+Manager my-citycloud:Buf1 ran task ServerList in 1.19975018501s
+Manager my-citycloud:Buf1 running task network.GET.ports
+Manager my-citycloud:Buf1 ran task network.GET.ports in 0.184152126312s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-9f49b751-d741-45ff-a5fc-ca750826df63
+Manager my-citycloud:Buf1 running task network.GET.floatingips
+Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.21661400795s
+GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json?port_id=af554254-ac9f-4440-b93a-e4ec64783758 used request id req-81e231c6-a4d5-466a-bffd-0af6f4dc4be8
+

+ +

The Direct Attached IP Case


+import shade
+
+shade.simple_logging(debug=True)
+cloud = shade.openstack_cloud(cloud='internap', region_name='ams01')
+cloud.create_server(
+    'my-server',
+    image='Ubuntu 16.04 LTS (Xenial Xerus)',
+    flavor=dict(id='A1.4'),
+    wait=True, auto_ip=True)
+

+ +

Direct Attached IP Log


+Manager my-internap:ams01 running task network.GET.networks
+Manager my-internap:ams01 ran task network.GET.networks in 0.783267021179s
+GET call to network for https://network.api.ams01.cloud.iweb.com/v2.0/networks.json used request id req-807686be-238a-46e1-b0e5-5dfa31949316
+Manager my-internap:ams01 running task network.GET.subnets
+Manager my-internap:ams01 ran task network.GET.subnets in 0.155231952667s
+GET call to network for https://network.api.ams01.cloud.iweb.com/v2.0/subnets.json used request id req-9e71772d-70dc-4dc9-8da4-734639a10dd4
+Manager my-internap:ams01 running task image.GET.discovery
+Manager my-internap:ams01 ran task image.GET.discovery in 0.589816093445s
+Manager my-internap:ams01 running task image.GET.images
+Manager my-internap:ams01 ran task image.GET.images in 0.568889856339s
+GET call to image for https://image.api.ams01.cloud.iweb.com/v2/images used request id req-b0e25143-a581-4c6a-baab-2376626e529a
+Manager my-internap:ams01 running task ServerCreate
+Manager my-internap:ams01 ran task ServerCreate in 1.55480098724s
+Manager my-internap:ams01 running task ServerList
+Manager my-internap:ams01 ran task ServerList in 0.316590070724s
+Waiting 2.0 seconds
+Manager my-internap:ams01 running task ServerList
+Manager my-internap:ams01 ran task ServerList in 0.317390918732s
+

+ +

Problem: Finding an Image

Ubuntu 16.04.1 LTS [2017-03-03]
Ubuntu 16.04 Xenial Xerus
Ubuntu 16.04 LTS (Xenial Xerus)
Easy for a human - hard for a computer

+ +

Solution: Sorry - no good solution yet

Always Upload your Own Images
Discussed in Glance Deployer Session Yesterday
Next step - collecting full set of User Stories
Hopefully we'll define common metadata
Hopefully we'll define "correct" image lifecycle

+ +

Problem: Crazy Dependencies

shade started life depending on python-*client
Giant dependency chain
Users should ALWAYS use the latest shade
Conflicting depends makes this hard

+ +

Solution: 'RESTification'

Have been hard at work moving to REST
Done with heat, magnum, swift, glance, trove
cinder and neutron done but unreleased
nova, ironic, designate left
Thanks Rosario and Slaweq!!!

BTW - The code is easier with REST

+ +

Advanced Things

shade should DTRT 95% of the time
Sometimes, you need to be extra advanced
We want to suppor that too

+ +

Task Manager

+Manager zetta ran task ServerGet in 0.726951122284s
+

Every API operation is a Task() that is run by a TaskManager()
Default shade TaskManager immediately executes the call
nodepool passes in a threaded TaskManager to manage API throttling


+cloud = shade.OpenStackCloud(
+    cloud_config=cloud_config,
+    manager=MyTaskManager())
+

+ +

list vs. get

get_server is a wrapper around list_servers
*sometimes* we can push filter conditions to the cloud
Actually done in SUPPORT of scaling

+ +

Caching and Rate Limiting

API Operations can be expensive
Support grew from nodepool's needs
shade uses dogpile.cache - default to NullCache

+ +

Cache Config

clouds.yaml


+cache:
+  class: dogpile.cache.dbm
+  expiration_time: 3600
+  arguments:
+    filename: /home/mordred/.cache/openstack/shade.dbm
+  expiration:
+      server: 5
+

+ +

Caching Special Case

servers, ports and floating-ips have custome caching
Specifically done for nodepool
After RESTification, will migrate them to dogpile
Two levels - caching and rate-limiting

+ +

What if you're not a Python developer?

Yes, we love you too

+ +

ansible


+- os_image:
+    cloud: vexxhost
+    name: centos-7
+    file: centos-7.qcow2
+    wait: true
+- os_server:
+    cloud: vexxhost
+    name: my-server
+    flavor_ram: 1024
+    image: centos-7
+    wait: true
+

+ +

ansible

Add my keypair to 30 Regions in 13 clouds


+- os_keypair:
+    cloud: "{{ item.cloud }}"
+    region_name: "{{ item.region_name }}"
+    name: mordred
+    public_key_file: ~/.ssh/id_rsa.pub
+    with-items:
+    - {cloud: vexxhost, region_name: ca-ymq-1}
+    - {cloud: ovh, region_name: SBG1}
+    - {cloud: ovh, region_name: BHS1}
+    - {cloud: ovh, region_name: GRA1}
+    - {cloud: citycloud, region_name: Buf1}
+    - {cloud: citycloud, region_name: La1}
+    - {cloud: citycloud, region_name: Fra1}
+    - {cloud: citycloud, region_name: Lon1}
+    - {cloud: citycloud, region_name: Sto2}
+    - {cloud: citycloud, region_name: Kna1}
+    - {cloud: internap, region_name: ams01}
+    - {cloud: internap, region_name: da01}
+    - {cloud: internap, region_name: nyj01}
+    - {cloud: internap, region_name: sin01}
+    - {cloud: internap, region_name: sjc01}
+    - {cloud: infracloud, region_name: vanilla}
+    - {cloud: infracloud, region_name: chocolate}
+    - {cloud: fuga, region_name: cystack}
+    - {cloud: datacentred, region_name: sal01}
+    - {cloud: clouda, region_name: regionOne}
+    - {cloud: auro, region_name: van1}
+    - {cloud: ustack, region_name: bj1}
+    - {cloud: zetta, region_name: no-osl1}
+    - {cloud: kiss, region_name: region1}
+    - {cloud: rax, region_name: IAD}
+    - {cloud: rax, region_name: ORD}
+    - {cloud: rax, region_name: SYD}
+    - {cloud: rax, region_name: LON}
+    - {cloud: rax, region_name: DFW}
+    - {cloud: rax, region_name: HKG}
+

+ +

The Future: Pushing Back Into OpenStack APIs

Discussions later today
Service Discovery
Version Discovery
Vendors Publishing Vendor Profile Documents
Capabilities Discovery
Connecting those to Interop Working Group

+ +

Thank You!

+ + + + -- cgit v1.2.3