Who am I?

Office of Technology

Zuul

Ansible

Who am I?

Technical Committee

Developer Infrastructure Core Team

What are we going to talk about?

• What
• Why
• Configuration
• Basics
• Advanced things
  • Caching
  • Task Management

shade: a Python library to wrap business logic around OpenStack resources and operations

Design Principles

• Expose a single API that works on all clouds
• Hide all vendor or deployer differences
• Support multi-cloud (write once, run anywhere)
• Simple to use (sane defaults)
• No plugins *
• Efficient at scale
• API always backwards compatible

Current Status

https://git.openstack.org/cgit/openstack-infra/shade

https://pypi.python.org/pypi/shade

• Used in Ansible
• Used in Infra Nodepool
• Official OpenStack Project
• Backend conversion to pure REST almost done

Why?

Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.

OpenStack Leaks Abstractions

OpenStack Breaks APIs

Basic concepts are needlessly complex

Client libraries are really for server-server communication

Infra solved these problems

Infra runs across 14 clouds at massive scale (20k servers per day)

Why not share what we've learned with other people?

simplicity

This is what using a cloud should look like

cloud = openstack_cloud('vexxhost')
image = cloud.create_image(
    'image-name', filename='image-filename.qcow2', wait=True)
flavor = cloud.get_flavor_by_ram(512)
cloud.create_server(
    'my-server', image=image, flavor=flavor, wait=True, auto_ip=True)
      

existence of shade is a bug

existence of shade is a feature

Using shade

Step One: Configuration

os-client-config

http://git.openstack.org/cgit/openstack/os-client-config

• A library to handle config information for openstack clients
• Tracks differences in vendors that can't be discovered
• Also used for ansible, python-openstackclient and openstacksdk
• Reads clouds.yaml config file, environment vars and argparse

clouds.yaml - simple

clouds:
  my-citycloud:
    profile: citycloud
    auth:
      username: mordred
      password: XXXXXXXXXXXXX
      project_id: 65222a4d09ea4c68934fa1028c77f394
      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38
      

clouds.yaml - mildly complex

  my-vexxhost:
    volume_api_version: 3
    profile: vexxhost
    auth:
      user_domain_id: default
      project_domain_id: default
      project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
      username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1
      password: XXXXXXXXXXXXX
    regions:
    - ca-ymq-1
    image_endpoint_override: https://image-ca-ymq-1.vexxhost.net/v2
      

clouds.yaml - complex

  my-internap:
    auth:
      auth_url: https://identity.api.cloud.iweb.com
      username: api-55f9a00fb2619
      project_name: inap-17037
      password: XXXXXXXXXXXXX
    identity_api_version: 3
    floating_ip_source: None
    regions:
    - name: ams01
      values:
        networks:
        - name: inap-17037-WAN1654
          routes_externally: true
          default_interface: true
        - name: inap-17037-LAN3631
          routes_externally: false
      

Environment Variables

• os-client-config also processes OS_ environment variables
• If present, they go into a cloud named "envvars"

auth_type

keystone has pluggable authentication

• auth dict contents vary based on plugin
• Defaults to 'password' which autodetects based on parameters
• Other options: admin_token, v3oidcpassword, v3oidcauthcode
• This is where I said "*" on "no plugins" earlier

auth_type - not part of auth_dict

  my-vexxhost:
    profile: vexxhost
    auth_type: v3password
    auth:
      user_domain_id: default
      project_domain_id: default
      project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
      username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1
      password: XXXXXXXXXXXXX
    regions:
    - ca-ymq-1
      

python-openstackclient

openstack --os-cloud=my-vexxhost servers list
      

export OS_CLOUD=my-vexxhost
openstack servers list
      

shade inventory

• Code behind ansible OpenStack dynamic inventory plugin
• All resources in all of your clouds

mordred@camelot:~$ shade-inventory --list --yaml
      

- accessIPv4: 199.204.45.50
  accessIPv6: 2604:e100:1:0:f816:3eff:feb5:ce98
  addresses:
    public:
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:b5:ce:98
      OS-EXT-IPS:type: fixed
      addr: 2604:e100:1:0:f816:3eff:feb5:ce98
      version: 6
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:b5:ce:98
      OS-EXT-IPS:type: fixed
      addr: 199.204.45.50
      version: 4
  adminPass: null
  created: '2017-04-02T17:04:49Z'
  disk_config: AUTO
  flavor:
    id: bbcb7eb5-5c8d-498f-9d7e-307c575d3566
    name: v1-standard-1
  has_config_drive: false
  host_id: 5b303fdae29a6b2c39c90a2a1c6dc1191297840d8c395e09c4e319b7
  id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
  image:
    id: dd33a4c8-5a86-40ec-9fe0-8578d6a6dc10
    name: Ubuntu 16.04.1 LTS [2017-03-03]
  interface_ip: 199.204.45.50
  key_name: mordred
  launched_at: '2017-04-02T17:05:02.000000'
  location:
    cloud: my-vexxhost
    project:
      domain_id: default
      domain_name: null
      id: db92b20496ae4fbda850a689ea9d563f
      name: d8af8a8f-a573-48e6-898a-af333b970a2d
    region_name: ca-ymq-1
    zone: ca-ymq-2
  metadata: {}
  name: irc-bouncer
  networks:
    public:
    - 2604:e100:1:0:f816:3eff:feb5:ce98
    - 199.204.45.50
  power_state: 1
  private_v4: ''
  progress: 0
  properties: {}
  public_v4: 199.204.45.50
  public_v6: 2604:e100:1:0:f816:3eff:feb5:ce98
  security_groups:
  - description: default
    id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
    location:
      cloud: my-vexxhost
      project:
        domain_id: default
        domain_name: null
        id: db92b20496ae4fbda850a689ea9d563f
        name: d8af8a8f-a573-48e6-898a-af333b970a2d
      region_name: ca-ymq-1
      zone: null
    name: default
    properties: {}
    security_group_rules:
    - direction: ingress
      ethertype: IPv4
      id: 2172daaa-e402-4cf0-b353-1a7cb7dae184
      location:
        cloud: my-vexxhost
        project:
          domain_id: default
          domain_name: null
          id: db92b20496ae4fbda850a689ea9d563f
          name: d8af8a8f-a573-48e6-898a-af333b970a2d
        region_name: ca-ymq-1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group: {}
      protocol: null
      remote_group_id: null
      remote_ip_prefix: 0.0.0.0/0
      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
    - direction: ingress
      ethertype: IPv4
      id: 47fb6b4c-2f9b-4279-a77c-a645f7fc964f
      location:
        cloud: my-vexxhost
        project:
          domain_id: default
          domain_name: null
          id: db92b20496ae4fbda850a689ea9d563f
          name: d8af8a8f-a573-48e6-898a-af333b970a2d
        region_name: ca-ymq-1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: db92b20496ae4fbda850a689ea9d563f
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
    - direction: ingress
      ethertype: IPv4
      id: 8287f9be-614b-4145-9f00-62fa9c856362
      location:
        cloud: my-vexxhost
        project:
          domain_id: default
          domain_name: null
          id: db92b20496ae4fbda850a689ea9d563f
          name: d8af8a8f-a573-48e6-898a-af333b970a2d
        region_name: ca-ymq-1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: db92b20496ae4fbda850a689ea9d563f
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
    - direction: ingress
      ethertype: IPv4
      id: fdd91863-c19b-4b4b-a1bc-822cb8b0adf7
      location:
        cloud: my-vexxhost
        project:
          domain_id: default
          domain_name: null
          id: db92b20496ae4fbda850a689ea9d563f
          name: d8af8a8f-a573-48e6-898a-af333b970a2d
        region_name: ca-ymq-1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group: {}
      protocol: null
      remote_group_id: null
      remote_ip_prefix: 0.0.0.0/0
      security_group_id: 22f8104a-2359-4091-aa61-9a4c48d6df3f
  status: ACTIVE
  task_state: null
  terminated_at: null
  updated: '2017-04-02T17:05:02Z'
  user_id: e9b21dc437d149858faee0898fb08e92
  vm_state: active
  volumes:
  - attachments:
    - attached_at: '2017-04-11T19:11:09.000000'
      attachment_id: 60f81620-4714-438b-b481-cd8fd7a4dd3e
      device: /dev/vdc
      host_name: null
      id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
      server_id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
      volume_id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
    can_multiattach: false
    consistencygroup_id: null
    created_at: '2017-04-11T19:10:20.000000'
    description: ''
    device: /dev/vdc
    host: null
    id: 7aa81299-bb1c-449b-8a73-b55a9bdc5fca
    is_bootable: false
    is_encrypted: false
    location:
      cloud: my-vexxhost
      project:
        domain_id: default
        domain_name: null
        id: db92b20496ae4fbda850a689ea9d563f
        name: d8af8a8f-a573-48e6-898a-af333b970a2d
      region_name: ca-ymq-1
      zone: ca-ymq-2
    metadata:
      attached_mode: rw
      readonly: 'False'
    migration_status: null
    name: foo
    properties:
      volume_image_metadata: {}
    replication_driver: null
    replication_extended_status: null
    replication_status: disabled
    size: 1
    snapshot_id: null
    source_volume_id: null
    status: in-use
    updated_at: '2017-04-11T19:11:09.000000'
    volume_type: null
  - attachments:
    - attached_at: '2017-04-02T17:06:27.000000'
      attachment_id: 4f24b276-ea5c-4a79-824f-434183c3c518
      device: /dev/vdb
      host_name: null
      id: 9b6110df-cd97-4e37-9596-af371653a791
      server_id: 5b3cde7d-ad17-4b28-9c08-9f4627abc659
      volume_id: 9b6110df-cd97-4e37-9596-af371653a791
    can_multiattach: false
    consistencygroup_id: null
    created_at: '2017-04-02T16:34:41.000000'
    description: ''
    device: /dev/vdb
    host: null
    id: 9b6110df-cd97-4e37-9596-af371653a791
    is_bootable: true
    is_encrypted: false
    location:
      cloud: my-vexxhost
      project:
        domain_id: default
        domain_name: null
        id: db92b20496ae4fbda850a689ea9d563f
        name: d8af8a8f-a573-48e6-898a-af333b970a2d
      region_name: ca-ymq-1
      zone: ca-ymq-2
    metadata:
      attached_mode: rw
      readonly: 'False'
    migration_status: null
    name: mordred-irc-save
    properties:
      volume_image_metadata:
        base_image_ref: 69c99b45-cd53-49de-afdc-f24789eb8f83
        checksum: dddc88434b12b5b61b6c7a3a5a880c35
        clean_attempts: '7'
        container_format: bare
        disk_format: raw
        hw_rng_model: virtio
        image_id: 1659da1f-7650-4b91-b316-c3d69623e3fc
        image_location: snapshot
        image_name: mordred-irc-save
        image_state: available
        image_type: snapshot
        instance_uuid: 811c5197-dba7-4d3a-a3f6-68ca5328b9a7
        libvirt_use_agent: 'True'
        min_disk: '40'
        min_ram: '0'
        network_allocated: 'True'
        os_type: linux
        owner_id: db92b20496ae4fbda850a689ea9d563f
        size: '42949672960'
        user_id: e9b21dc437d149858faee0898fb08e92
    replication_driver: null
    replication_extended_status: null
    replication_status: disabled
    size: 40
    snapshot_id: null
    source_volume_id: null
    status: in-use
    updated_at: '2017-04-02T17:06:27.000000'
    volume_type: null
- accessIPv4: 89.40.216.229
  accessIPv6: ''
  addresses:
    private:
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:29:6d:20
      OS-EXT-IPS:type: fixed
      addr: 10.4.0.22
      version: 4
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:29:6d:20
      OS-EXT-IPS:type: floating
      addr: 89.40.216.229
      version: 4
  adminPass: null
  created: '2017-05-08T18:48:34Z'
  disk_config: MANUAL
  flavor:
    id: 10973fe6-81ba-480c-bc36-bec93ee03d5c
    name: 4C-4GB-100GB
  has_config_drive: false
  host_id: 8ed21447e12616c0c24f6e246f0ec423d6be1532716869a3617fbe80
  id: a72fd3ea-9357-482d-b33a-fe61d7c37bfc
  image:
    id: 95e4c449-8abf-486e-97d9-dc3f82417d2d
    name: Ubuntu 16.04 Xenial Xerus
  interface_ip: 89.40.216.229
  key_name: null
  launched_at: '2017-05-08T18:48:39.000000'
  location:
    cloud: my-citycloud
    project:
      domain_id: d0919bd5e8d74e49adf0e145807ffc38
      domain_name: null
      id: 65222a4d09ea4c68934fa1028c77f394
      name: null
    region_name: Buf1
    zone: nova
  metadata: {}
  name: my-server
  networks:
    private:
    - 10.4.0.22
    - 89.40.216.229
  power_state: 1
  private_v4: 10.4.0.22
  progress: 0
  properties: {}
  public_v4: 89.40.216.229
  public_v6: ''
  security_groups:
  - description: Default security group
    id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
    location:
      cloud: my-citycloud
      project:
        domain_id: d0919bd5e8d74e49adf0e145807ffc38
        domain_name: null
        id: 65222a4d09ea4c68934fa1028c77f394
        name: null
      region_name: Buf1
      zone: null
    name: default
    properties: {}
    security_group_rules:
    - direction: ingress
      ethertype: IPv4
      id: 67cbd716-ace1-4822-93e3-1a14a2763efa
      location:
        cloud: my-citycloud
        project:
          domain_id: d0919bd5e8d74e49adf0e145807ffc38
          domain_name: null
          id: 65222a4d09ea4c68934fa1028c77f394
          name: null
        region_name: Buf1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: 65222a4d09ea4c68934fa1028c77f394
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
    - direction: ingress
      ethertype: IPv4
      id: f5e80c4c-fe7a-4bfc-a8c0-0105b2c272d5
      location:
        cloud: my-citycloud
        project:
          domain_id: d0919bd5e8d74e49adf0e145807ffc38
          domain_name: null
          id: 65222a4d09ea4c68934fa1028c77f394
          name: null
        region_name: Buf1
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: 65222a4d09ea4c68934fa1028c77f394
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 9fb5ba44-5c46-4357-8e60-8b55526cab54
  status: ACTIVE
  task_state: null
  terminated_at: null
  updated: '2017-05-08T18:48:39Z'
  user_id: c17534835f8f42bf98fc367e0bf35e09
  vm_state: active
  volumes: []
- accessIPv4: 173.231.180.251
  accessIPv6: ''
  addresses:
    inap-17037-WAN1654:
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:6b:97:9c
      OS-EXT-IPS:type: fixed
      addr: 173.231.180.251
      version: 4
  adminPass: null
  created: '2017-05-08T18:48:56Z'
  disk_config: null
  flavor:
    id: A1.4
    name: A1.4
  has_config_drive: true
  host_id: 6d0c22f3176c3bcbb78a418a93809ba6bfa16267a4c38bd614a892d9
  id: ba864796-b56a-4154-b220-e1b96daf39c0
  image:
    id: a6fefc9b-c208-4cd0-acc3-2ab09a38054b
    name: Ubuntu 16.04 LTS (Xenial Xerus)
  interface_ip: 173.231.180.251
  key_name: null
  launched_at: null
  location:
    cloud: my-internap
    project:
      domain_id: null
      domain_name: null
      id: 760e6c137b3840d78472d313dfa3df45
      name: inap-17037
    region_name: ams01
    zone: null
  metadata: {}
  name: my-server
  networks:
    inap-17037-WAN1654:
    - 173.231.180.251
  power_state: 1
  private_v4: ''
  progress: 0
  properties: {}
  public_v4: 173.231.180.251
  public_v6: ''
  security_groups:
  - description: default
    id: 69871901-68c6-486b-9a1b-6a9adb88ea23
    location:
      cloud: my-internap
      project:
        domain_id: null
        domain_name: null
        id: 760e6c137b3840d78472d313dfa3df45
        name: inap-17037
      region_name: ams01
      zone: null
    name: default
    properties: {}
    security_group_rules:
    - direction: ingress
      ethertype: IPv4
      id: 24b2ce3a-bf29-44fb-849f-f958e3efb736
      location:
        cloud: my-internap
        project:
          domain_id: null
          domain_name: null
          id: 760e6c137b3840d78472d313dfa3df45
          name: inap-17037
        region_name: ams01
        zone: null
      port_range_max: 65535
      port_range_min: 1
      properties:
        group: {}
      protocol: tcp
      remote_group_id: null
      remote_ip_prefix: 0.0.0.0/0
      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
    - direction: ingress
      ethertype: IPv4
      id: 374aa520-c101-4de3-be88-f0025396bdb7
      location:
        cloud: my-internap
        project:
          domain_id: null
          domain_name: null
          id: 760e6c137b3840d78472d313dfa3df45
          name: inap-17037
        region_name: ams01
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: 760e6c137b3840d78472d313dfa3df45
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
    - direction: ingress
      ethertype: IPv4
      id: 376b67c2-284b-4a81-a989-e7b8f6d44a07
      location:
        cloud: my-internap
        project:
          domain_id: null
          domain_name: null
          id: 760e6c137b3840d78472d313dfa3df45
          name: inap-17037
        region_name: ams01
        zone: null
      port_range_max: 65535
      port_range_min: 1
      properties:
        group: {}
      protocol: udp
      remote_group_id: null
      remote_ip_prefix: 0.0.0.0/0
      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
    - direction: ingress
      ethertype: IPv4
      id: 481f9402-753d-4286-8079-be45a414023c
      location:
        cloud: my-internap
        project:
          domain_id: null
          domain_name: null
          id: 760e6c137b3840d78472d313dfa3df45
          name: inap-17037
        region_name: ams01
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group: {}
      protocol: icmp
      remote_group_id: null
      remote_ip_prefix: 0.0.0.0/0
      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
    - direction: ingress
      ethertype: IPv4
      id: 832aaf3b-b0fb-4103-b2fa-def955077667
      location:
        cloud: my-internap
        project:
          domain_id: null
          domain_name: null
          id: 760e6c137b3840d78472d313dfa3df45
          name: inap-17037
        region_name: ams01
        zone: null
      port_range_max: null
      port_range_min: null
      properties:
        group:
          name: default
          tenant_id: 760e6c137b3840d78472d313dfa3df45
      protocol: null
      remote_group_id: null
      remote_ip_prefix: null
      security_group_id: 69871901-68c6-486b-9a1b-6a9adb88ea23
  status: ACTIVE
  task_state: null
  terminated_at: null
  updated: '2017-05-08T18:49:05Z'
  user_id: c18bd37c27d14b1ba4c424e6d42f2ccd
  vm_state: active
  volumes: []
      

Cloud-Region

Each OpenStackCloud object represents one region of one cloud

Simplest Cloud Construction

• Works if you only have one cloud

import shade
cloud = shade.openstack_cloud()
      

Simplest Cloud Construction - More than one

clients:
  default:
    cloud: vexxhost
      

import shade
cloud = shade.openstack_cloud()
      

Simple Cloud Construction

import shade
cloud = shade.openstack_cloud(cloud='ustack', region_name='bj1')
      

Complex Cloud Construction

import os_client_config
import shade

config = os_client_config.OpenStackConfig()
cloud_config = config.get_one_cloud(
    cloud='ustack', region_name='bj1',
    argparse=my_args, **other_arguments)
cloud = shade.OpenStackCloud(cloud_config=cloud_config)
      

logging

• Python logging
• shade.simple_logging() helper function
• Turns off annoying warnings
• debug=True - turn on debug logging and request_id logging
• http_debug=True - debug=True + http request tracing

A note on Exceptions

• All (remaining) python*client exceptions are hidden and re-raised
• Hiding exceptions is evil and I'm a bad person
• Exceptions are part of the interface
• Can't hide vendor choice if shade user catches underlying exception
• See also: Restification later

Exceptions

• All Exceptions are subclasses of OpenStackCloudException
• Direct REST calls throw OpenStackCloudHTTPError
• OpenStackCloudHTTPError subclasses OpenStackCloudException and requests.exceptions.HTTPError
• OpenStackCloudURINotFound for 404
• OpenStackCloudBadRequest for 400

Putting it all together

import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='vexxhost')
image = cloud.create_image(
    'ubuntu-trusty', filename='ubuntu-trusty.qcow2', wait=True)
flavor = cloud.get_flavor_by_ram(512)
cloud.create_server(
    'my-server', image=image['id'], flavor=flavor['id'],
     wait=True, auto_ip=True)
      

Problem: Image API version

• v1 PUT: Catalyst, Datacentred, Internap
• v2 PUT: Auro, City Cloud, Dreamhost, Elastx, Enter Cloud Suite, OVH, RunAbove, Vexxhost, Ultimum, UnitedStack
• v2 Tasks: Switch Engines, Rackspace

Image Upload Code

cloud = shade.openstack_cloud(cloud='vexxhost', region_name='ca-ymq-1')
cloud.create_image('my-image', filename='my-image.qcow2', wait=True)
      

Image Upload PUT v2 Log

Calculating hashes for my-image.qcow2
Image file my-image.vhd md5:b28fa44077b94ba635842a73eecc1d8c sha256:9a3da71e37ef1579657c4a77ff2caf936dd9be8361d1a02595cc116f0ffd328d
Manager my-vexxhost:ca-ymq-1 running task image.GET.images
Manager my-vexxhost:ca-ymq-1 ran task image.GET.images in 1.38961982727s
GET call to image for https://image-ca-ymq-1.vexxhost.net/v2/images used request id req-2bbc8c10-6c4c-4aa3-8bb7-6ba425cb0ea7
Manager my-vexxhost:ca-ymq-1 running task image.GET.images
Manager my-vexxhost:ca-ymq-1 ran task image.GET.images in 0.245140790939s
GET call to image for https://image-ca-ymq-1.vexxhost.net/v2/images?marker=5ebb74ba-c9c5-4006-8c77-ceebf9833428 used request id req-da204dc3-eb07-460d-b9f5-74b6ea5fe944
Manager my-vexxhost:ca-ymq-1 running task image.POST.images
Manager my-vexxhost:ca-ymq-1 ran task image.POST.images in 0.114197969437s
POST call to image for https://image-ca-ymq-1.vexxhost.net/v2/images used request id req-02a5fd04-ca9a-4a71-bfe1-4ff80e49cdea returning object 28d3e4b2-4607-4778-84bd-e84ef217d21c
Manager my-vexxhost:ca-ymq-1 running task image.PUT.images.file
Manager my-vexxhost:ca-ymq-1 ran task image.PUT.images.file in 2.10132312775s
PUT call to image for https://image-ca-ymq-1.vexxhost.net/v2/images/28d3e4b2-4607-4778-84bd-e84ef217d21c/file used request id req-71e3ed5e-77dc-4062-9ba8-13d495713e13
      

Image Tasks

cloud = shade.openstack_cloud(cloud='rax', region_name='DFW')
cloud.create_image('my-image', filename='my-image.vhd', wait=True)
      

Image Upload Tasks Log

Calculating hashes for my-image.vhd
Image file my-image.vhd md5:b28fa44077b94ba635842a73eecc1d8c sha256:9a3da71e37ef1579657c4a77ff2caf936dd9be8361d1a02595cc116f0ffd328d
Manager rax:DFW running task image.GET.discovery
Manager rax:DFW ran task image.GET.discovery in 1.65571808815s
Glance version discovery failed, assuming endpoint in the catalog is already versioned. (404) Client Error: Not Found for https://dfw.images.api.rackspacecloud.com/v2/
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.637840032578s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/images used request id req-b716bd86-121f-4cdc-b386-2f50546e8d88
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.491132974625s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=e389ab51-1327-488a-867c-4d7c14e6c916 used request id req-90746cbf-610a-4c3d-aab8-f81e673b315a
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.371850967407s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=3eba4fbb-51da-4233-b699-8a4030561add used request id req-00937f32-f573-4ac3-86d5-0aeada7432d1
Manager rax:DFW running task object-store.GET.info
Manager rax:DFW ran task object-store.GET.info in 0.335763931274s
Manager rax:DFW running task object-store.HEAD.images
Manager rax:DFW ran task object-store.HEAD.images in 0.393274784088s
Manager rax:DFW running task object-store.HEAD.images
Manager rax:DFW ran task object-store.HEAD.images in 3.68135595322s
swift stale check, no object: images/my-image
swift uploading my-image.vhd to images/my-image
Manager rax:DFW running task object-store.PUT.images
Manager rax:DFW ran task object-store.PUT.images in 3.57636904716s
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.616628885269s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/images used request id req-ee7091ae-88a2-4cdb-ac8e-35159ef4cda0
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.527029037476s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=e389ab51-1327-488a-867c-4d7c14e6c916 used request id req-d2caefd2-d69f-46d2-86c1-a062f039ce74
Manager rax:DFW running task image.GET.images
Manager rax:DFW ran task image.GET.images in 0.317733049393s
GET call to image for https://dfw.images.api.rackspacecloud.com/v2/v2/images?marker=3eba4fbb-51da-4233-b699-8a4030561add used request id req-56411197-65ce-4ac9-847d-0a7e7041766d
Manager rax:DFW running task image.POST.tasks
Manager rax:DFW ran task image.POST.tasks in 0.796432971954s
POST call to image for https://dfw.images.api.rackspacecloud.com/v2/tasks used request id req-edc7842f-5fca-4a08-96b3-55a67c840d16 returning object 45df1ffa-3837-4fbc-a8fb-f213f845b18c
      

Problem: Version Discovery

• OpenStack Services have version discovery documents
• Deployers register versioned endpoints in catalog
• Some services (cinder, mistral) have versioned service types (volumev2, workflowv2)
• Users can't (sanely) do version discovery
• shade fixes this for glance and cinder today

Version Discovery Solutions

• shade will fix for everything by end of the month
• OpenStack Service Types Authority
• API-WG specs on Full Service Type and Version Discovery Process
• Work with other languages to implement API-WG specs
• User requests:
  • image version 2
  • workflow LATEST
  • compute
  • volume versions 2 or 3
• Later today we'll be discussing moving this forward (Room 102 at 4:40)

Problem: Networking choices

• Cloud has externally routable IP from neutron (OVH)
• Cloud has externally routable IP neutron AND supports optional private tenant networks (vexxhost)
• Cloud has private tenant network provided by neutron and requires floating IP (citycloud)
• Cloud has private tenant network provided by nova-network and requires floating-ip for external routing (auro)
• Cloud has externally routable IP from neutron but no neutron APIs (Rackspace)

The Floating IP Case

import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='my-citycloud')
cloud.create_server(
    'my-server',
    image='Ubuntu 16.04 Xenial Xerus',
    flavor=dict(id='0dab10b5-42a2-438e-be7b-505741a7ffcc'),
    wait=True, auto_ip=True)
      

The Floating IP Case Log

Manager my-citycloud:Buf1 running task network.GET.networks
Manager my-citycloud:Buf1 ran task network.GET.networks in 1.29657912254s
GET call to network for https://buf1.citycloud.com:9696/v2.0/networks.json used request id req-a65496e6-984f-4323-98d5-3adb339875c6
Manager my-citycloud:Buf1 running task network.GET.subnets
Manager my-citycloud:Buf1 ran task network.GET.subnets in 0.713509082794s
GET call to network for https://buf1.citycloud.com:9696/v2.0/subnets.json used request id req-66ccbfce-ceb6-4e0a-a969-0eaf62605d92
Manager my-citycloud:Buf1 running task image.GET.discovery
Manager my-citycloud:Buf1 ran task image.GET.discovery in 0.474808931351s
Manager my-citycloud:Buf1 running task image.GET.images
Manager my-citycloud:Buf1 ran task image.GET.images in 0.714179039001s
GET call to image for https://buf1.citycloud.com:9292/v2/images used request id req-9a7756c3-2ee4-416d-a6ad-804027330e5f
Manager my-citycloud:Buf1 running task ServerCreate
Manager my-citycloud:Buf1 ran task ServerCreate in 3.38534998894s
Manager my-citycloud:Buf1 running task ServerList
Manager my-citycloud:Buf1 ran task ServerList in 0.959898948669s
Waiting 2.0 seconds
Manager my-citycloud:Buf1 running task ServerList
Manager my-citycloud:Buf1 ran task ServerList in 1.21962594986s
Manager my-citycloud:Buf1 running task network.GET.ports
Manager my-citycloud:Buf1 ran task network.GET.ports in 0.287118911743s
GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-fd5acf95-21bd-40bb-ad08-e15ac7107874
Manager my-citycloud:Buf1 running task network.GET.floatingips
Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.228282928467s
GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json?port_id=af554254-ac9f-4440-b93a-e4ec64783758 used request id req-ed52ca15-9924-488e-a3c0-3a805348eca3
Manager my-citycloud:Buf1 running task network.GET.ports
Manager my-citycloud:Buf1 ran task network.GET.ports in 0.200226068497s
GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-f808b9e0-d136-4bd7-a51b-b08d3c6eddb3
Manager my-citycloud:Buf1 running task network.GET.floatingips
Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.140635967255s
GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json used request id req-880919b7-4af3-42a3-8e07-4b80a6ae5bac
Manager my-citycloud:Buf1 running task network.GET.ports
Manager my-citycloud:Buf1 ran task network.GET.ports in 0.317744970322s
GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-af4bb447-e157-44c1-a727-2cecfb595c0e
Manager my-citycloud:Buf1 running task network.PUT.floatingips
Manager my-citycloud:Buf1 ran task network.PUT.floatingips in 0.425180912018s
PUT call to network for https://buf1.citycloud.com:9696/v2.0/floatingips/e69179dc-a904-4c9a-a4c9-891e2ecb984c.json used request id req-2241d8b6-bf50-41c1-9129-c8cfca2c73bc returning object e69179dc-a904-4c9a-a4c9-891e2ecb984c
Manager my-citycloud:Buf1 running task ServerList
Manager my-citycloud:Buf1 ran task ServerList in 1.19975018501s
Manager my-citycloud:Buf1 running task network.GET.ports
Manager my-citycloud:Buf1 ran task network.GET.ports in 0.184152126312s
GET call to network for https://buf1.citycloud.com:9696/v2.0/ports.json?device_id=1e2211f2-993f-4ba0-aa54-01743cc49181 used request id req-9f49b751-d741-45ff-a5fc-ca750826df63
Manager my-citycloud:Buf1 running task network.GET.floatingips
Manager my-citycloud:Buf1 ran task network.GET.floatingips in 0.21661400795s
GET call to network for https://buf1.citycloud.com:9696/v2.0/floatingips.json?port_id=af554254-ac9f-4440-b93a-e4ec64783758 used request id req-81e231c6-a4d5-466a-bffd-0af6f4dc4be8
      

The Direct Attached IP Case

import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='internap', region_name='ams01')
cloud.create_server(
    'my-server',
    image='Ubuntu 16.04 LTS (Xenial Xerus)',
    flavor=dict(id='A1.4'),
    wait=True, auto_ip=True)
      

Direct Attached IP Log

Manager my-internap:ams01 running task network.GET.networks
Manager my-internap:ams01 ran task network.GET.networks in 0.783267021179s
GET call to network for https://network.api.ams01.cloud.iweb.com/v2.0/networks.json used request id req-807686be-238a-46e1-b0e5-5dfa31949316
Manager my-internap:ams01 running task network.GET.subnets
Manager my-internap:ams01 ran task network.GET.subnets in 0.155231952667s
GET call to network for https://network.api.ams01.cloud.iweb.com/v2.0/subnets.json used request id req-9e71772d-70dc-4dc9-8da4-734639a10dd4
Manager my-internap:ams01 running task image.GET.discovery
Manager my-internap:ams01 ran task image.GET.discovery in 0.589816093445s
Manager my-internap:ams01 running task image.GET.images
Manager my-internap:ams01 ran task image.GET.images in 0.568889856339s
GET call to image for https://image.api.ams01.cloud.iweb.com/v2/images used request id req-b0e25143-a581-4c6a-baab-2376626e529a
Manager my-internap:ams01 running task ServerCreate
Manager my-internap:ams01 ran task ServerCreate in 1.55480098724s
Manager my-internap:ams01 running task ServerList
Manager my-internap:ams01 ran task ServerList in 0.316590070724s
Waiting 2.0 seconds
Manager my-internap:ams01 running task ServerList
Manager my-internap:ams01 ran task ServerList in 0.317390918732s
      

Problem: Finding an Image

• Ubuntu 16.04.1 LTS [2017-03-03]
• Ubuntu 16.04 Xenial Xerus
• Ubuntu 16.04 LTS (Xenial Xerus)
• Easy for a human - hard for a computer

Solution: Sorry - no good solution yet

• Always Upload your Own Images
• Discussed in Glance Deployer Session Yesterday
• Next step - collecting full set of User Stories
• Hopefully we'll define common metadata
• Hopefully we'll define "correct" image lifecycle

Problem: Crazy Dependencies

• shade started life depending on python-*client
• Giant dependency chain
• Users should ALWAYS use the latest shade
• Conflicting depends makes this hard

Solution: 'RESTification'

• Have been hard at work moving to REST
• Done with heat, magnum, swift, glance, trove
• cinder and neutron done but unreleased
• nova, ironic, designate left
• Thanks Rosario and Slaweq!!!

BTW - The code is easier with REST

Advanced Things

• shade should DTRT 95% of the time
• Sometimes, you need to be extra advanced
• We want to suppor that too

Task Manager

Manager zetta ran task ServerGet in 0.726951122284s
      

• Every API operation is a Task() that is run by a TaskManager()
• Default shade TaskManager immediately executes the call
• nodepool passes in a threaded TaskManager to manage API throttling

cloud = shade.OpenStackCloud(
    cloud_config=cloud_config,
    manager=MyTaskManager())
        

list vs. get

• get_server is a wrapper around list_servers
• *sometimes* we can push filter conditions to the cloud
• Actually done in SUPPORT of scaling

Caching and Rate Limiting

• API Operations can be expensive
• Support grew from nodepool's needs
• shade uses dogpile.cache - default to NullCache

Cache Config

clouds.yaml

cache:
  class: dogpile.cache.dbm
  expiration_time: 3600
  arguments:
    filename: /home/mordred/.cache/openstack/shade.dbm
  expiration:
      server: 5
      

Caching Special Case

• servers, ports and floating-ips have custome caching
• Specifically done for nodepool
• After RESTification, will migrate them to dogpile
• Two levels - caching and rate-limiting

What if you're not a Python developer?

Yes, we love you too

ansible

- os_image:
    cloud: vexxhost
    name: centos-7
    file: centos-7.qcow2
    wait: true
- os_server:
    cloud: vexxhost
    name: my-server
    flavor_ram: 1024
    image: centos-7
    wait: true
      

ansible

Add my keypair to 30 Regions in 13 clouds

- os_keypair:
    cloud: "{{ item.cloud }}"
    region_name: "{{ item.region_name }}"
    name: mordred
    public_key_file: ~/.ssh/id_rsa.pub
    with-items:
    - {cloud: vexxhost, region_name: ca-ymq-1}
    - {cloud: ovh, region_name: SBG1}
    - {cloud: ovh, region_name: BHS1}
    - {cloud: ovh, region_name: GRA1}
    - {cloud: citycloud, region_name: Buf1}
    - {cloud: citycloud, region_name: La1}
    - {cloud: citycloud, region_name: Fra1}
    - {cloud: citycloud, region_name: Lon1}
    - {cloud: citycloud, region_name: Sto2}
    - {cloud: citycloud, region_name: Kna1}
    - {cloud: internap, region_name: ams01}
    - {cloud: internap, region_name: da01}
    - {cloud: internap, region_name: nyj01}
    - {cloud: internap, region_name: sin01}
    - {cloud: internap, region_name: sjc01}
    - {cloud: infracloud, region_name: vanilla}
    - {cloud: infracloud, region_name: chocolate}
    - {cloud: fuga, region_name: cystack}
    - {cloud: datacentred, region_name: sal01}
    - {cloud: clouda, region_name: regionOne}
    - {cloud: auro, region_name: van1}
    - {cloud: ustack, region_name: bj1}
    - {cloud: zetta, region_name: no-osl1}
    - {cloud: kiss, region_name: region1}
    - {cloud: rax, region_name: IAD}
    - {cloud: rax, region_name: ORD}
    - {cloud: rax, region_name: SYD}
    - {cloud: rax, region_name: LON}
    - {cloud: rax, region_name: DFW}
    - {cloud: rax, region_name: HKG}
      

The Future: Pushing Back Into OpenStack APIs

• Discussions later today
• Service Discovery
• Version Discovery
• Vendors Publishing Vendor Profile Documents
• Capabilities Discovery
• Connecting those to Interop Working Group

Thank You!