Who am I?

Distinguished Engineer

IBM Cloud

Who am I?

Technical Committee

Foundation Board of Directors

Developer Infrastructure Core Team

What are we going to talk about?

shade: a Python library to wrap business logic around OpenStack resources and operations

Design Principles

Current Status

https://git.openstack.org/cgit/openstack-infra/shade

https://pypi.python.org/pypi/shade

Why?

Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.

OpenStack Leaks Abstractions

OpenStack Breaks APIs

Basic concepts are needlessly complex

libcloud doens't really work

Client libraries are really for server-server communication

Infra solved these problems

Infra runs across five clouds at massive scale

Why not share what we've learned with other people?

simplicity

This is what using a cloud should look like


cloud = openstack_cloud('vexxhost')
image = cloud.create_image(
    'image-name', filename='image-filename.qcow2', wait=True)
flavor = cloud.get_flavor_by_ram(512)
cloud.create_server(
    'my-server', image=image['id'], flavor=flavor['id'],
     wait=True, auto_ip=True)
      

existence of shade is a bug

Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.

Let's drive towards profitable homogenization

Using shade

Step One: Configuration

os-client-config

http://git.openstack.org/cgit/openstack/os-client-config

os-client-config


clouds:
  dreamhost:
    profile: dreamhost
    auth:
      username: montay6
      project_name: dhc2111978
      password: XXXXXXXXXXXXX
  montytaylor-sjc:
    region_name: RegionOne
    auth:
      username: openstackjenkins
      password: XXXXXXXXXXXXXX
      project_name: openstackjenkins
      auth_url: https://montytaylor-sjc.openstack.blueboxgrid.com:5001/v2.0
  ustack:
    profile: unitedstack
    verify: False
    auth_type: v3password
    auth:
      username: mordred
      password: XXXXXXXXXXXXX
      project_name: mordred_project
      user_domain_id: 7fda84e224134e118158afd7d69936fd
      project_domain_id: 7fda84e224134e118158afd7d69936fd
      

auth_type

keystone has pluggable authentication

python-openstackclient

openstack --os-cloud=rax --os-region=DFW servers list
      

How I do it


function use {
  declare -a CloudRegion=(${1//:/ })
  export OS_CLOUD=${CloudRegion[0]}
  export OS_REGION_NAME=${CloudRegion[1]}
}
PS1='${debian_chroot:+($debian_chroot)}${OS_CLOUD:+${OS_CLOUD}:}${OS_REGION_NAME:+${OS_REGION_NAME}:}\u@\h:\w\$ '
      
mordred@camelot:~$ use rax:DFW
rax:DFW:mordred@camelot:~$ openstack servers list
      

shade inventory

mordred@camelot:~$ shade-inventory --list --yaml
      

- HUMAN_ID: true
  NAME_ATTR: name
  OS-EXT-AZ:availability_zone: az2
  OS-EXT-STS:power_state: 1
  OS-EXT-STS:task_state: null
  OS-EXT-STS:vm_state: active
  accessIPv4: 15.126.131.74
  accessIPv6: ''
  addresses:
    mordred@inaugust.com-network:
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:f3:20:9b
      OS-EXT-IPS:type: fixed
      addr: 10.0.0.106
      version: 4
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:f3:20:9b
      OS-EXT-IPS:type: floating
      addr: 15.126.131.74
      version: 4
  az: az2
  cloud: hp
  config_drive: 'True'
  created: '2015-04-25T15:15:22Z'
  flavor:
    id: '100'
    name: standard.xsmall
  hostId: 2a84ca599420d68466b782308f7892b81b4eda74debd037fe76c3ded
  human_id: launch_node
  id: f39df771-dd82-49b3-b325-b0b1a6842259
  image:
    id: c47a348c-fd37-437d-b7c4-e17ff6c86211
  interface_ip: 15.126.131.74
  key_name: null
  metadata: {}
  name: launch_node
  networks:
    mordred@inaugust.com-network:
    - 10.0.0.106
    - 15.126.131.74
  private_v4: 10.0.0.106
  progress: 0
  public_v4: 15.126.131.74
  public_v6: ''
  region: region-b.geo-1
  security_groups:
  - name: default
  status: ACTIVE
  tenant_id: '51595564575618'
  updated: '2015-06-06T17:26:09Z'
  user_id: '71312979441162'
  volumes: []
- HUMAN_ID: true
  NAME_ATTR: name
  OS-DCF:diskConfig: MANUAL
  OS-EXT-AZ:availability_zone: nova
  OS-EXT-STS:power_state: 1
  OS-EXT-STS:task_state: null
  OS-EXT-STS:vm_state: active
  OS-SRV-USG:launched_at: '2015-08-01T19:52:02.000000'
  OS-SRV-USG:terminated_at: null
  accessIPv4: 162.253.54.192
  accessIPv6: ''
  addresses:
    public:
    - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:9f:46:3e
      OS-EXT-IPS:type: fixed
      addr: 162.253.54.192
      version: 4
  az: nova
  cloud: vexxhost
  config_drive: 'True'
  created: '2015-08-01T19:52:16Z'
  flavor:
    id: bbcb7eb5-5c8d-498f-9d7e-307c575d3566
    name: v1-standard-1
  hostId: d998a94193237f8d12111cfdc856b83559e3a5b2e7716326ff46ad65
  human_id: mordred-irc
  id: 811c5197-dba7-4d3a-a3f6-68ca5328b9a7
  image:
    id: 69c99b45-cd53-49de-afdc-f24789eb8f83
    name: Ubuntu 14.04.2 LTS
  interface_ip: 162.253.54.192
  key_name: mordred
  metadata: {}
  name: mordred-irc
  networks:
    public:
    - 162.253.54.192
  os-extended-volumes:volumes_attached: []
  private_v4: ''
  progress: 0
  public_v4: 162.253.54.192
  public_v6: ''
  region: ca-ymq-1
  security_groups:
  - name: default
  status: ACTIVE
  tenant_id: db92b20496ae4fbda850a689ea9d563f
  updated: '2015-08-01T19:52:02Z'
  user_id: e9b21dc437d149858faee0898fb08e92
  volumes: []
      

Cloud-Region

Each OpenStackCloud object represents one region of one cloud

Simplest Cloud Construction


import shade
cloud = shade.openstack_cloud()
      

Simple Cloud Construction


import shade
cloud = shade.openstack_cloud(cloud='ustack', region_name='bj1')
      

Complex Cloud Construction


import os_client_config
import shade

config = os_client_config.OpenStackConfig()
cloud_config = config.get_one_cloud(
    cloud='ustack', region_name='bj1',
    argparse=my_args, **other_arguments)
cloud = shade.OpenStackCloud(cloud_config=cloud_config)
      

logging

A note on Exceptions

Putting it all together


import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='vexxhost')
image = cloud.create_image(
    'ubuntu-trusty', filename='ubuntu-trusty.qcow2', wait=True)
flavor = cloud.get_flavor_by_ram(512)
cloud.create_server(
    'my-server', image=image['id'], flavor=flavor['id'],
     wait=True, auto_ip=True)
      
Manager vexxhost ran task GlanceImageList in 1.52185106277s
Manager vexxhost ran task ImageCreate in 0.535313844681s
Manager vexxhost ran task ImageUpload in 3.40991711617s
Manager vexxhost ran task GlanceImageList in 0.513506174088s
Manager vexxhost ran task FlavorList in 1.30724596977s
Manager vexxhost ran task ServerCreate in 0.609236955643s
Manager vexxhost ran task ServerGet in 0.275503873825s
Manager vexxhost ran task ServerList in 0.338351964951s
Waiting 5 seconds Poll loop
Manager vexxhost ran task ServerList in 0.318593025208s
Manager vexxhost ran task NetworkList in 1.11756396294s
      

Problem: Image API version

Manager vexxhost ran task GlanceImageList in 1.52185106277s
Manager vexxhost ran task ImageCreate in 0.535313844681s
Manager vexxhost ran task ImageUpload in 3.40991711617s
Manager vexxhost ran task GlanceImageList in 0.513506174088s
      

Image Tasks


cloud = shade.openstack_cloud(cloud='rax')
cloud.create_image('ubuntu-trusty', filename='ubuntu-trusty.vhd', wait=True)
      
Manager rax ran task GlanceImageList in 2.0544719696s
Manager rax ran task ObjectCapabilities in 0.616773843765s
Manager rax ran task ContainerGet in 0.978188037872s
Manager rax ran task ObjectMetadata in 0.427683115005s
swift stale check, no object: images/ubuntu-trusty
swift uploading ubuntu-trusty.vhd to images/ubuntu-trusty Uses Swift Service
Manager rax ran task ObjectCreate in 0.00043797492981s
Manager rax ran task GlanceImageList in 0.844120025635s
Manager rax ran task ImageTaskCreate in 1.22424006462s
Manager rax ran task ImageTaskGet in 0.235086917877s
Waiting 2 seconds
Manager rax ran task ImageTaskGet in 0.215805053711s
      

Problem: Networking choices

The Floating IP Case


import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='hp')
flavor = cloud.get_flavor_by_ram(512)
image = cloud.get_image('ubuntu-trusty')
cloud.create_server(
    'my-server', image=image['id'], flavor=flavor['id'],
     wait=True, auto_ip=True)
      

The Floating IP Case

Manager hp ran task ServerCreate in 1.52772402763s
Manager hp ran task ServerGet in 0.483407020569s
Manager hp ran task ServerList in 0.809002876282s
Waiting 5 seconds
Manager hp ran task ServerList in 1.091807127s
Manager hp ran task NetworkList in 21.8774280548s
Manager hp ran task NeutronFloatingIPList in 0.213263988495s
Manager hp ran task PortList in 0.342208862305s
Manager hp ran task NeutronFloatingIPUpdate in 0.356763124466s
Manager hp ran task ServerGet in 0.433247089386s
Waiting 2 seconds
Manager hp ran task ServerGet in 0.56911110878s
Manager hp ran task ServerGet in 0.302348852158s
      

Don't Reuse Floating IP Case


import shade

shade.simple_logging(debug=True)
cloud = shade.openstack_cloud(cloud='bluebox')
flavor = cloud.get_flavor_by_ram(512)
image = cloud.get_image('ubuntu-trusty')
cloud.create_server(
    'my-server', image=image['id'], flavor=flavor['id'],
     wait=True, auto_ip=True, reuse=False)
      
Manager bluebox ran task NetworkList in 0.432415008545s
Manager bluebox ran task NeutronFloatingIPList in 0.54718708992s
Manager bluebox ran task NetworkList in 0.495000839233s
Manager bluebox ran task NeutronFloatingIPCreate in 0.819630146027s
Manager bluebox ran task PortList in 0.736854076385s
Manager bluebox ran task NeutronFloatingIPUpdate in 0.588222026825s
Manager bluebox ran task ServerGet in 0.646552085876s
Waiting 2 seconds
Manager bluebox ran task ServerGet in 0.726951122284s
Manager bluebox ran task ServerGet in 0.668478965759s
      

Advanced Topics

Task Manager

Manager bluebox ran task ServerGet in 0.726951122284s
      

Caching

Cache Config

clouds.yaml


cache:
  class: dogpile.cache.dbm
  expiration_time: 3600
  arguments:
    filename: /home/mordred/.cache/openstack/shade.dbm
  expiration:
      server: 5
      

ansible 2.0

Brand new modules, based on shade

Coming in 2.0 release


- os_image:
    cloud: vexxhost
    name: ubuntu-trusty
    file: ubuntu-trusty.qcow2
    wait: true
- os_server:
    cloud: vexxhost
    name: my-server
    flavor_ram: 1024
    image: ubuntu-trusty
    wait: true
      

ansible

multi-cloud support


- os_keypair:
    cloud: "{{ item }"
    name: mordred
    public_key_file: ~/.ssh/id_rsa.pub
    with-items:
    - vexxhost
    - rackspace
    - hp
    - ovh