Who am I?

Distinguished Technologist at HP
OpenStack Technical Committee
OpenStack Foundation Board of Directors
OpenStack Infra Core Team

What are we going to talk about?

TripleO
Ironic
Ansible

What is TripleO?

an idea
OpenStack On OpenStack
use Ironic-based OpenStack to Operate Openstack

Velocity

Cloud Empowers developers
Cloud Enables Increased Velocity
Cloud Drives Agility

OpenStack is not a virtualization layer, it's an abstraction layer.

What if your developers aren't writing Angry Birds?

What if your developers aren't developers?

If infrastructure is code ...

What if your developers are operators?

Why wouldn't you give your operators the same power as your developers?

Don't you like them?

Ironic is a service capable of
managing and provisioning
physical machines.

Nova + Ironic

Same but different


$ ironic node-create -d pxe_ipmitool \
  -i ipmi_username=admin -i ipmi_password=fake -i ipmi_address=10.1.2.3 \
  -p cpus=4 -p memory_mb=8192 -p local_gb=500 \
  -e note='spare server'
+--------------+------------------------------------------------------------+
| Property     | Value                                                      |
+--------------+------------------------------------------------------------+
| chassis_uuid | None                                                       |
| driver       | pxe_ipmitool                                               |
| driver_info  | {u'ipmi_address': u'10.1.2.3', u'ipmi_username': u'admin', |
|              | u'ipmi_password': u'fake'}                                 |
| extra        | {u'note': u'spare server'}                                 |
| properties   | {u'memory_mb': u'8192', u'local_gb': u'500', u'cpus': u'4'}|
| uuid         | 7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f                       |
+--------------+------------------------------------------------------------+


$ tail -f /var/log/nova/n-cpu.log
...
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free ram (MB): 8192
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free disk (GB): 500
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free VCPUS: 4

Diversity is good

Cloud providers: HP, Rackspace, Dreamhost, Cloudwatt
Deployments: Public, Managed, Private
Compute drivers: virtual, bare metal, container
Ironic drivers: pxe/impi, ilo, HP OneView

What can you do?

anything - it's a cloud

What the heck has the TripleO team been doing then?

TripleO

Community developed
Opinionated
Avoid distro and config management religion
Be a usable real deployment we can gate on

Community Developed

Exist as part of the OpenStack project
Prove the story end to end
Subject to TC governance
Tighter feedback loop

Avoid distro and config management religion

rpm vs. deb - in the gate == rpm + deb
puppet vs. chef vs. salt vs. ansible == all of them
Choosing one excludes other folks from participating

Opinionated

Golden Images
Upgrade tied to HA
Target Continual Delivery
Drive fixes into OpenStack directly

Major Components

nova+ironic
heat
diskimage-builder
os-collect-config
os-apply-config
os-refresh-config

Lesson from os-*-config

If you ever think "oh, that's silly, it would be so much easier if I just ..."
it will almost never actually be easier if you just ...

The Deployment Story

disk-image-builder builds images and uploads to glance
Heat drives Nova/Ironic
Heat delivers metadata to os-collect-config
os-collect-config applies any in-instance changes needed

The Update Story

Heat magically just updates things

BUT I ALREADY USE ?????

(screw you guys, I'm going home)

Yeah, I do to

(I use puppet and ansible myself)

That's fine - use them - it's a cloud!

This is supposed to be empowering, not enforcing

Whatever you want!

Heat to deploy and update images, os-*-*config for config
Heat to deploy images, ansible to update images, puppet for config
Ansible to deploy base image + packages, salt to update packages, chef for config
juju to deploy ... nah, I'm just kidding

The New Update Story

Ansible takes over for upgrades

Ansible for Orchestration

About Ansible

Open Source System Management tool
Written in Python
Sequence of steps to perform
Works over SSH
Incremental Adoption

ad-hoc operation

ansible '*' -m shell -p uptime

YAML Syntax


- hosts: '*.slave.openstack.org'
  tasks:
    - shell: 'rm -rf ~jenkins/workspace/*{{ project }}*'

That's executed:

ansible-playbook -f 10 /etc/ansible/clean_workspaces.yaml --extra-vars "project=$PROJECTNAME"

Ansible Organization

modules
plays
playbooks
roles

Use Ansible to Run Puppet!

puppet module

def main():
    module = AnsibleModule(argument_spec=dict(
        timeout=dict(default="30m"),
        puppetmaster=dict(required=True),
        show_diff=dict(default=False, aliases=['show-diff'], type='bool'),
    ))
    p = module.params

    puppet_cmd = module.get_bin_path("puppet", False)
    if not puppet_cmd:
        module.fail_json(msg="Could not find puppet. Please ensure it is installed.")

puppet module (cont)


    cmd = ("timeout -s 9 %(timeout)s %(puppet_cmd)s agent --onetime"
           " --server %(puppetmaster)s"
           " --ignorecache --no-daemonize --no-usecacheonfailure --no-splay"
           " --detailed-exitcodes --verbose") % dict(
               timeout=pipes.quote(p['timeout']), puppet_cmd=PUPPET_CMD,
               puppetmaster=pipes.quote(p['puppetmaster']))
    if p['show_diff']:
        cmd += " --show-diff"
    rc, stdout, stderr = module.run_command(cmd)

Please. Everyone. Marvel at the following logic


    if rc == 0:  # success
        module.exit_json(rc=rc, changed=False, stdout=stdout)
    elif rc == 1:
        # rc==1 could be because it's disabled OR there was a compilation failure
        disabled = "administratively disabled" in stdout
        if disabled:
            msg = "puppet is disabled"
        else:
            msg = "puppet compilation failed"
        module.fail_json(rc=rc, disabled=disabled, msg=msg, stdout=stdout, stderr=stderr)
    elif rc == 2:  # success with changes
        module.exit_json(changed=True)
    elif rc == 124:  # timeout
        module.exit_json(rc=rc, msg="%s timed out" % cmd, stdout=stdout, stderr=stderr)
    else:  # failure
        module.fail_json(rc=rc, msg="%s failed" % (cmd), stdout=stdout, stderr=stderr)

puppet play


- name: run puppet
  puppet:
    puppetmaster: "{{puppetmaster}}"

puppet role

roles/remote_puppet/tasks/main.yml

remote puppet playbook


- hosts: git0*
  gather_facts: false
  max_fail_percentage: 1
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }
- hosts: review.openstack.org
  gather_facts: false
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }
- hosts: "!review.openstack.org:!git0*:!afs*"
  gather_facts: false
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }

ansible inventory

List of servers to operate on
Optionally variables associated with each server
Optional groups of servers
Simple file in /etc/ansible/hosts
Dynamic executable that returns JSON

Simple inventory

review.openstack.org
git01.openstack.org
git02.openstack.org
pypi.dfw.openstack.org
pypi.iad.openstack.org

[pypi]
pypi.dfw.openstack.org
pypi.iad.openstack.org

[git]
git01.openstack.org
git02.openstack.org

ansible inventory from puppet certs


import json
import subprocess

output = [
    x.split()[1][1:-1] for x in subprocess.check_output(
        ["puppet","cert","list","-a"]).split('\n')
    if x.startswith('+')
]

data = {
    '_meta': {'hostvars': dict()},
    'ungrouped': output,
}
print json.dumps(data, sort_keys=True, indent=2)

Ansible for Cloud Management

ansible and OpenStack

Ansible modules are just python
playbooks are lists of steps to take
Have plays/roles that provision servers
Infrastructure as code - for real!

Consider this data


pypi:
  image_name: Ubuntu 12.04.4
  flavor_ram: 2048
  provision_group: ubuntu_hosts
  volumes:
    - size: 200
      mount: /srv
  hosts:
    pypi.dfw:
      region: DFW
    pypi.iad:
      region: IAD
    pypi.ord:
      region: ORD
    pypi.region-b.geo-1:
      cloud: hp

Steps to launch a node

Create a compute instance
Wait for instance to exist
Create a floating IP
Attach floating IP to instance
Create one or more volumes
Attach volumes to instance
Wait for SSH to work
On host, format each volume
On host, mount each volume
On host, install config management software
On host, run config management software

Launch a node


---
- name: Launch Node
  os_compute:
      cloud: "{{ cloud }}"
      region_name: "{{ region_name }}"
      name: "{{ name }}"
      image_name: "{{ image_name }}"
      flavor_ram: "{{ flavor_ram }}"
      flavor_include: "{{ flavor_include }}"
      meta:
          group: "{{ group }}"
      key_name: "{{ launch_keypair }}"
  register: node
- name: Create volumes
  os_volume:
      cloud: "{{ cloud }}"
      size: "{{ item.size }}"
      display_name: "{{ item.display_name }}"
  with_items: volumes
- name: Attach volumes
  os_compute_volume:
      cloud: "{{ cloud }}"
      server_id: "{{ node.id }}"
      volume_name: "{{ item.display_name }}"
  with_items: volumes
  register: attached_volumes
- debug: var=attached_volumes
- name: Re-request server to get up to date metadata after the volume loop
  os_compute_facts:
      cloud: "{{ cloud }}"
      name: "{{ name }}"
  when: attached_volumes.changed
- name: Wait for SSH to work
  wait_for: host={{ node.openstack.interface_ip }} port=22
  when: node.changed == True
- name: Add SSH host key to known hosts
  shell: ssh-keyscan "{{ node.openstack.interface_ip|quote }}" >> ~/.ssh/known_hosts
  when: node.changed == True
- name: Add all instance public IPs to host group
  add_host:
      name: "{{ node.openstack.interface_ip }}"
      groups: "{{ provision_group }}"
      openstack: "{{ node.openstack }}"
  when: attached_volumes|length == 0
- name: Add all instance public IPs to host and volumes group
  add_host:
      name: "{{ node.openstack.interface_ip }}"
      groups: "{{ provision_group }},hasvolumes"
      openstack: "{{ node.openstack }}"
  when: attached_volumes|length != 0

Cloud based inventory

Just ask the cloud for the inventory
All of the meta-data the cloud knows is available


      "pypi.dfw.openstack.org": {
        "ansible_ssh_host": "23.253.237.8",
        "openstack": {
          "HUMAN_ID": true,
          "NAME_ATTR": "name",
          "OS-DCF:diskConfig": "MANUAL",
          "OS-EXT-STS:power_state": 1,
          "OS-EXT-STS:task_state": null,
          "OS-EXT-STS:vm_state": "active",
          "accessIPv4": "23.253.237.8",
          "accessIPv6": "2001:4800:7817:104:d256:7a33:5187:7e1b",
          "addresses": {
            "private": [
              {
                "addr": "10.208.195.50",
                "version": 4
              }
            ],
            "public": [
              {
                "addr": "23.253.237.8",
                "version": 4
              },
              {
                "addr": "2001:4800:7817:104:d256:7a33:5187:7e1b",
                "version": 6
              }
            ]
          },
          "cloud": "rax",
          "config_drive": "",
          "created": "2014-09-05T15:32:14Z",
          "flavor": {
            "id": "performance1-4",
            "links": [
              {
                "href": "https://dfw.servers.api.rackspacecloud.com/610275/flavors/performance1-4",
                "rel": "bookmark"
              }
            ],
            "name": "4 GB Performance"
          },
          "hostId": "adb603d4566efe0392756c76dab38ffcba22099368837c7973321e77",
          "human_id": "pypidfwopenstackorg",
          "id": "de672205-9245-46b6-b3df-489ccf9e0c17",
          "image": {
            "id": "928e709d-35f0-47eb-b296-d18e1b0a76b7",
            "links": [
              {
                "href": "https://dfw.servers.api.rackspacecloud.com/610275/images/928e709d-35f0-47eb-b296-d18e1b0a76b7",
                "rel": "bookmark"
              }
            ]
          },
          "interface_ip": "23.253.237.8",
          "key_name": "launch-node-root",
          "links": [
            {
              "href": "https://dfw.servers.api.rackspacecloud.com/v2/610275/servers/de672205-9245-46b6-b3df-489ccf9e0c17",
              "rel": "self"
            },
            {
              "href": "https://dfw.servers.api.rackspacecloud.com/610275/servers/de672205-9245-46b6-b3df-489ccf9e0c17",
              "rel": "bookmark"
            }
          ],
          "metadata": {},
          "name": "pypi.dfw.openstack.org",
          "networks": {
            "private": [
              "10.208.195.50"
            ],
            "public": [
              "23.253.237.8",
              "2001:4800:7817:104:d256:7a33:5187:7e1b"
            ]
          },
          "progress": 100,
          "region": "DFW",
          "status": "ACTIVE",
          "tenant_id": "610275",
          "updated": "2014-09-05T15:32:49Z",
          "user_id": "156284",
          "volumes": [
            {
              "HUMAN_ID": false,
              "NAME_ATTR": "name",
              "attachments": [
                {
                  "device": "/dev/xvdb",
                  "host_name": null,
                  "id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8",
                  "server_id": "de672205-9245-46b6-b3df-489ccf9e0c17",
                  "volume_id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8"
                }
              ],
              "availability_zone": "nova",
              "bootable": "false",
              "created_at": "2014-09-05T14:37:42.000000",
              "device": "/dev/xvdb",
              "display_description": null,
              "display_name": "pypi.dfw.openstack.org/main01",
              "human_id": null,
              "id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8",
              "metadata": {
                "readonly": "False",
                "storage-node": "1845027a-5e07-47a1-9572-3eea4716f726"
              },
              "os-vol-tenant-attr:tenant_id": "610275",
              "size": 200,
              "snapshot_id": null,
              "source_volid": null,
              "status": "in-use",
              "volume_type": "SATA"
            }
          ]
        }
      },

Who am I?

What are we going to talk about?

What is TripleO?

Velocity

OpenStack is not a virtualization layer, it's an abstraction layer.

What if your developers aren't writing Angry Birds?

What if your developers aren't developers?

If infrastructure is code ...

What if your developers are operators?

Why wouldn't you give your operators the same power as your developers?

Ironic is a service capable of managing and provisioning physical machines.

Nova + Ironic

Diversity is good

What can you do?

anything - it's a cloud

What the heck has the TripleO team been doing then?

TripleO

Community Developed

Avoid distro and config management religion

Opinionated

Major Components

Lesson from os-*-config

Lesson from os-*-config

Lesson from os-*-config

The Deployment Story

The Update Story

BUT I ALREADY USE ?????

(screw you guys, I'm going home)

Yeah, I do to

(I use puppet and ansible myself)

That's fine - use them - it's a cloud!

This is supposed to be empowering, not enforcing

Whatever you want!

The New Update Story

Ansible for Orchestration

About Ansible

ad-hoc operation

YAML Syntax

Ansible Organization

Use Ansible to Run Puppet!

puppet module

puppet module (cont)

puppet play

puppet role

remote puppet playbook

ansible inventory

Simple inventory

ansible inventory from puppet certs

Ansible for Cloud Management

ansible and OpenStack

Consider this data

Steps to launch a node

Launch a node

Cloud based inventory

Ironic is a service capable of
managing and provisioning
physical machines.