OpenStack

TripleO and Ansible

Monty Taylor

http://inaugust.com/talks/tripleo-ansible.html

twitter: @e_monty

Who am I?

  • Distinguished Technologist at HP
  • OpenStack Technical Committee
  • OpenStack Foundation Board of Directors
  • OpenStack Infra Core Team

What are we going to talk about?

  • TripleO
  • Ironic
  • Ansible

What is TripleO?

  • an idea
  • OpenStack On OpenStack
  • use Ironic-based OpenStack to Operate Openstack

Velocity

  • Cloud Empowers developers
  • Cloud Enables Increased Velocity
  • Cloud Drives Agility

OpenStack is not a virtualization layer, it's an abstraction layer.

What if your developers aren't writing Angry Birds?

What if your developers aren't developers?

If infrastructure is code ...

What if your developers are operators?

Why wouldn't you give your operators the same power as your developers?

Don't you like them?

Ironic is a service capable of
managing and provisioning
physical machines.

Nova + Ironic

Same but different 

$ ironic node-create -d pxe_ipmitool \
  -i ipmi_username=admin -i ipmi_password=fake -i ipmi_address=10.1.2.3 \
  -p cpus=4 -p memory_mb=8192 -p local_gb=500 \
  -e note='spare server'
+--------------+------------------------------------------------------------+
| Property     | Value                                                      |
+--------------+------------------------------------------------------------+
| chassis_uuid | None                                                       |
| driver       | pxe_ipmitool                                               |
| driver_info  | {u'ipmi_address': u'10.1.2.3', u'ipmi_username': u'admin', |
|              | u'ipmi_password': u'fake'}                                 |
| extra        | {u'note': u'spare server'}                                 |
| properties   | {u'memory_mb': u'8192', u'local_gb': u'500', u'cpus': u'4'}|
| uuid         | 7a1ce8d0-9679-4d87-8f54-b11f6e8adb8f                       |
+--------------+------------------------------------------------------------+
          

$ tail -f /var/log/nova/n-cpu.log
...
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free ram (MB): 8192
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free disk (GB): 500
2014-05-01 03:47:05.878 AUDIT nova.compute.resource_tracker [-]
    Free VCPUS: 4

Diversity is good

  • Cloud providers: HP, Rackspace, Dreamhost, Cloudwatt
  • Deployments: Public, Managed, Private
  • Compute drivers: virtual, bare metal, container
  • Ironic drivers: pxe/impi, ilo, HP OneView

What can you do?

anything - it's a cloud

What the heck has the TripleO team been doing then?

TripleO

  • Community developed
  • Opinionated
  • Avoid distro and config management religion
  • Be a usable real deployment we can gate on

Community Developed

  • Exist as part of the OpenStack project
  • Prove the story end to end
  • Subject to TC governance
  • Tighter feedback loop

Avoid distro and config management religion

  • rpm vs. deb - in the gate == rpm + deb
  • puppet vs. chef vs. salt vs. ansible == all of them
  • Choosing one excludes other folks from participating

Opinionated

  • Golden Images
  • Upgrade tied to HA
  • Target Continual Delivery
  • Drive fixes into OpenStack directly

Major Components

  • nova+ironic
  • heat
  • diskimage-builder
  • os-collect-config
  • os-apply-config
  • os-refresh-config

Lesson from os-*-config

Lesson from os-*-config

Lesson from os-*-config

If you ever think "oh, that's silly, it would be so much easier if I just ..."
it will almost never actually be easier if you just ...

The Deployment Story

  1. disk-image-builder builds images and uploads to glance
  2. Heat drives Nova/Ironic
  3. Heat delivers metadata to os-collect-config
  4. os-collect-config applies any in-instance changes needed

The Update Story

Heat magically just updates things

BUT I ALREADY USE ?????

(screw you guys, I'm going home)

Yeah, I do to

(I use puppet and ansible myself)

That's fine - use them - it's a cloud!

This is supposed to be empowering, not enforcing

Whatever you want!

  • Heat to deploy and update images, os-*-*config for config
  • Heat to deploy images, ansible to update images, puppet for config
  • Ansible to deploy base image + packages, salt to update packages, chef for config
  • juju to deploy ... nah, I'm just kidding

The New Update Story

Ansible takes over for upgrades

Ansible for Orchestration

About Ansible

  • Open Source System Management tool
  • Written in Python
  • Sequence of steps to perform
  • Works over SSH
  • Incremental Adoption

ad-hoc operation

ansible '*' -m shell -p uptime

YAML Syntax


- hosts: '*.slave.openstack.org'
  tasks:
    - shell: 'rm -rf ~jenkins/workspace/*{{ project }}*'

That's executed:

ansible-playbook -f 10 /etc/ansible/clean_workspaces.yaml --extra-vars "project=$PROJECTNAME"

Ansible Organization

  • modules
  • plays
  • playbooks
  • roles

Use Ansible to Run Puppet!

puppet module

def main():
    module = AnsibleModule(argument_spec=dict(
        timeout=dict(default="30m"),
        puppetmaster=dict(required=True),
        show_diff=dict(default=False, aliases=['show-diff'], type='bool'),
    ))
    p = module.params

    puppet_cmd = module.get_bin_path("puppet", False)
    if not puppet_cmd:
        module.fail_json(msg="Could not find puppet. Please ensure it is installed.")

puppet module (cont)


    cmd = ("timeout -s 9 %(timeout)s %(puppet_cmd)s agent --onetime"
           " --server %(puppetmaster)s"
           " --ignorecache --no-daemonize --no-usecacheonfailure --no-splay"
           " --detailed-exitcodes --verbose") % dict(
               timeout=pipes.quote(p['timeout']), puppet_cmd=PUPPET_CMD,
               puppetmaster=pipes.quote(p['puppetmaster']))
    if p['show_diff']:
        cmd += " --show-diff"
    rc, stdout, stderr = module.run_command(cmd)
Please. Everyone. Marvel at the following logic 

    if rc == 0:  # success
        module.exit_json(rc=rc, changed=False, stdout=stdout)
    elif rc == 1:
        # rc==1 could be because it's disabled OR there was a compilation failure
        disabled = "administratively disabled" in stdout
        if disabled:
            msg = "puppet is disabled"
        else:
            msg = "puppet compilation failed"
        module.fail_json(rc=rc, disabled=disabled, msg=msg, stdout=stdout, stderr=stderr)
    elif rc == 2:  # success with changes
        module.exit_json(changed=True)
    elif rc == 124:  # timeout
        module.exit_json(rc=rc, msg="%s timed out" % cmd, stdout=stdout, stderr=stderr)
    else:  # failure
        module.fail_json(rc=rc, msg="%s failed" % (cmd), stdout=stdout, stderr=stderr)

puppet play


- name: run puppet
  puppet:
    puppetmaster: "{{puppetmaster}}"

puppet role

roles/remote_puppet/tasks/main.yml

remote puppet playbook


- hosts: git0*
  gather_facts: false
  max_fail_percentage: 1
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }
- hosts: review.openstack.org
  gather_facts: false
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }
- hosts: "!review.openstack.org:!git0*:!afs*"
  gather_facts: false
  roles:
    - { role: remote_puppet, puppetmaster: puppetmaster.openstack.org }

ansible inventory

  • List of servers to operate on
  • Optionally variables associated with each server
  • Optional groups of servers
  • Simple file in /etc/ansible/hosts
  • Dynamic executable that returns JSON

Simple inventory

review.openstack.org
git01.openstack.org
git02.openstack.org
pypi.dfw.openstack.org
pypi.iad.openstack.org

[pypi]
pypi.dfw.openstack.org
pypi.iad.openstack.org

[git]
git01.openstack.org
git02.openstack.org

ansible inventory from puppet certs


import json
import subprocess

output = [
    x.split()[1][1:-1] for x in subprocess.check_output(
        ["puppet","cert","list","-a"]).split('\n')
    if x.startswith('+')
]

data = {
    '_meta': {'hostvars': dict()},
    'ungrouped': output,
}
print json.dumps(data, sort_keys=True, indent=2)

Ansible for Cloud Management

ansible and OpenStack

  • Ansible modules are just python
  • playbooks are lists of steps to take
  • Have plays/roles that provision servers
  • Infrastructure as code - for real!

Consider this data


pypi:
  image_name: Ubuntu 12.04.4
  flavor_ram: 2048
  provision_group: ubuntu_hosts
  volumes:
    - size: 200
      mount: /srv
  hosts:
    pypi.dfw:
      region: DFW
    pypi.iad:
      region: IAD
    pypi.ord:
      region: ORD
    pypi.region-b.geo-1:
      cloud: hp

Steps to launch a node

  1. Create a compute instance
  2. Wait for instance to exist
  3. Create a floating IP
  4. Attach floating IP to instance
  5. Create one or more volumes
  6. Attach volumes to instance
  7. Wait for SSH to work
  8. On host, format each volume
  9. On host, mount each volume
  10. On host, install config management software
  11. On host, run config management software

Launch a node


---
- name: Launch Node
  os_compute:
      cloud: "{{ cloud }}"
      region_name: "{{ region_name }}"
      name: "{{ name }}"
      image_name: "{{ image_name }}"
      flavor_ram: "{{ flavor_ram }}"
      flavor_include: "{{ flavor_include }}"
      meta:
          group: "{{ group }}"
      key_name: "{{ launch_keypair }}"
  register: node
- name: Create volumes
  os_volume:
      cloud: "{{ cloud }}"
      size: "{{ item.size }}"
      display_name: "{{ item.display_name }}"
  with_items: volumes
- name: Attach volumes
  os_compute_volume:
      cloud: "{{ cloud }}"
      server_id: "{{ node.id }}"
      volume_name: "{{ item.display_name }}"
  with_items: volumes
  register: attached_volumes
- debug: var=attached_volumes
- name: Re-request server to get up to date metadata after the volume loop
  os_compute_facts:
      cloud: "{{ cloud }}"
      name: "{{ name }}"
  when: attached_volumes.changed
- name: Wait for SSH to work
  wait_for: host={{ node.openstack.interface_ip }} port=22
  when: node.changed == True
- name: Add SSH host key to known hosts
  shell: ssh-keyscan "{{ node.openstack.interface_ip|quote }}" >> ~/.ssh/known_hosts
  when: node.changed == True
- name: Add all instance public IPs to host group
  add_host:
      name: "{{ node.openstack.interface_ip }}"
      groups: "{{ provision_group }}"
      openstack: "{{ node.openstack }}"
  when: attached_volumes|length == 0
- name: Add all instance public IPs to host and volumes group
  add_host:
      name: "{{ node.openstack.interface_ip }}"
      groups: "{{ provision_group }},hasvolumes"
      openstack: "{{ node.openstack }}"
  when: attached_volumes|length != 0

Cloud based inventory

  • Just ask the cloud for the inventory
  • All of the meta-data the cloud knows is available 

      "pypi.dfw.openstack.org": {
        "ansible_ssh_host": "23.253.237.8",
        "openstack": {
          "HUMAN_ID": true,
          "NAME_ATTR": "name",
          "OS-DCF:diskConfig": "MANUAL",
          "OS-EXT-STS:power_state": 1,
          "OS-EXT-STS:task_state": null,
          "OS-EXT-STS:vm_state": "active",
          "accessIPv4": "23.253.237.8",
          "accessIPv6": "2001:4800:7817:104:d256:7a33:5187:7e1b",
          "addresses": {
            "private": [
              {
                "addr": "10.208.195.50",
                "version": 4
              }
            ],
            "public": [
              {
                "addr": "23.253.237.8",
                "version": 4
              },
              {
                "addr": "2001:4800:7817:104:d256:7a33:5187:7e1b",
                "version": 6
              }
            ]
          },
          "cloud": "rax",
          "config_drive": "",
          "created": "2014-09-05T15:32:14Z",
          "flavor": {
            "id": "performance1-4",
            "links": [
              {
                "href": "https://dfw.servers.api.rackspacecloud.com/610275/flavors/performance1-4",
                "rel": "bookmark"
              }
            ],
            "name": "4 GB Performance"
          },
          "hostId": "adb603d4566efe0392756c76dab38ffcba22099368837c7973321e77",
          "human_id": "pypidfwopenstackorg",
          "id": "de672205-9245-46b6-b3df-489ccf9e0c17",
          "image": {
            "id": "928e709d-35f0-47eb-b296-d18e1b0a76b7",
            "links": [
              {
                "href": "https://dfw.servers.api.rackspacecloud.com/610275/images/928e709d-35f0-47eb-b296-d18e1b0a76b7",
                "rel": "bookmark"
              }
            ]
          },
          "interface_ip": "23.253.237.8",
          "key_name": "launch-node-root",
          "links": [
            {
              "href": "https://dfw.servers.api.rackspacecloud.com/v2/610275/servers/de672205-9245-46b6-b3df-489ccf9e0c17",
              "rel": "self"
            },
            {
              "href": "https://dfw.servers.api.rackspacecloud.com/610275/servers/de672205-9245-46b6-b3df-489ccf9e0c17",
              "rel": "bookmark"
            }
          ],
          "metadata": {},
          "name": "pypi.dfw.openstack.org",
          "networks": {
            "private": [
              "10.208.195.50"
            ],
            "public": [
              "23.253.237.8",
              "2001:4800:7817:104:d256:7a33:5187:7e1b"
            ]
          },
          "progress": 100,
          "region": "DFW",
          "status": "ACTIVE",
          "tenant_id": "610275",
          "updated": "2014-09-05T15:32:49Z",
          "user_id": "156284",
          "volumes": [
            {
              "HUMAN_ID": false,
              "NAME_ATTR": "name",
              "attachments": [
                {
                  "device": "/dev/xvdb",
                  "host_name": null,
                  "id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8",
                  "server_id": "de672205-9245-46b6-b3df-489ccf9e0c17",
                  "volume_id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8"
                }
              ],
              "availability_zone": "nova",
              "bootable": "false",
              "created_at": "2014-09-05T14:37:42.000000",
              "device": "/dev/xvdb",
              "display_description": null,
              "display_name": "pypi.dfw.openstack.org/main01",
              "human_id": null,
              "id": "c6f5229c-1cc0-47c4-aab7-60db1f6cf8e8",
              "metadata": {
                "readonly": "False",
                "storage-node": "1845027a-5e07-47a1-9572-3eea4716f726"
              },
              "os-vol-tenant-attr:tenant_id": "610275",
              "size": 200,
              "snapshot_id": null,
              "source_volid": null,
              "status": "in-use",
              "volume_type": "SATA"
            }
          ]
        }
      },

Thank you!

Monty Taylor

twitter: @e_monty

http://inaugust.com/talks/tripleo-ansible.html

Creative Commons License
Licensed under a Creative Commons Attribution 4.0 International License .
Source code available at http://git.inaugust.com/cgit/inaugust.com