Date: Thu, 31 Aug 2017 12:32:23 -0500 Subject: Add ansible openstack talk for Huawei Connect --- src/talks/ansible-openstack.hbs | 502 ++++++++++++++++++++++++++++++++++++++++ src/talks/global-interop.hbs | 250 ++++++-------------- 2 files changed, 579 insertions(+), 173 deletions(-) create mode 100644 src/talks/ansible-openstack.hbs diff --git a/src/talks/ansible-openstack.hbs b/src/talks/ansible-openstack.hbs new file mode 100644 index 0000000..4b2af33 --- /dev/null +++ b/src/talks/ansible-openstack.hbs @@ -0,0 +1,502 @@ + + + + + + + Using Ansible with OpenStack + + + + +

Who am I?

Office of Technology

Zuul

Ansible

+ +

Who am I?

Technical Committee

Developer Infrastructure Core Team

Former Foundation Board of Directors

PTL of shade project

+ +

Ansible

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

+ +

Why Ansible is Great

simple declarative YAML syntax
list of tasks to perform - easy to debug
agentless - don't infect machines under management
Open Source - thousands of contributors worldwide

+ +

Ansible Overview

task - an action to perform
module - code that exposes a type of action
role - reusable collection of tasks to acomplish a goal
play - tasks and roles run on a specific host or hosts
inventory - list of locations available to run plays on
playbook - file containing one or more plays

+ +

Ansible Simple Example

simple.yaml


+- hosts: all
+  tasks:
+  - name: Print hostname of server
+    command: hostname
+

A playbook containing a play that will run + against all hosts in the inventory. It has one task + that uses the command module to run the "hostname" command. +

+ +

Ansible Longer Example

longer.yaml


+- hosts: git
+  roles:
+    - create-mirror-locations
+- hosts: code-review.example.com
+  roles:
+    - create-repositories
+

A playboook containing two plays. The first runs + a role called "create-mirror-locations" against a group of hosts + from the inventory called "git". Then it runs a role + called "create-repositories" against a host from the inventory + called "code-review.example.com". +

+ +

Ansible Modules

Usually written in Python
Can do anything, not limited to executing commands on a server
Manipulate Network Switch Config (ansible-network)
Build Containers (ansible-container)
Interact with REST APIs

+ +

Ansible OpenStack Modules

http://docs.ansible.com/ansible/list_of_cloud_modules.html#openstack
Focused on consuming OpenStack APIs
Some are end-user focused
Some are deployer focused

+ +

Interop - Work on All OpenStack Clouds

Ansible's OpenStack modules work the same on every OpenStack cloud

+ Don't let the existence of Rackspace modules confuse you. + The OpenStack modules work just great on Rackspace

+ +

Work Around Deployer Differences ... To a Point

Herculean effort is made to hide a giant amount of differences
Even so - some bugs are EWONTFIX

+ A provider decided to redefine the OpenStack Availability Zone concept + complete with incompatible API changes. That is unsupportable.

PS. Don't do that

+ +

Based on shade library

OpenStack python library to abstract deployment differences
designed for multi-cloud
simple to use
massive scale +
- optional advanced features to handle >20k servers a day
Initial logic/design extracted from OpenStack Infra's nodepool
Turned in to library for re-use in Ansible

+ +

Integration Testing

Every shade patch is integration tested in OpenStack Infra
Every shade patch tests the ansible modules in OpenStack Infra
Support coming very soon to test PRs to ansible modules in OpenStack Infra

+ +

Let's Take a Few Steps Back

OpenStack in Ansible and Multi-cloud Operations are easy...

but you need to know a few things.

Module structure
Terminology
Config

+ +

Module Structure

All modules start with os_
End-user oriented modules are named for the resource +
- os_image
- os_sever
+
Operator oriented modules are named for the service +
- os_nova_flavor
- os_keystone_domain
+
If it could go both ways, we prefer user orientation +
- os_user
- os_group
+
If more than one service provides a resource, they DTRT +
- os_security_group
- os_floating_ip
+

+ +

OpenStack Dynamic Inventory Script

github.com/ansible/ansible/contrib/inventory/openstack.py
Uses all hosts in all clouds as one inventory
Excludes hosts without IPs
Makes a bunch of auto-groups (flavor, image, az, region, cloud)
Makes groups for metadata['groups']

New inventory plugin coming in Ansible 2.4

+ +

Modules for All OpenStack Resources are Welcome Upstream

If there is a thing you want to manage that doesn't have a module, + add one
We're very welcoming/accepting
Same goes for features to existing modules

+ +

To Serve All Users, We Have to be Strict

Modules that touch OpenStack APIs must do so via shade library
In user modules, vendor differences should be hidden
In operator modules, exposing them is ok

+ +

clouds.yaml

Information about the clouds you want to connect to is stored in a + file called clouds.yaml.
In OpenStack Pike release, Horizon provides downloadable clouds.yaml
Both yaml and yml are acceptable.
clouds.yaml can be in your homedir: ~/.config/openstack/clouds.yaml or system-wide: /etc/openstack/clouds.yaml.
Information in your homedir, if it exists, takes precedence.
Full docs on clouds.yaml are at https://docs.openstack.org/developer/os-client-config/

+ +

What about Mac and Windows?

USER_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: ~/.config/openstack
OSX: ~/Library/Application Support/openstack
Windows: C:\\Users\\USERNAME\\AppData\\Local\\OpenStack\\openstack

SITE_CONFIG_DIR is different on Linux, OSX and Windows.

Linux: /etc/openstack
OSX: /Library/Application Support/openstack
Windows: C:\\ProgramData\\OpenStack\\openstack

+ +

Config Terminology

For multi-cloud, think of two types:

profile - Facts about the cloud that are true for everyone
cloud - Information specific to a given user

+ +

Remember your Execution Context!

ansible executes code on remote systems
clouds.yaml needs to be on the host where the modules run
auth information can be passed to modules directly
other config (currently) MUST go in clouds.yaml
Ability to pass entire cloud config to module coming soon

+ +

basic clouds.yaml for the example code

Simple example of a clouds.yaml

Config for a named cloud "my-citycloud"
Reference a well-known "named" profile: citycloud
os-client-config has a built-in list of profiles at https://docs.openstack.org/developer/os-client-config/vendor-support.html
Vendor profiles contain various advanced config
cloud name can match profile name (using different names for clarity)


+clouds:
+  my-citycloud:
+    profile: citycloud
+    auth:
+      username: mordred
+      project_id: 65222a4d09ea4c68934fa1028c77f394
+      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
+      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38
+

Where's the password?

+ +

secure.yaml

Optional additional file just like clouds.yaml
Values overlaid on clouds.yaml
Useful if you want to protect secrets more stringently

+ +

Example secure.yaml

No, my password isn't XXXXXXXX
cloud name should match clouds.yaml
Optional - I actually keep mine in my clouds.yaml


+clouds:
+  my-citycloud:
+    auth:
+      password: XXXXXXXX
+

+ +

more clouds.yaml

More information can be provided.

Use v3 of the identity API - even if others are present
Use https://image-ca-ymq-1.vexxhost.net/v2 for image API instead of what's in the catalog


+my-vexxhost:
+  identity_api_version: 3
+  image_endpoint_override: https://image-ca-ymq-1.vexxhost.net/v2
+  profile: vexxhost
+  auth:
+    user_domain_id: default
+    project_domain_id: default
+    project_name: d8af8a8f-a573-48e6-898a-af333b970a2d
+    username: 0b8c435b-cc4d-4e05-8a47-a2ada0539af1
+

+ +

Much more complex clouds.yaml example

Not using a profile - all settings included
In the ams01 region there are two networks with undiscoverable qualities
Each one are labeled here so choices can be made
Any of the settings can be specific to a region if needed
region settings override cloud settings
cloud does not support floating-ips

+ +


+my-internap:
+  auth:
+    auth_url: https://identity.api.cloud.iweb.com
+    username: api-55f9a00fb2619
+    project_name: inap-17037
+  identity_api_version: 3
+  floating_ip_source: None
+  regions:
+  - name: ams01
+    values:
+      networks:
+      - name: inap-17037-WAN1654
+        routes_externally: true
+        default_interface: true
+      - name: inap-17037-LAN3631
+        routes_externally: false

+ +

Extra Variables to Control Inventory Behavior

+
+
expand_hostvars
+
whether to make extra API calls to fill out additional + information about each server +
+
+
+
+
use_hostnames
+
changes the behavior from registering every host with its + UUID and making a group of its hostname to only doing this if the + hostname in question has more than one server +
+
+
+
+
fail_on_errors
+
causes the inventory to fail and return no hosts if one + cloud has failed
+
+

+ +

ansible:
+  use_hostnames: False
+  expand_hostvars: True
+  fail_on_errors: True

Test Your Config

---
+- hosts: localhost
+  tasks:
+  - os_auth:
+      cloud: "{{ item.cloud }}"
+      region_name: "{{ item.region }}"
+    with_items:
+    - cloud: my-vexxhost
+      region: ca-ymq-1
+    - cloud: my-citycloud
+      region: Buf1
+    - cloud: my-internap
+      region: ams01
+

+ +

More Interesting


+- hosts: localhost
+  tasks:
+  - os_server:
+      name: "my-server"
+      cloud: "{{ item.cloud }}"
+      region_name: "{{ item.region }}"
+      image: "{{ item.image }}"
+      flavor: "{{ item.flavor }}"
+      auto_ip: true
+    with_items:
+    - cloud: my-vexxhost
+      region: ca-ymq-1
+      image: Ubuntu 16.04.1 LTS [2017-03-03]
+      flavor: v1-standard-4
+    - cloud: my-citycloud
+      region: Buf1
+      image: Ubuntu 16.04 Xenial Xerus
+      flavor: 4C-4GB-100GB
+    - cloud: my-internap
+      region: ams01
+      image: Ubuntu 16.04 LTS (Xenial Xerus)
+      flavor: A1.4

+ +

Check That There is an Inventory

+ python ~/src/github.com/ansible/ansible/contrib/inventory/openstack.py --list +

+ +

Cleanup After Ourselves


+- hosts: localhost
+  tasks:
+  - os_server:
+      cloud: "{{ item.cloud }}"
+      region_name: "{{ item.region }}"
+      name: my-server
+      state: absent
+    with_items:
+    - cloud: my-vexxhost
+      region: ca-ymq-1
+    - cloud: my-citycloud
+      region: Buf1
+    - cloud: my-internap
+      region: ams01
+

+ +

Check out Ansible Cloud Launcher

https://git.openstack.org/cgit/openstack/ansible-role-cloud-launcher

Role for launching resources on an OpenStack Cloud

+ +

Check out Linch-pin

http://linch-pin.readthedocs.io/en/develop/

From our friends in CentOS

Linch-pin provides a collection of Ansible playbooks for provisioning, decommissioning, and managing resources across multiple infrastructures. The main goal of linch-pin is to facilitate provisioning and orchestration of resources in a multi-cloud environment through a topology file.

+ + + diff --git a/src/talks/global-interop.hbs b/src/talks/global-interop.hbs index 90f17c6..52de08f 100644 --- a/src/talks/global-interop.hbs +++ b/src/talks/global-interop.hbs @@ -44,12 +44,23 @@ so that I can comply with regulatory demands.

+ I want to write some new Cloud Native applications while continuing + to make use of my existing investments.

+ +

+ I need to move faster than before AND I need increased stability at + the same time.

THIS WORKS

I'm doing it myself as we speak

20k VMs per-day

20 Cloud Regions in 9 clouds

Only using OpenStack APIs

@@ -61,20 +72,27 @@

Tooling, Automation and CI for OpenStack Project

Tooling and Automation for the development of OpenStack Project

2000 Developers

>2500 Developers

Gated Commits

Every commit is fully integration tested (twice) before landing

Gated Changes

Every change goes through enforced automatic full integration testing + (at least twice) before landing

Each Test Runs on a Single Use Cloud Slave

Integration Testing

Install and run one or more clouds then validate that those clouds + work.

+ +

Each Test Job Runs on Single Use Cloud Servers

This is that "cloud scale out" part

@@ -82,8 +100,8 @@

2 KJPH (kilo-jobs per hour)

Our VMs are everywhere

Our Cloud Servers are everywhere

Public Clouds

Rackspace: Dallas, Chicago, DC

Gerrit

- Traditional 'Enterprise' Java Application
Single Nova VM, Cinder Volume
Code Review and Code Hosting
Traditional 'Enterprise' Java Application
Single OpenStack Nova VM, Cinder Volume
Scale out farm of git replicas

nodepool

Cloud Native
- Purpose built in Python
Cloud Native resource manager
Purpose built in Python
Keeps a pool of ready to go nodes
Multi-cloud
Fully elastic - responds to demand

How do we do this?

Control plane

http://git.openstack.org/cgit/openstack-infra/system-config

All server config management in git
Puppet manages the servers: puppet apply
Ansible runs puppet: ansible puppet module
Ansible OpenStack Dynamic Inventory
Only thing not public are keys and secrets

os-client-config

http://git.openstack.org/cgit/openstack/os-client-config

https://docs.openstack.org/os-client-config/latest/

A library to handle config information for openstack clients

Tracks differences in vendors that can't be discovered

In use in python-openstackclient, shade and ansible

https://docs.openstack.org/os-client-config/latest/user/vendor-support.html

os-client-config

~/.config/openstack/clouds.yaml

-clouds:
-  citycloud:
-     profile: citycloud
-     auth:
-      username: mordred
-      password: XXXXXXXXXXXXXXXXXXXXX
-      project_id: 65222a4d09ea4c68934fa1028c77f394
-      user_domain_id: d0919bd5e8d74e49adf0e145807ffc38
-      project_domain_id: d0919bd5e8d74e49adf0e145807ffc38
-    regions:
-      - Kna1
-      - Sto2
-      - Lon1
-  dreamcompute:
-    profile: dreamhost
-    auth:
-      username: montay6
-      project_name: dhc2111978
-      password: XXXXXXXXXXXXX
-    region_name: RegionOne
-

shade

http://git.openstack.org/cgit/openstack-infra/shade

A library to wrap business logic around client libraries

-cloud.create_image('image-name', filename='image-filename.qcow2')
-cloud.create_server('my-server', image='immage-name', auto_ip=True)
-

In use in Infra Nodepool and ansible

ansible

Based on shade

-- os_keypair:
-    cloud: citycloud
-    name: mordred
-    public_key_file: ~/.ssh/id_rsa.pub
-- os_image:
-    cloud: citycloud
-    name: Monty Ubuntu
-    file: ubuntu.vhd
-- os_server:
-    cloud: citycloud
-    name: my-server
-    flavor_ram: 1024
-    image: Monty Ubuntu
-

ansible

Add my keypair to 12 clouds


-- os_keypair:
-    cloud: "{{ item.cloud }}"
-    region_name: "{{ item.region_name }}"
-    name: mordred
-    public_key_file: ~/.ssh/id_rsa.pub
-    with-items:
-    - {cloud: vexxhost, region_name: ca-ymq-1}
-    - {cloud: ovh, region_name: GRA1}
-    - {cloud: ustack, region_name: bj1}
-    - {cloud: citycloud, region_name: Sto2}
-    - {cloud: internap, region_name: sin01}
-    - {cloud: fuga, region_name: cystack}
-    - {cloud: datacentred, region_name: sal01}
-    - {cloud: rax, region_name: SYD}
-    - {cloud: clouda, region_name: regionOne}
-    - {cloud: auro, region_name: van1}
-    - {cloud: zetta, region_name: no-osl1}
-    - {cloud: kiss, region_name: region1}
-

nodepool

zuul

Use shade to treat all cloud regions as one giant cloud
diskimage-builder makes identical base images for each
Pre-cache network artifacts in disk images
glean instead of cloud-init to handle no-DHCP on Rackspace
Pre-spins warm pool - always keep min-ready number of nodes
"Test it like you deploy it"
The Gatekeeper
Microservices but with a centralized scheduler
Responds to code review events from Gerrit
Runs Ansible content on nodes from nodepool
Use production Ansible to test proposed changes

Talk about Zuul Tomorrow 15:50-16:15 on stage OP5

OpenStack Works!

Why should you care?

Is interop really important?

Is interoperability really important?

@@ -269,31 +179,26 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

Run it where you want

Geography
Delivery Model

- -

Geography

Put it near your users
Don't put it in an US data center
Avoid putting it in an US data center
Locality Requirements
What if you have customers in China?
What if you also have customers in Sweden?
What if you also have customers in Sweden? + New Zealand? + Brazil? +

Delivery Model

Run it how you want

Public
Hosted Private
Managed Private On-Premise
Run your own
Run your own with a partner/vendor
Run your own on your own

@@ -312,7 +217,7 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

Anybody REALLY think that should run in a - Public Cloud owned by a US company?

+ Public Cloud owned and operated by a US company?

@@ -328,34 +233,21 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

Chinese OpenStack

Huawei

sorted alphabetically

99 Cloud

China Mobile

China National Offshore Oil Corp

China Telecom

China Unicom

EasyStack

Inspur

Sinorail

State Grid of China

T2 Cloud

Tencent

UMCloud

UnitedStack

ZTE

OpenStack in Asia

Huawei: First Chinese Platinum Member of OpenStack Foundation

+ Only 8 of the 24 Gold Member Companies are non-Asian

European Public OpenStack

European OpenStack Public Clouds

CityCloud: Sweden, UK, US

Datacentred: UK

Elastx: Sweden

Enter Cloud Suite: Italy, Germany

Fugo: Netherlands

Internap: Netherlands, US, Singapore

OTC: Frankfurt

OTC: Germany

OVH: France, US

Switch: Switzerland

Ultimum: Czech Republic

@@ -364,13 +256,15 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

Other Public OpenStack

Other Public OpenStack Clouds

That I personally have an account on

Auro: Vancouver

Catalyst: New Zealand

Conoha: Japan, Singapore, US

Dreamhost: US

Ormuco: Canada + Federated

Ormuco: Canada, Federated Worldwide

Vexxhost: Canada

UnitedStack: China

@@ -379,13 +273,11 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

The OpenStack Project Believes all Participants are Equal

Constant work in progress, help us when we get this - wrong

We aren't going to remove something you need...

because we ARE you.

OpenStack isn't going to remove something you need...

because OpenStack IS you.

@@ -419,10 +311,22 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

It's all about the tools

Ansible
OpenShift
Terraform
Spinnaker
Zuul -- v3 is almost out and will blow your mind
Zuul

+ +

Ansible OpenStack Modules

Based on the API consumption code from nodepool: "shade"
Support all of the known OpenStack Public Clouds
Interoperable == you can use them with your cloud
Maintained by me

+ Talk about Ansible and OpenStack Today 16:00 - 17:00

@@ -442,8 +346,8 @@ cloud.create_server('my-server', image='immage-name', auto_ip=True)

China, and the rest of our Global Community, understands the power - of Open Collaboration on shared problems.

China understands the power of Open Collaboration on + shared problems.

-- cgit v1.2.3