. display in 68x24
.. display in 88x24

.. pygments yaml? (only file breaks (---) tinted)
.. slide on high level v3 changes
.. slide on nodepool

.. transition:: dissolve
   :duration: 0.4

Test Slide
==========
.. hidetitle::

.. ansi:: images/testslide.ans

Preshow
=======
.. hidetitle::

.. ansi:: images/cursor.ans images/cursor2.ans

Zuul
====
.. hidetitle::
.. ansi:: images/title.ans

Red Hat
=======
.. hidetitle::
.. container:: handout
   i work for

.. ansi:: images/redhat.ans


Ansible
=======
.. hidetitle::
.. ansi:: images/ansible.ans

Zuul
====
.. hidetitle::
.. ansi:: images/zuul.ans


What Zuul Does
==============

  * gated changes
  * one or more git repositories
  * integrated deliverable
  * testing like deployment

Underlying Philosophy
=====================

  * All changes flow through code review
  * Changes only land if they pass all tests
  * Computers are cheaper than humans

Ramifications of Philosophy
===========================

  * No direct push access for anyone
  * Software should be installable from source
  * Testing should be automated and repeatable
  * Developers write tests with their patches
  * Code always works

Getting to Gating
=================

No Tests / Manual Tests
=======================

  * No test automation exists or ...
  * Developer runs test suite before pushing code
  * Prone to developer skipping tests for "trivial" changes
  * Doesn't scale organizationally

Periodic Testing
================

  * Developers push changes directly to shared branch
  * CI system runs tests from time to time - report if things still work
  * "Who broke the build?"
  * Leads to hacks like NVIE model

Post-Merge Testing
==================

  * Developers push changes directly to shared branch
  * CI system is triggered by push - reports if push broke something
  * Frequently batched / rolled up
  * Easier to diagnose which change broke things
  * Reactive - the bad changes are already in

Pre-review Testing
==================

  * Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
  * CI system is triggered by code review change creation
  * Test results inform review decisions
  * Proactive - testing code before it lands
  * Reviewers can get bored waiting for tests
  * Only tests code as written, not potential result of merging code

Gating
======

  * Changes are pushed to code review
  * CI system is triggered by code review approval
  * CI system merges code IFF tests pass
  * Proactive - testing code before it lands
  * Future state resulting from merge of code is tested
  * Reviewers can fire-and-forget safely

Mix and Match
=============

  * Zuul supports all of those modes
  * Zuul users frequently combine them
  * Run pre-review (check) and gating (gate) on each change
  * Post-merge/post-tag for release/publication automation
  * Periodic for catching bitrot

Multi-repository integration
============================

  * Multiple source repositories are needed for deliverable
  * Future state to be tested is the future state of all involved repos

To test proposed future state
=============================

  * Get tip of each project. Merge appropriate change. Test.
  * Changes must be serialized, otherwise state under test is invalid.
  * Integrated deliverable repos share serialized queue

Speculative Execution
=====================

  * Correct parallel processing of serialized future states
  * Create virtual serial queue of changes for each deliverable
  * Assume each change will pass its tests
  * Test successive changes with previous changes applied to starting state

Nearest Non-Failing Change
==========================

(aka 'The Jim Blair Algorithm')

  * If a change fails, move it aside
  * Cancel all test jobs behind it in the queue
  * Reparent queue items on the nearest non-failing change
  * Restart tests with new state

Zuul Simulation
===============
.. transition:: pan
.. container:: handout

  * todo

.. ansi:: images/zsim-00.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-01.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-02.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-03.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-04.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-05.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-06.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-07.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-08.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-09.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-10.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-11.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-12.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-13.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-14.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-15.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-16.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-17.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-18.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-19.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-20.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-21.ans

Zuul Simulation
===============
.. transition:: cut
.. container:: handout

  * todo

.. ansi:: images/zsim-22.ans


Cross-Project Dependencies
==========================

Testing or gating dependencies manually specified by developers

.. container:: progressive

   * nodepool https://review.openstack.org/612168

       Make functional src jobs actually install from source
   * openstacksdk https://review.openstack.org/612186

       Don't start task managers passed in to Connection

       Depends-On: https://review.openstack.org/612168
   * openstacksdk https://review.openstack.org/604521

       Add support for per-service rate limits

       (git parent is 612186)
   * nodepool https://review.openstack.org/612169

       Consume rate limiting task manager from openstacksdk 

       Depends-On: https://review.openstack.org/604521

       (nodepool-functional-py35-src should pass, but
        nodepool-functional-py35 should not fail until openstacksdk release)

Live Configuration Changes
==========================

.. container:: handout

  Zuul is a distributed system, with a distributed configuration.

.. code:: yaml

   - tenant:
       name: openstack
       source:
         gerrit:
           config-repos:
             - openstack-infra/project-config
           project-repos:
             - openstack/nova
             - openstack/keystone
             - openstack-infra/devstack-gate

Zuul Startup
============

* Read config file

Zuul Startup
============

* Read config file
* Ask mergers for branches of each repo

.. ansi:: images/startup1.ans

Zuul Startup
============

* Read config file
* Ask mergers for branches of each repo
* Ask mergers for .zuul.yaml for each branch

    of each repo

.. ansi:: images/startup2.ans

When .zuul.yaml Changes
=======================

.. container:: progressive

  * Zuul looks for changes to .zuul.yaml
  * Asks mergers for updated content
  * Splices into configuration used for that change
  * Works with cross-repo dependencies

    ("This change depends on a change to the job definition")

Zuul Architecture
=================

.. ansi:: images/architecture.ans


Nodepool
========

  * A separate program that works very closely with *Zuul*
  * Creates and destroys zero or more node resources
  * Resources can include VMs, Containers, COE contexts or Bare Metals
  * Static driver for allocating pre-existing nodes to jobs
  * Optionally periodically builds images and uploads to clouds

Nodepool Launcher
=================

  Where build nodes should come from

  * OpenStack
  * Static

  In review:

  * Kubernetes
  * OpenShift
  * AWS

  In work / coming soon:

  * Azure
  * GCE
  * Mac Stadium

What about test/job content?
============================

  * Written in Ansible
  * Ansible is excellent at running one or more tasks in one or more places
  * The answer to "how do I" is almost always "Ansible"

What Zuul Does
==============

  * Listens for code events
  * Prepares appropriate job config and git repo states
  * Allocates nodes for test jobs
  * Pushes git repo states to nodes
  * Runs user-defined Ansible playbooks
  * Collects/reports results
  * Potentially merges change

OpenStack Infra - Largest Known Zuul
====================================

  * 2KJPH (2,000 jobs per hour)
  * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
  * Rackspace, Internap, OVH, Vexxhost, CityCloud
  * Linaro, Limestone, Packethost
  * 10,000 changes merged per month

Zuul is not New
===============

  * Has been in Production for OpenStack for Six Years
  * Zuul is now a top-level effort of OpenStack Foundation
  * Zuul is in production for OpenStack (Control Plane in OpenStack VMs)
  * Zuul v3 first release where not-OpenStack is first-class use case

Not just for OpenStack
======================

  * BMW (control plane in OpenShift)
  * GoDaddy (control plane in Kubernetes)
  * Easystack
  * OpenContrail
  * OpenLab
  * Red Hat
  * others ...

Code Review Systems
===================

  * Gerrit
  * GitHub (Public and Enterprise)

  In work / coming soon:

  * GitLab
  * Bitbucket

Support for non-git
===================

.. container:: progressive

  * Nope
  * helix4git *may* work for perforce, but is untested

Installation of Software
========================

Ways to Install Zuul
====================

* Windmill: http://git.openstack.org/cgit/openstack/windmill
* Software Factory: https://softwarefactory-project.io/
* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul
* Containers: https://hub.docker.com/_/zuul/

Zuul Containers
===============

* Published on every commit
* Application/Process containers
* Config / Data should be bind-mounted in

zuul/zuul-executor
==================

* In k8s, zuul-executor must be run privileged
* Uses bubblewrap for unprivileged sanboxing
* Restriction may be lifted in the future

Release Management
==================

* Zuul is a CI system
* C stands for "Continuous"
* It is run Continuously Delivered and Deployed upstream
* Releases are tagged from code run upstream
* There is no intent to have a 'stable' release
* 'stable' is a synonym for "old and buggy"

zuul/zuul-scheduler
===================

* SPOF
* We're working on it
* Recommend running scheduler from tags

Quick Start
===========

https://zuul-ci.org/docs/zuul/admin/quick-start.html

Important Links
===============

* https://zuul-ci.org/
* https://git.zuul-ci.org/cgit/zuul
* https://zuul-ci.org/docs/zuul
* https://zuul-ci.org/docs/zuul-jobs/
* https://docs.openstack.org/infra/openstack-zuul-jobs/
* freenode:#zuul

Questions
=========

.. ansi:: images/questions.ans

Quick Start Prereq
==================

* Install docker, docker-compose, git-review

Debian/Ubuntu:

::

  sudo apt-get install docker-compose git git-review

RHEL / CentOS / Fedora:

::

  sudo yum install docker docker-compose git git-review

OpenSuse:

::

  sudo zypper install docker docker-compose git git-review

RHEL / CentOS / Fedora / OpenSuse

::

  sudo systemctl enable docker.service
  sudo systemctl start docker.service

Actual Quick Start
==================

* git clone https://git.zuul-ci.org/zuul
* cd zuul
* cd doc/source/admin/examples
* docker-compose up

What's Running
==============

* Zookeeper
* Gerrit
* Nodepool Launcher
* Zuul Scheduler
* Zuul Web Server
* Zuul Executor
* Apache HTTPD
* A container to use as a 'static' build node

How they're connected
=====================

* End Users talk to Gerrit and Apache HTTPD
* Zuul Scheduler talks to Gerrit
* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
* Zuul Executor talks to Zuul Scheduler (using Gearman)

Initial provided config
=======================

* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf``
  and ``etc_zuul/main.yaml``
* Gerrit Connection named "gerrit"
* Zuul user for that connection
* Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library

Initial tenant
==============

* Zuul is (always) multi-tenant
* Example config contains a tenant called ``example-tenant``
* Three projects in the ``example-tenant`` tenant:
  ``zuul-config``, ``test1``, ``test2``
* Three projects are also in gerrit ready to use

zuul.conf
=========

::

  [gearman]
  server=scheduler

  [gearman_server]
  start=true

  [zookeeper]
  hosts=zk

  [scheduler]
  tenant_config=/etc/zuul/main.yaml

  [web]
  listen_address=0.0.0.0

  [executor]
  private_key_file=/var/ssh/nodepool
  default_username=root

zuul.conf part 2
================

::

  [connection "gerrit"]
  name=gerrit
  driver=gerrit
  server=gerrit
  sshkey=/var/ssh/zuul
  user=zuul
  password=secret
  baseurl=http://gerrit:8080
  auth_type=basic

  [connection "zuul-ci.org"]
  name=zuul-ci
  driver=git
  baseurl=https://git.zuul-ci.org/

main.yaml
=========

::

  - tenant:
      name: example-tenant
      source:
        gerrit:
          config-projects:
            - zuul-config
          untrusted-projects:
            - test1
            - test2
        zuul-ci.org:
          untrusted-projects:
            - zuul-jobs:
                include:
                  - job

Gerrit Account
==============

* Need a user account to interact with Gerrit
* Gerrit is configured in dev mode - no passwords required
* Visit http://localhost:8080
* Click "Become"
* Click "New Account"
* Click "Register"
* Enter Full Name
* Click "Save Changes"
* Enter username in Username field (match your local laptop user)
* Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field
* Click Continue

Config Repo
===========

* ``zuul-config`` is a trusted ``config-repo``
* Security and functionality of system depend on this repo
* Limit its contents to minimum required

Config Files vs. Directories
============================

* Zuul reads config from:
  ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d``
* For projects with substantial zuul config, like ``zuul-config``
  ``zuul.d`` directory is likely best.
* The directories are read run-parts style.
* Recommended practice is splitting by type of object

Setting up Gating
=================

* We want to have changes to ``zuul-config`` be gated
* We need to define pipelines: ``check`` and ``gate``
* Need to attach ``zuul-config`` to them
* Start with builtin ``noop`` job (always return success)
* Use regex to attach all projects to ``check`` and ``gate``

Pipeline Definitions
====================

* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate``

check pipeline
==============

::

  - pipeline:
      name: check
      description: |
        Newly uploaded patchsets enter this pipeline to receive an
        initial +/-1 Verified vote.
      manager: independent
      require:
        gerrit:
          open: True
          current-patchset: True
      trigger:
        gerrit:
          - event: patchset-created
          - event: change-restored
      success:
        gerrit:
          Verified: 1
      failure:
        gerrit:
          Verified: -1

gate pipeline
=============

:: 
    - pipeline:
        name: gate
        description: |
          Changes that have been approved are enqueued in order in this
          pipeline, and if they pass tests, will be merged.
        manager: dependent
        post-review: True
        require:
          gerrit:
            open: True
            current-patchset: True
            approval:
              - Workflow: 1
        trigger:
          gerrit:
            - event: comment-added
              approval:
                - Workflow: 1
        start:
          gerrit:
            Verified: 0
        success:
          gerrit:
            Verified: 2
            submit: true
        failure:
          gerrit:
            Verified: -2

Add the pipeline definitions
============================

.. code-block:: bash

  git clone http://localhost:8080/zuul-config
  cd zuul-config
  mkdir zuul.d
  cp ../examples/zuul-config/zuul.d/pipelines.yaml .

Shared Project Pipeline Definition
==================================

In ``examples/zuul-config/zuul.d/projects.yaml``

.. code-block:: yaml

  - project:
      name: ^.*$
      check:
        jobs: []
      gate:
        jobs: []

  - project:
      name: zuul-config
      check:
        jobs:
          - noop
      gate:
        jobs:
          - noop

Attach the projects to the pipelines
====================================

.. code-block:: bash

  cp ../examples/zuul-config/zuul.d/projects.yaml .

Commit the changes and push up for review
=========================================

.. code-block:: bash

   git add zuul.d
   git commit
   git review

Force merging bootstrap config
==============================

* Zuul is running with no config, so it won't do anything
* For this change (and this change only) we will bypass gating

Reviewing normally
==================

* visit http://localhost:8080/#/c/zuul-config/+/1001/
* click reply
* vote +2 Code Review +1 Approved

Verified +2 is Missing
======================

Verified +2 is what we have zuul configured to do.

::
        success:
          gerrit:
            Verified: 2
            submit: true


Bypassing Gating
================

* visit http://localhost:8080/
* click 'switch account'
* click 'admin'
* visit http://localhost:8080/#/c/zuul-config/+/1001/
* click reply
* vote +2 Verified (normal users do not see this)
* click submit (normal users do not see this)
* click 'switch account'
* click your username

Base Job
========

* Every Zuul installation must define a ``base`` job
* Push git repos to build node
* Publish logs/artifacts
* Any local specific setup
* Goes in config repo - because it impacts EVERY job

Add Base Job to zuul-config
===========================

::

  cp ../examples/zuul-config/zuul.d/jobs.yaml .
  git add jobs.yaml
  git commit
  git review

Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it

Zuul should merge the patch
===========================

zuul-config is configured to use the ``noop`` job

Zuul tests syntax automatically
===============================

* Edit jobs.yaml
* Change ``parent: null`` to ``parent: broken``
* git commit ; git review
* Check out the review in gerrit ... there should be errors!

Presentty
=========
.. hidetitle::
.. transition:: pan
.. figlet:: Presentty

* Console presentations written in reStructuredText
* Cross-fade, pan, tilt, cut transitions
* https://pypi.python.org/pypi/presentty