Add getting-started talk

1 files changed, 989 insertions, 0 deletions

diff --git a/src/zuulv3/getting-started.rst b/src/zuulv3/getting-started.rst
new file mode 100644
index 0000000..227d98e
--- /dev/null
+++ b/src/zuulv3/getting-started.rst
@@ -0,0 +1,989 @@
+. display in 68x24
+.. display in 88x24
+
+.. pygments yaml? (only file breaks (---) tinted)
+.. slide on high level v3 changes
+.. slide on nodepool
+
+.. transition:: dissolve
+   :duration: 0.4
+
+Test Slide
+==========
+.. hidetitle::
+
+.. ansi:: images/testslide.ans
+
+Preshow
+=======
+.. hidetitle::
+
+.. ansi:: images/cursor.ans images/cursor2.ans
+
+Zuul
+====
+.. hidetitle::
+.. ansi:: images/title.ans
+
+Red Hat
+=======
+.. hidetitle::
+.. container:: handout
+   i work for
+
+.. ansi:: images/redhat.ans
+
+
+Ansible
+=======
+.. hidetitle::
+.. ansi:: images/ansible.ans
+
+Zuul
+====
+.. hidetitle::
+.. ansi:: images/zuul.ans
+
+
+What Zuul Does
+==============
+
+  * gated changes
+  * one or more git repositories
+  * integrated deliverable
+  * testing like deployment
+
+Underlying Philosophy
+=====================
+
+  * All changes flow through code review
+  * Changes only land if they pass all tests
+  * Computers are cheaper than humans
+
+Ramifications of Philosophy
+===========================
+
+  * No direct push access for anyone
+  * Software should be installable from source
+  * Testing should be automated and repeatable
+  * Developers write tests with their patches
+  * Code always works
+
+Getting to Gating
+=================
+
+No Tests / Manual Tests
+=======================
+
+  * No test automation exists or ...
+  * Developer runs test suite before pushing code
+  * Prone to developer skipping tests for "trivial" changes
+  * Doesn't scale organizationally
+
+Periodic Testing
+================
+
+  * Developers push changes directly to shared branch
+  * CI system runs tests from time to time - report if things still work
+  * "Who broke the build?"
+  * Leads to hacks like NVIE model
+
+Post-Merge Testing
+==================
+
+  * Developers push changes directly to shared branch
+  * CI system is triggered by push - reports if push broke something
+  * Frequently batched / rolled up
+  * Easier to diagnose which change broke things
+  * Reactive - the bad changes are already in
+
+Pre-review Testing
+==================
+
+  * Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
+  * CI system is triggered by code review change creation
+  * Test results inform review decisions
+  * Proactive - testing code before it lands
+  * Reviewers can get bored waiting for tests
+  * Only tests code as written, not potential result of merging code
+
+Gating
+======
+
+  * Changes are pushed to code review
+  * CI system is triggered by code review approval
+  * CI system merges code IFF tests pass
+  * Proactive - testing code before it lands
+  * Future state resulting from merge of code is tested
+  * Reviewers can fire-and-forget safely
+
+Mix and Match
+=============
+
+  * Zuul supports all of those modes
+  * Zuul users frequently combine them
+  * Run pre-review (check) and gating (gate) on each change
+  * Post-merge/post-tag for release/publication automation
+  * Periodic for catching bitrot
+
+Multi-repository integration
+============================
+
+  * Multiple source repositories are needed for deliverable
+  * Future state to be tested is the future state of all involved repos
+
+To test proposed future state
+=============================
+
+  * Get tip of each project. Merge appropriate change. Test.
+  * Changes must be serialized, otherwise state under test is invalid.
+  * Integrated deliverable repos share serialized queue
+
+Speculative Execution
+=====================
+
+  * Correct parallel processing of serialized future states
+  * Create virtual serial queue of changes for each deliverable
+  * Assume each change will pass its tests
+  * Test successive changes with previous changes applied to starting state
+
+Nearest Non-Failing Change
+==========================
+
+(aka 'The Jim Blair Algorithm')
+
+  * If a change fails, move it aside
+  * Cancel all test jobs behind it in the queue
+  * Reparent queue items on the nearest non-failing change
+  * Restart tests with new state
+
+Zuul Simulation
+===============
+.. transition:: pan
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-00.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-01.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-02.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-03.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-04.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-05.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-06.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-07.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-08.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-09.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-10.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-11.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-12.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-13.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-14.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-15.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-16.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-17.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-18.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-19.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-20.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-21.ans
+
+Zuul Simulation
+===============
+.. transition:: cut
+.. container:: handout
+
+  * todo
+
+.. ansi:: images/zsim-22.ans
+
+
+Cross-Project Dependencies
+==========================
+
+Testing or gating dependencies manually specified by developers
+
+.. container:: progressive
+
+   * nodepool https://review.openstack.org/612168
+
+       Make functional src jobs actually install from source
+   * openstacksdk https://review.openstack.org/612186
+
+       Don't start task managers passed in to Connection
+
+       Depends-On: https://review.openstack.org/612168
+   * openstacksdk https://review.openstack.org/604521
+
+       Add support for per-service rate limits
+
+       (git parent is 612186)
+   * nodepool https://review.openstack.org/612169
+
+       Consume rate limiting task manager from openstacksdk 
+
+       Depends-On: https://review.openstack.org/604521
+
+       (nodepool-functional-py35-src should pass, but
+        nodepool-functional-py35 should not fail until openstacksdk release)
+
+Live Configuration Changes
+==========================
+
+.. container:: handout
+
+  Zuul is a distributed system, with a distributed configuration.
+
+.. code:: yaml
+
+   - tenant:
+       name: openstack
+       source:
+         gerrit:
+           config-repos:
+             - openstack-infra/project-config
+           project-repos:
+             - openstack/nova
+             - openstack/keystone
+             - openstack-infra/devstack-gate
+
+Zuul Startup
+============
+
+* Read config file
+
+Zuul Startup
+============
+
+* Read config file
+* Ask mergers for branches of each repo
+
+.. ansi:: images/startup1.ans
+
+Zuul Startup
+============
+
+* Read config file
+* Ask mergers for branches of each repo
+* Ask mergers for .zuul.yaml for each branch
+
+    of each repo
+
+.. ansi:: images/startup2.ans
+
+When .zuul.yaml Changes
+=======================
+
+.. container:: progressive
+
+  * Zuul looks for changes to .zuul.yaml
+  * Asks mergers for updated content
+  * Splices into configuration used for that change
+  * Works with cross-repo dependencies
+
+    ("This change depends on a change to the job definition")
+
+Zuul Architecture
+=================
+
+.. ansi:: images/architecture.ans
+
+
+Nodepool
+========
+
+  * A separate program that works very closely with *Zuul*
+  * Creates and destroys zero or more node resources
+  * Resources can include VMs, Containers, COE contexts or Bare Metals
+  * Static driver for allocating pre-existing nodes to jobs
+  * Optionally periodically builds images and uploads to clouds
+
+Nodepool Launcher
+=================
+
+  Where build nodes should come from
+
+  * OpenStack
+  * Static
+
+  In review:
+
+  * Kubernetes
+  * OpenShift
+  * AWS
+
+  In work / coming soon:
+
+  * Azure
+  * GCE
+  * Mac Stadium
+
+What about test/job content?
+============================
+
+  * Written in Ansible
+  * Ansible is excellent at running one or more tasks in one or more places
+  * The answer to "how do I" is almost always "Ansible"
+
+What Zuul Does
+==============
+
+  * Listens for code events
+  * Prepares appropriate job config and git repo states
+  * Allocates nodes for test jobs
+  * Pushes git repo states to nodes
+  * Runs user-defined Ansible playbooks
+  * Collects/reports results
+  * Potentially merges change
+
+OpenStack Infra - Largest Known Zuul
+====================================
+
+  * 2KJPH (2,000 jobs per hour)
+  * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
+  * Rackspace, Internap, OVH, Vexxhost, CityCloud
+  * Linaro, Limestone, Packethost
+  * 10,000 changes merged per month
+
+Zuul is not New
+===============
+
+  * Has been in Production for OpenStack for Six Years
+  * Zuul is now a top-level effort of OpenStack Foundation
+  * Zuul is in production for OpenStack (Control Plane in OpenStack VMs)
+  * Zuul v3 first release where not-OpenStack is first-class use case
+
+Not just for OpenStack
+======================
+
+  * BMW (control plane in OpenShift)
+  * GoDaddy (control plane in Kubernetes)
+  * Easystack
+  * OpenContrail
+  * OpenLab
+  * Red Hat
+  * others ...
+
+Code Review Systems
+===================
+
+  * Gerrit
+  * GitHub (Public and Enterprise)
+
+  In work / coming soon:
+
+  * GitLab
+  * Bitbucket
+
+Support for non-git
+===================
+
+.. container:: progressive
+
+  * Nope
+  * helix4git *may* work for perforce, but is untested
+
+Installation of Software
+========================
+
+Ways to Install Zuul
+====================
+
+* Windmill: http://git.openstack.org/cgit/openstack/windmill
+* Software Factory: https://softwarefactory-project.io/
+* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul
+* Containers: https://hub.docker.com/_/zuul/
+
+Zuul Containers
+===============
+
+* Published on every commit
+* Application/Process containers
+* Config / Data should be bind-mounted in
+
+zuul/zuul-executor
+==================
+
+* In k8s, zuul-executor must be run privileged
+* Uses bubblewrap for unprivileged sanboxing
+* Restriction may be lifted in the future
+
+Release Management
+==================
+
+* Zuul is a CI system
+* C stands for "Continuous"
+* It is run Continuously Delivered and Deployed upstream
+* Releases are tagged from code run upstream
+* There is no intent to have a 'stable' release
+* 'stable' is a synonym for "old and buggy"
+
+zuul/zuul-scheduler
+===================
+
+* SPOF
+* We're working on it
+* Recommend running scheduler from tags
+
+Quick Start
+===========
+
+https://zuul-ci.org/docs/zuul/admin/quick-start.html
+
+Important Links
+===============
+
+* https://zuul-ci.org/
+* https://git.zuul-ci.org/cgit/zuul
+* https://zuul-ci.org/docs/zuul
+* https://zuul-ci.org/docs/zuul-jobs/
+* https://docs.openstack.org/infra/openstack-zuul-jobs/
+* freenode:#zuul
+
+Questions
+=========
+
+.. ansi:: images/questions.ans
+
+Quick Start Prereq
+==================
+
+* Install docker, docker-compose, git-review
+
+Debian/Ubuntu:
+
+::
+
+  sudo apt-get install docker-compose git git-review
+
+RHEL / CentOS / Fedora:
+
+::
+
+  sudo yum install docker docker-compose git git-review
+
+OpenSuse:
+
+::
+
+  sudo zypper install docker docker-compose git git-review
+
+RHEL / CentOS / Fedora / OpenSuse
+
+::
+
+  sudo systemctl enable docker.service
+  sudo systemctl start docker.service
+
+Actual Quick Start
+==================
+
+* git clone https://git.zuul-ci.org/zuul
+* cd zuul
+* cd doc/source/admin/examples
+* docker-compose up
+
+What's Running
+==============
+
+* Zookeeper
+* Gerrit
+* Nodepool Launcher
+* Zuul Scheduler
+* Zuul Web Server
+* Zuul Executor
+* Apache HTTPD
+* A container to use as a 'static' build node
+
+How they're connected
+=====================
+
+* End Users talk to Gerrit and Apache HTTPD
+* Zuul Scheduler talks to Gerrit
+* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
+* Zuul Executor talks to Zuul Scheduler (using Gearman)
+
+Initial provided config
+=======================
+
+* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf``
+  and ``etc_zuul/main.yaml``
+* Gerrit Connection named "gerrit"
+* Zuul user for that connection
+* Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library
+
+Initial tenant
+==============
+
+* Zuul is (always) multi-tenant
+* Example config contains a tenant called ``example-tenant``
+* Three projects in the ``example-tenant`` tenant:
+  ``zuul-config``, ``test1``, ``test2``
+* Three projects are also in gerrit ready to use
+
+zuul.conf
+=========
+
+::
+
+  [gearman]
+  server=scheduler
+
+  [gearman_server]
+  start=true
+
+  [zookeeper]
+  hosts=zk
+
+  [scheduler]
+  tenant_config=/etc/zuul/main.yaml
+
+  [web]
+  listen_address=0.0.0.0
+
+  [executor]
+  private_key_file=/var/ssh/nodepool
+  default_username=root
+
+zuul.conf part 2
+================
+
+::
+
+  [connection "gerrit"]
+  name=gerrit
+  driver=gerrit
+  server=gerrit
+  sshkey=/var/ssh/zuul
+  user=zuul
+  password=secret
+  baseurl=http://gerrit:8080
+  auth_type=basic
+
+  [connection "zuul-ci.org"]
+  name=zuul-ci
+  driver=git
+  baseurl=https://git.zuul-ci.org/
+
+main.yaml
+=========
+
+::
+
+  - tenant:
+      name: example-tenant
+      source:
+        gerrit:
+          config-projects:
+            - zuul-config
+          untrusted-projects:
+            - test1
+            - test2
+        zuul-ci.org:
+          untrusted-projects:
+            - zuul-jobs:
+                include:
+                  - job
+
+Gerrit Account
+==============
+
+* Need a user account to interact with Gerrit
+* Gerrit is configured in dev mode - no passwords required
+* Visit http://localhost:8080
+* Click "Become"
+* Click "New Account"
+* Click "Register"
+* Enter Full Name
+* Click "Save Changes"
+* Enter username in Username field (match your local laptop user)
+* Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field
+* Click Continue
+
+Config Repo
+===========
+
+* ``zuul-config`` is a trusted ``config-repo``
+* Security and functionality of system depend on this repo
+* Limit its contents to minimum required
+
+Config Files vs. Directories
+============================
+
+* Zuul reads config from:
+  ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d``
+* For projects with substantial zuul config, like ``zuul-config``
+  ``zuul.d`` directory is likely best.
+* The directories are read run-parts style.
+* Recommended practice is splitting by type of object
+
+Setting up Gating
+=================
+
+* We want to have changes to ``zuul-config`` be gated
+* We need to define pipelines: ``check`` and ``gate``
+* Need to attach ``zuul-config`` to them
+* Start with builtin ``noop`` job (always return success)
+* Use regex to attach all projects to ``check`` and ``gate``
+
+Pipeline Definitions
+====================
+
+* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate``
+
+check pipeline
+==============
+
+::
+
+  - pipeline:
+      name: check
+      description: |
+        Newly uploaded patchsets enter this pipeline to receive an
+        initial +/-1 Verified vote.
+      manager: independent
+      require:
+        gerrit:
+          open: True
+          current-patchset: True
+      trigger:
+        gerrit:
+          - event: patchset-created
+          - event: change-restored
+      success:
+        gerrit:
+          Verified: 1
+      failure:
+        gerrit:
+          Verified: -1
+
+gate pipeline
+=============
+
+:: 
+    - pipeline:
+        name: gate
+        description: |
+          Changes that have been approved are enqueued in order in this
+          pipeline, and if they pass tests, will be merged.
+        manager: dependent
+        post-review: True
+        require:
+          gerrit:
+            open: True
+            current-patchset: True
+            approval:
+              - Workflow: 1
+        trigger:
+          gerrit:
+            - event: comment-added
+              approval:
+                - Workflow: 1
+        start:
+          gerrit:
+            Verified: 0
+        success:
+          gerrit:
+            Verified: 2
+            submit: true
+        failure:
+          gerrit:
+            Verified: -2
+
+Add the pipeline definitions
+============================
+
+.. code-block:: bash
+
+  git clone http://localhost:8080/zuul-config
+  cd zuul-config
+  mkdir zuul.d
+  cp ../examples/zuul-config/zuul.d/pipelines.yaml .
+
+Shared Project Pipeline Definition
+==================================
+
+In ``examples/zuul-config/zuul.d/projects.yaml``
+
+.. code-block:: yaml
+
+  - project:
+      name: ^.*$
+      check:
+        jobs: []
+      gate:
+        jobs: []
+
+  - project:
+      name: zuul-config
+      check:
+        jobs:
+          - noop
+      gate:
+        jobs:
+          - noop
+
+Attach the projects to the pipelines
+====================================
+
+.. code-block:: bash
+
+  cp ../examples/zuul-config/zuul.d/projects.yaml .
+
+Commit the changes and push up for review
+=========================================
+
+.. code-block:: bash
+
+   git add zuul.d
+   git commit
+   git review
+
+Force merging bootstrap config
+==============================
+
+* Zuul is running with no config, so it won't do anything
+* For this change (and this change only) we will bypass gating
+
+Reviewing normally
+==================
+
+* visit http://localhost:8080/#/c/zuul-config/+/1001/
+* click reply
+* vote +2 Code Review +1 Approved
+
+Verified +2 is Missing
+======================
+
+Verified +2 is what we have zuul configured to do.
+
+::
+        success:
+          gerrit:
+            Verified: 2
+            submit: true
+
+
+Bypassing Gating
+================
+
+* visit http://localhost:8080/
+* click 'switch account'
+* click 'admin'
+* visit http://localhost:8080/#/c/zuul-config/+/1001/
+* click reply
+* vote +2 Verified (normal users do not see this)
+* click submit (normal users do not see this)
+* click 'switch account'
+* click your username
+
+Base Job
+========
+
+* Every Zuul installation must define a ``base`` job
+* Push git repos to build node
+* Publish logs/artifacts
+* Any local specific setup
+* Goes in config repo - because it impacts EVERY job
+
+Add Base Job to zuul-config
+===========================
+
+::
+
+  cp ../examples/zuul-config/zuul.d/jobs.yaml .
+  git add jobs.yaml
+  git commit
+  git review
+
+Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it
+
+Zuul should merge the patch
+===========================
+
+zuul-config is configured to use the ``noop`` job
+
+Zuul tests syntax automatically
+===============================
+
+* Edit jobs.yaml
+* Change ``parent: null`` to ``parent: broken``
+* git commit ; git review
+* Check out the review in gerrit ... there should be errors!
+
+Presentty
+=========
+.. hidetitle::
+.. transition:: pan
+.. figlet:: Presentty
+
+* Console presentations written in reStructuredText
+* Cross-fade, pan, tilt, cut transitions
+* https://pypi.python.org/pypi/presentty