From 6dddf880243376b9e3f3ef73be31ab4548b10cc0 Mon Sep 17 00:00:00 2001 From: Monty Taylor Date: Wed, 24 Oct 2018 06:10:20 +0100 Subject: Add getting-started talk --- src/zuulv3/getting-started.rst | 989 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 989 insertions(+) create mode 100644 src/zuulv3/getting-started.rst diff --git a/src/zuulv3/getting-started.rst b/src/zuulv3/getting-started.rst new file mode 100644 index 0000000..227d98e --- /dev/null +++ b/src/zuulv3/getting-started.rst @@ -0,0 +1,989 @@ +. display in 68x24 +.. display in 88x24 + +.. pygments yaml? (only file breaks (---) tinted) +.. slide on high level v3 changes +.. slide on nodepool + +.. transition:: dissolve + :duration: 0.4 + +Test Slide +========== +.. hidetitle:: + +.. ansi:: images/testslide.ans + +Preshow +======= +.. hidetitle:: + +.. ansi:: images/cursor.ans images/cursor2.ans + +Zuul +==== +.. hidetitle:: +.. ansi:: images/title.ans + +Red Hat +======= +.. hidetitle:: +.. container:: handout + i work for + +.. ansi:: images/redhat.ans + + +Ansible +======= +.. hidetitle:: +.. ansi:: images/ansible.ans + +Zuul +==== +.. hidetitle:: +.. ansi:: images/zuul.ans + + +What Zuul Does +============== + + * gated changes + * one or more git repositories + * integrated deliverable + * testing like deployment + +Underlying Philosophy +===================== + + * All changes flow through code review + * Changes only land if they pass all tests + * Computers are cheaper than humans + +Ramifications of Philosophy +=========================== + + * No direct push access for anyone + * Software should be installable from source + * Testing should be automated and repeatable + * Developers write tests with their patches + * Code always works + +Getting to Gating +================= + +No Tests / Manual Tests +======================= + + * No test automation exists or ... + * Developer runs test suite before pushing code + * Prone to developer skipping tests for "trivial" changes + * Doesn't scale organizationally + +Periodic Testing +================ + + * Developers push changes directly to shared branch + * CI system runs tests from time to time - report if things still work + * "Who broke the build?" + * Leads to hacks like NVIE model + +Post-Merge Testing +================== + + * Developers push changes directly to shared branch + * CI system is triggered by push - reports if push broke something + * Frequently batched / rolled up + * Easier to diagnose which change broke things + * Reactive - the bad changes are already in + +Pre-review Testing +================== + + * Changes are pushed to code review (Gerrit Change, GitHub PR, etc) + * CI system is triggered by code review change creation + * Test results inform review decisions + * Proactive - testing code before it lands + * Reviewers can get bored waiting for tests + * Only tests code as written, not potential result of merging code + +Gating +====== + + * Changes are pushed to code review + * CI system is triggered by code review approval + * CI system merges code IFF tests pass + * Proactive - testing code before it lands + * Future state resulting from merge of code is tested + * Reviewers can fire-and-forget safely + +Mix and Match +============= + + * Zuul supports all of those modes + * Zuul users frequently combine them + * Run pre-review (check) and gating (gate) on each change + * Post-merge/post-tag for release/publication automation + * Periodic for catching bitrot + +Multi-repository integration +============================ + + * Multiple source repositories are needed for deliverable + * Future state to be tested is the future state of all involved repos + +To test proposed future state +============================= + + * Get tip of each project. Merge appropriate change. Test. + * Changes must be serialized, otherwise state under test is invalid. + * Integrated deliverable repos share serialized queue + +Speculative Execution +===================== + + * Correct parallel processing of serialized future states + * Create virtual serial queue of changes for each deliverable + * Assume each change will pass its tests + * Test successive changes with previous changes applied to starting state + +Nearest Non-Failing Change +========================== + +(aka 'The Jim Blair Algorithm') + + * If a change fails, move it aside + * Cancel all test jobs behind it in the queue + * Reparent queue items on the nearest non-failing change + * Restart tests with new state + +Zuul Simulation +=============== +.. transition:: pan +.. container:: handout + + * todo + +.. ansi:: images/zsim-00.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-01.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-02.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-03.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-04.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-05.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-06.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-07.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-08.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-09.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-10.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-11.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-12.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-13.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-14.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-15.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-16.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-17.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-18.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-19.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-20.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-21.ans + +Zuul Simulation +=============== +.. transition:: cut +.. container:: handout + + * todo + +.. ansi:: images/zsim-22.ans + + +Cross-Project Dependencies +========================== + +Testing or gating dependencies manually specified by developers + +.. container:: progressive + + * nodepool https://review.openstack.org/612168 + + Make functional src jobs actually install from source + * openstacksdk https://review.openstack.org/612186 + + Don't start task managers passed in to Connection + + Depends-On: https://review.openstack.org/612168 + * openstacksdk https://review.openstack.org/604521 + + Add support for per-service rate limits + + (git parent is 612186) + * nodepool https://review.openstack.org/612169 + + Consume rate limiting task manager from openstacksdk + + Depends-On: https://review.openstack.org/604521 + + (nodepool-functional-py35-src should pass, but + nodepool-functional-py35 should not fail until openstacksdk release) + +Live Configuration Changes +========================== + +.. container:: handout + + Zuul is a distributed system, with a distributed configuration. + +.. code:: yaml + + - tenant: + name: openstack + source: + gerrit: + config-repos: + - openstack-infra/project-config + project-repos: + - openstack/nova + - openstack/keystone + - openstack-infra/devstack-gate + +Zuul Startup +============ + +* Read config file + +Zuul Startup +============ + +* Read config file +* Ask mergers for branches of each repo + +.. ansi:: images/startup1.ans + +Zuul Startup +============ + +* Read config file +* Ask mergers for branches of each repo +* Ask mergers for .zuul.yaml for each branch + + of each repo + +.. ansi:: images/startup2.ans + +When .zuul.yaml Changes +======================= + +.. container:: progressive + + * Zuul looks for changes to .zuul.yaml + * Asks mergers for updated content + * Splices into configuration used for that change + * Works with cross-repo dependencies + + ("This change depends on a change to the job definition") + +Zuul Architecture +================= + +.. ansi:: images/architecture.ans + + +Nodepool +======== + + * A separate program that works very closely with *Zuul* + * Creates and destroys zero or more node resources + * Resources can include VMs, Containers, COE contexts or Bare Metals + * Static driver for allocating pre-existing nodes to jobs + * Optionally periodically builds images and uploads to clouds + +Nodepool Launcher +================= + + Where build nodes should come from + + * OpenStack + * Static + + In review: + + * Kubernetes + * OpenShift + * AWS + + In work / coming soon: + + * Azure + * GCE + * Mac Stadium + +What about test/job content? +============================ + + * Written in Ansible + * Ansible is excellent at running one or more tasks in one or more places + * The answer to "how do I" is almost always "Ansible" + +What Zuul Does +============== + + * Listens for code events + * Prepares appropriate job config and git repo states + * Allocates nodes for test jobs + * Pushes git repo states to nodes + * Runs user-defined Ansible playbooks + * Collects/reports results + * Potentially merges change + +OpenStack Infra - Largest Known Zuul +==================================== + + * 2KJPH (2,000 jobs per hour) + * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds + * Rackspace, Internap, OVH, Vexxhost, CityCloud + * Linaro, Limestone, Packethost + * 10,000 changes merged per month + +Zuul is not New +=============== + + * Has been in Production for OpenStack for Six Years + * Zuul is now a top-level effort of OpenStack Foundation + * Zuul is in production for OpenStack (Control Plane in OpenStack VMs) + * Zuul v3 first release where not-OpenStack is first-class use case + +Not just for OpenStack +====================== + + * BMW (control plane in OpenShift) + * GoDaddy (control plane in Kubernetes) + * Easystack + * OpenContrail + * OpenLab + * Red Hat + * others ... + +Code Review Systems +=================== + + * Gerrit + * GitHub (Public and Enterprise) + + In work / coming soon: + + * GitLab + * Bitbucket + +Support for non-git +=================== + +.. container:: progressive + + * Nope + * helix4git *may* work for perforce, but is untested + +Installation of Software +======================== + +Ways to Install Zuul +==================== + +* Windmill: http://git.openstack.org/cgit/openstack/windmill +* Software Factory: https://softwarefactory-project.io/ +* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul +* Containers: https://hub.docker.com/_/zuul/ + +Zuul Containers +=============== + +* Published on every commit +* Application/Process containers +* Config / Data should be bind-mounted in + +zuul/zuul-executor +================== + +* In k8s, zuul-executor must be run privileged +* Uses bubblewrap for unprivileged sanboxing +* Restriction may be lifted in the future + +Release Management +================== + +* Zuul is a CI system +* C stands for "Continuous" +* It is run Continuously Delivered and Deployed upstream +* Releases are tagged from code run upstream +* There is no intent to have a 'stable' release +* 'stable' is a synonym for "old and buggy" + +zuul/zuul-scheduler +=================== + +* SPOF +* We're working on it +* Recommend running scheduler from tags + +Quick Start +=========== + +https://zuul-ci.org/docs/zuul/admin/quick-start.html + +Important Links +=============== + +* https://zuul-ci.org/ +* https://git.zuul-ci.org/cgit/zuul +* https://zuul-ci.org/docs/zuul +* https://zuul-ci.org/docs/zuul-jobs/ +* https://docs.openstack.org/infra/openstack-zuul-jobs/ +* freenode:#zuul + +Questions +========= + +.. ansi:: images/questions.ans + +Quick Start Prereq +================== + +* Install docker, docker-compose, git-review + +Debian/Ubuntu: + +:: + + sudo apt-get install docker-compose git git-review + +RHEL / CentOS / Fedora: + +:: + + sudo yum install docker docker-compose git git-review + +OpenSuse: + +:: + + sudo zypper install docker docker-compose git git-review + +RHEL / CentOS / Fedora / OpenSuse + +:: + + sudo systemctl enable docker.service + sudo systemctl start docker.service + +Actual Quick Start +================== + +* git clone https://git.zuul-ci.org/zuul +* cd zuul +* cd doc/source/admin/examples +* docker-compose up + +What's Running +============== + +* Zookeeper +* Gerrit +* Nodepool Launcher +* Zuul Scheduler +* Zuul Web Server +* Zuul Executor +* Apache HTTPD +* A container to use as a 'static' build node + +How they're connected +===================== + +* End Users talk to Gerrit and Apache HTTPD +* Zuul Scheduler talks to Gerrit +* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper +* Zuul Executor talks to Zuul Scheduler (using Gearman) + +Initial provided config +======================= + +* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf`` + and ``etc_zuul/main.yaml`` +* Gerrit Connection named "gerrit" +* Zuul user for that connection +* Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library + +Initial tenant +============== + +* Zuul is (always) multi-tenant +* Example config contains a tenant called ``example-tenant`` +* Three projects in the ``example-tenant`` tenant: + ``zuul-config``, ``test1``, ``test2`` +* Three projects are also in gerrit ready to use + +zuul.conf +========= + +:: + + [gearman] + server=scheduler + + [gearman_server] + start=true + + [zookeeper] + hosts=zk + + [scheduler] + tenant_config=/etc/zuul/main.yaml + + [web] + listen_address=0.0.0.0 + + [executor] + private_key_file=/var/ssh/nodepool + default_username=root + +zuul.conf part 2 +================ + +:: + + [connection "gerrit"] + name=gerrit + driver=gerrit + server=gerrit + sshkey=/var/ssh/zuul + user=zuul + password=secret + baseurl=http://gerrit:8080 + auth_type=basic + + [connection "zuul-ci.org"] + name=zuul-ci + driver=git + baseurl=https://git.zuul-ci.org/ + +main.yaml +========= + +:: + + - tenant: + name: example-tenant + source: + gerrit: + config-projects: + - zuul-config + untrusted-projects: + - test1 + - test2 + zuul-ci.org: + untrusted-projects: + - zuul-jobs: + include: + - job + +Gerrit Account +============== + +* Need a user account to interact with Gerrit +* Gerrit is configured in dev mode - no passwords required +* Visit http://localhost:8080 +* Click "Become" +* Click "New Account" +* Click "Register" +* Enter Full Name +* Click "Save Changes" +* Enter username in Username field (match your local laptop user) +* Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field +* Click Continue + +Config Repo +=========== + +* ``zuul-config`` is a trusted ``config-repo`` +* Security and functionality of system depend on this repo +* Limit its contents to minimum required + +Config Files vs. Directories +============================ + +* Zuul reads config from: + ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d`` +* For projects with substantial zuul config, like ``zuul-config`` + ``zuul.d`` directory is likely best. +* The directories are read run-parts style. +* Recommended practice is splitting by type of object + +Setting up Gating +================= + +* We want to have changes to ``zuul-config`` be gated +* We need to define pipelines: ``check`` and ``gate`` +* Need to attach ``zuul-config`` to them +* Start with builtin ``noop`` job (always return success) +* Use regex to attach all projects to ``check`` and ``gate`` + +Pipeline Definitions +==================== + +* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate`` + +check pipeline +============== + +:: + + - pipeline: + name: check + description: | + Newly uploaded patchsets enter this pipeline to receive an + initial +/-1 Verified vote. + manager: independent + require: + gerrit: + open: True + current-patchset: True + trigger: + gerrit: + - event: patchset-created + - event: change-restored + success: + gerrit: + Verified: 1 + failure: + gerrit: + Verified: -1 + +gate pipeline +============= + +:: + - pipeline: + name: gate + description: | + Changes that have been approved are enqueued in order in this + pipeline, and if they pass tests, will be merged. + manager: dependent + post-review: True + require: + gerrit: + open: True + current-patchset: True + approval: + - Workflow: 1 + trigger: + gerrit: + - event: comment-added + approval: + - Workflow: 1 + start: + gerrit: + Verified: 0 + success: + gerrit: + Verified: 2 + submit: true + failure: + gerrit: + Verified: -2 + +Add the pipeline definitions +============================ + +.. code-block:: bash + + git clone http://localhost:8080/zuul-config + cd zuul-config + mkdir zuul.d + cp ../examples/zuul-config/zuul.d/pipelines.yaml . + +Shared Project Pipeline Definition +================================== + +In ``examples/zuul-config/zuul.d/projects.yaml`` + +.. code-block:: yaml + + - project: + name: ^.*$ + check: + jobs: [] + gate: + jobs: [] + + - project: + name: zuul-config + check: + jobs: + - noop + gate: + jobs: + - noop + +Attach the projects to the pipelines +==================================== + +.. code-block:: bash + + cp ../examples/zuul-config/zuul.d/projects.yaml . + +Commit the changes and push up for review +========================================= + +.. code-block:: bash + + git add zuul.d + git commit + git review + +Force merging bootstrap config +============================== + +* Zuul is running with no config, so it won't do anything +* For this change (and this change only) we will bypass gating + +Reviewing normally +================== + +* visit http://localhost:8080/#/c/zuul-config/+/1001/ +* click reply +* vote +2 Code Review +1 Approved + +Verified +2 is Missing +====================== + +Verified +2 is what we have zuul configured to do. + +:: + success: + gerrit: + Verified: 2 + submit: true + + +Bypassing Gating +================ + +* visit http://localhost:8080/ +* click 'switch account' +* click 'admin' +* visit http://localhost:8080/#/c/zuul-config/+/1001/ +* click reply +* vote +2 Verified (normal users do not see this) +* click submit (normal users do not see this) +* click 'switch account' +* click your username + +Base Job +======== + +* Every Zuul installation must define a ``base`` job +* Push git repos to build node +* Publish logs/artifacts +* Any local specific setup +* Goes in config repo - because it impacts EVERY job + +Add Base Job to zuul-config +=========================== + +:: + + cp ../examples/zuul-config/zuul.d/jobs.yaml . + git add jobs.yaml + git commit + git review + +Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it + +Zuul should merge the patch +=========================== + +zuul-config is configured to use the ``noop`` job + +Zuul tests syntax automatically +=============================== + +* Edit jobs.yaml +* Change ``parent: null`` to ``parent: broken`` +* git commit ; git review +* Check out the review in gerrit ... there should be errors! + +Presentty +========= +.. hidetitle:: +.. transition:: pan +.. figlet:: Presentty + +* Console presentations written in reStructuredText +* Cross-fade, pan, tilt, cut transitions +* https://pypi.python.org/pypi/presentty -- cgit v1.2.3