summaryrefslogtreecommitdiff
path: root/src/zuulv3
diff options
context:
space:
mode:
Diffstat (limited to 'src/zuulv3')
-rw-r--r--src/zuulv3/getting-started.rst989
1 files changed, 989 insertions, 0 deletions
diff --git a/src/zuulv3/getting-started.rst b/src/zuulv3/getting-started.rst
new file mode 100644
index 0000000..227d98e
--- /dev/null
+++ b/src/zuulv3/getting-started.rst
@@ -0,0 +1,989 @@
1. display in 68x24
2.. display in 88x24
3
4.. pygments yaml? (only file breaks (---) tinted)
5.. slide on high level v3 changes
6.. slide on nodepool
7
8.. transition:: dissolve
9 :duration: 0.4
10
11Test Slide
12==========
13.. hidetitle::
14
15.. ansi:: images/testslide.ans
16
17Preshow
18=======
19.. hidetitle::
20
21.. ansi:: images/cursor.ans images/cursor2.ans
22
23Zuul
24====
25.. hidetitle::
26.. ansi:: images/title.ans
27
28Red Hat
29=======
30.. hidetitle::
31.. container:: handout
32 i work for
33
34.. ansi:: images/redhat.ans
35
36
37Ansible
38=======
39.. hidetitle::
40.. ansi:: images/ansible.ans
41
42Zuul
43====
44.. hidetitle::
45.. ansi:: images/zuul.ans
46
47
48What Zuul Does
49==============
50
51 * gated changes
52 * one or more git repositories
53 * integrated deliverable
54 * testing like deployment
55
56Underlying Philosophy
57=====================
58
59 * All changes flow through code review
60 * Changes only land if they pass all tests
61 * Computers are cheaper than humans
62
63Ramifications of Philosophy
64===========================
65
66 * No direct push access for anyone
67 * Software should be installable from source
68 * Testing should be automated and repeatable
69 * Developers write tests with their patches
70 * Code always works
71
72Getting to Gating
73=================
74
75No Tests / Manual Tests
76=======================
77
78 * No test automation exists or ...
79 * Developer runs test suite before pushing code
80 * Prone to developer skipping tests for "trivial" changes
81 * Doesn't scale organizationally
82
83Periodic Testing
84================
85
86 * Developers push changes directly to shared branch
87 * CI system runs tests from time to time - report if things still work
88 * "Who broke the build?"
89 * Leads to hacks like NVIE model
90
91Post-Merge Testing
92==================
93
94 * Developers push changes directly to shared branch
95 * CI system is triggered by push - reports if push broke something
96 * Frequently batched / rolled up
97 * Easier to diagnose which change broke things
98 * Reactive - the bad changes are already in
99
100Pre-review Testing
101==================
102
103 * Changes are pushed to code review (Gerrit Change, GitHub PR, etc)
104 * CI system is triggered by code review change creation
105 * Test results inform review decisions
106 * Proactive - testing code before it lands
107 * Reviewers can get bored waiting for tests
108 * Only tests code as written, not potential result of merging code
109
110Gating
111======
112
113 * Changes are pushed to code review
114 * CI system is triggered by code review approval
115 * CI system merges code IFF tests pass
116 * Proactive - testing code before it lands
117 * Future state resulting from merge of code is tested
118 * Reviewers can fire-and-forget safely
119
120Mix and Match
121=============
122
123 * Zuul supports all of those modes
124 * Zuul users frequently combine them
125 * Run pre-review (check) and gating (gate) on each change
126 * Post-merge/post-tag for release/publication automation
127 * Periodic for catching bitrot
128
129Multi-repository integration
130============================
131
132 * Multiple source repositories are needed for deliverable
133 * Future state to be tested is the future state of all involved repos
134
135To test proposed future state
136=============================
137
138 * Get tip of each project. Merge appropriate change. Test.
139 * Changes must be serialized, otherwise state under test is invalid.
140 * Integrated deliverable repos share serialized queue
141
142Speculative Execution
143=====================
144
145 * Correct parallel processing of serialized future states
146 * Create virtual serial queue of changes for each deliverable
147 * Assume each change will pass its tests
148 * Test successive changes with previous changes applied to starting state
149
150Nearest Non-Failing Change
151==========================
152
153(aka 'The Jim Blair Algorithm')
154
155 * If a change fails, move it aside
156 * Cancel all test jobs behind it in the queue
157 * Reparent queue items on the nearest non-failing change
158 * Restart tests with new state
159
160Zuul Simulation
161===============
162.. transition:: pan
163.. container:: handout
164
165 * todo
166
167.. ansi:: images/zsim-00.ans
168
169Zuul Simulation
170===============
171.. transition:: cut
172.. container:: handout
173
174 * todo
175
176.. ansi:: images/zsim-01.ans
177
178Zuul Simulation
179===============
180.. transition:: cut
181.. container:: handout
182
183 * todo
184
185.. ansi:: images/zsim-02.ans
186
187Zuul Simulation
188===============
189.. transition:: cut
190.. container:: handout
191
192 * todo
193
194.. ansi:: images/zsim-03.ans
195
196Zuul Simulation
197===============
198.. transition:: cut
199.. container:: handout
200
201 * todo
202
203.. ansi:: images/zsim-04.ans
204
205Zuul Simulation
206===============
207.. transition:: cut
208.. container:: handout
209
210 * todo
211
212.. ansi:: images/zsim-05.ans
213
214Zuul Simulation
215===============
216.. transition:: cut
217.. container:: handout
218
219 * todo
220
221.. ansi:: images/zsim-06.ans
222
223Zuul Simulation
224===============
225.. transition:: cut
226.. container:: handout
227
228 * todo
229
230.. ansi:: images/zsim-07.ans
231
232Zuul Simulation
233===============
234.. transition:: cut
235.. container:: handout
236
237 * todo
238
239.. ansi:: images/zsim-08.ans
240
241Zuul Simulation
242===============
243.. transition:: cut
244.. container:: handout
245
246 * todo
247
248.. ansi:: images/zsim-09.ans
249
250Zuul Simulation
251===============
252.. transition:: cut
253.. container:: handout
254
255 * todo
256
257.. ansi:: images/zsim-10.ans
258
259Zuul Simulation
260===============
261.. transition:: cut
262.. container:: handout
263
264 * todo
265
266.. ansi:: images/zsim-11.ans
267
268Zuul Simulation
269===============
270.. transition:: cut
271.. container:: handout
272
273 * todo
274
275.. ansi:: images/zsim-12.ans
276
277Zuul Simulation
278===============
279.. transition:: cut
280.. container:: handout
281
282 * todo
283
284.. ansi:: images/zsim-13.ans
285
286Zuul Simulation
287===============
288.. transition:: cut
289.. container:: handout
290
291 * todo
292
293.. ansi:: images/zsim-14.ans
294
295Zuul Simulation
296===============
297.. transition:: cut
298.. container:: handout
299
300 * todo
301
302.. ansi:: images/zsim-15.ans
303
304Zuul Simulation
305===============
306.. transition:: cut
307.. container:: handout
308
309 * todo
310
311.. ansi:: images/zsim-16.ans
312
313Zuul Simulation
314===============
315.. transition:: cut
316.. container:: handout
317
318 * todo
319
320.. ansi:: images/zsim-17.ans
321
322Zuul Simulation
323===============
324.. transition:: cut
325.. container:: handout
326
327 * todo
328
329.. ansi:: images/zsim-18.ans
330
331Zuul Simulation
332===============
333.. transition:: cut
334.. container:: handout
335
336 * todo
337
338.. ansi:: images/zsim-19.ans
339
340Zuul Simulation
341===============
342.. transition:: cut
343.. container:: handout
344
345 * todo
346
347.. ansi:: images/zsim-20.ans
348
349Zuul Simulation
350===============
351.. transition:: cut
352.. container:: handout
353
354 * todo
355
356.. ansi:: images/zsim-21.ans
357
358Zuul Simulation
359===============
360.. transition:: cut
361.. container:: handout
362
363 * todo
364
365.. ansi:: images/zsim-22.ans
366
367
368Cross-Project Dependencies
369==========================
370
371Testing or gating dependencies manually specified by developers
372
373.. container:: progressive
374
375 * nodepool https://review.openstack.org/612168
376
377 Make functional src jobs actually install from source
378 * openstacksdk https://review.openstack.org/612186
379
380 Don't start task managers passed in to Connection
381
382 Depends-On: https://review.openstack.org/612168
383 * openstacksdk https://review.openstack.org/604521
384
385 Add support for per-service rate limits
386
387 (git parent is 612186)
388 * nodepool https://review.openstack.org/612169
389
390 Consume rate limiting task manager from openstacksdk
391
392 Depends-On: https://review.openstack.org/604521
393
394 (nodepool-functional-py35-src should pass, but
395 nodepool-functional-py35 should not fail until openstacksdk release)
396
397Live Configuration Changes
398==========================
399
400.. container:: handout
401
402 Zuul is a distributed system, with a distributed configuration.
403
404.. code:: yaml
405
406 - tenant:
407 name: openstack
408 source:
409 gerrit:
410 config-repos:
411 - openstack-infra/project-config
412 project-repos:
413 - openstack/nova
414 - openstack/keystone
415 - openstack-infra/devstack-gate
416
417Zuul Startup
418============
419
420* Read config file
421
422Zuul Startup
423============
424
425* Read config file
426* Ask mergers for branches of each repo
427
428.. ansi:: images/startup1.ans
429
430Zuul Startup
431============
432
433* Read config file
434* Ask mergers for branches of each repo
435* Ask mergers for .zuul.yaml for each branch
436
437 of each repo
438
439.. ansi:: images/startup2.ans
440
441When .zuul.yaml Changes
442=======================
443
444.. container:: progressive
445
446 * Zuul looks for changes to .zuul.yaml
447 * Asks mergers for updated content
448 * Splices into configuration used for that change
449 * Works with cross-repo dependencies
450
451 ("This change depends on a change to the job definition")
452
453Zuul Architecture
454=================
455
456.. ansi:: images/architecture.ans
457
458
459Nodepool
460========
461
462 * A separate program that works very closely with *Zuul*
463 * Creates and destroys zero or more node resources
464 * Resources can include VMs, Containers, COE contexts or Bare Metals
465 * Static driver for allocating pre-existing nodes to jobs
466 * Optionally periodically builds images and uploads to clouds
467
468Nodepool Launcher
469=================
470
471 Where build nodes should come from
472
473 * OpenStack
474 * Static
475
476 In review:
477
478 * Kubernetes
479 * OpenShift
480 * AWS
481
482 In work / coming soon:
483
484 * Azure
485 * GCE
486 * Mac Stadium
487
488What about test/job content?
489============================
490
491 * Written in Ansible
492 * Ansible is excellent at running one or more tasks in one or more places
493 * The answer to "how do I" is almost always "Ansible"
494
495What Zuul Does
496==============
497
498 * Listens for code events
499 * Prepares appropriate job config and git repo states
500 * Allocates nodes for test jobs
501 * Pushes git repo states to nodes
502 * Runs user-defined Ansible playbooks
503 * Collects/reports results
504 * Potentially merges change
505
506OpenStack Infra - Largest Known Zuul
507====================================
508
509 * 2KJPH (2,000 jobs per hour)
510 * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
511 * Rackspace, Internap, OVH, Vexxhost, CityCloud
512 * Linaro, Limestone, Packethost
513 * 10,000 changes merged per month
514
515Zuul is not New
516===============
517
518 * Has been in Production for OpenStack for Six Years
519 * Zuul is now a top-level effort of OpenStack Foundation
520 * Zuul is in production for OpenStack (Control Plane in OpenStack VMs)
521 * Zuul v3 first release where not-OpenStack is first-class use case
522
523Not just for OpenStack
524======================
525
526 * BMW (control plane in OpenShift)
527 * GoDaddy (control plane in Kubernetes)
528 * Easystack
529 * OpenContrail
530 * OpenLab
531 * Red Hat
532 * others ...
533
534Code Review Systems
535===================
536
537 * Gerrit
538 * GitHub (Public and Enterprise)
539
540 In work / coming soon:
541
542 * GitLab
543 * Bitbucket
544
545Support for non-git
546===================
547
548.. container:: progressive
549
550 * Nope
551 * helix4git *may* work for perforce, but is untested
552
553Installation of Software
554========================
555
556Ways to Install Zuul
557====================
558
559* Windmill: http://git.openstack.org/cgit/openstack/windmill
560* Software Factory: https://softwarefactory-project.io/
561* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul
562* Containers: https://hub.docker.com/_/zuul/
563
564Zuul Containers
565===============
566
567* Published on every commit
568* Application/Process containers
569* Config / Data should be bind-mounted in
570
571zuul/zuul-executor
572==================
573
574* In k8s, zuul-executor must be run privileged
575* Uses bubblewrap for unprivileged sanboxing
576* Restriction may be lifted in the future
577
578Release Management
579==================
580
581* Zuul is a CI system
582* C stands for "Continuous"
583* It is run Continuously Delivered and Deployed upstream
584* Releases are tagged from code run upstream
585* There is no intent to have a 'stable' release
586* 'stable' is a synonym for "old and buggy"
587
588zuul/zuul-scheduler
589===================
590
591* SPOF
592* We're working on it
593* Recommend running scheduler from tags
594
595Quick Start
596===========
597
598https://zuul-ci.org/docs/zuul/admin/quick-start.html
599
600Important Links
601===============
602
603* https://zuul-ci.org/
604* https://git.zuul-ci.org/cgit/zuul
605* https://zuul-ci.org/docs/zuul
606* https://zuul-ci.org/docs/zuul-jobs/
607* https://docs.openstack.org/infra/openstack-zuul-jobs/
608* freenode:#zuul
609
610Questions
611=========
612
613.. ansi:: images/questions.ans
614
615Quick Start Prereq
616==================
617
618* Install docker, docker-compose, git-review
619
620Debian/Ubuntu:
621
622::
623
624 sudo apt-get install docker-compose git git-review
625
626RHEL / CentOS / Fedora:
627
628::
629
630 sudo yum install docker docker-compose git git-review
631
632OpenSuse:
633
634::
635
636 sudo zypper install docker docker-compose git git-review
637
638RHEL / CentOS / Fedora / OpenSuse
639
640::
641
642 sudo systemctl enable docker.service
643 sudo systemctl start docker.service
644
645Actual Quick Start
646==================
647
648* git clone https://git.zuul-ci.org/zuul
649* cd zuul
650* cd doc/source/admin/examples
651* docker-compose up
652
653What's Running
654==============
655
656* Zookeeper
657* Gerrit
658* Nodepool Launcher
659* Zuul Scheduler
660* Zuul Web Server
661* Zuul Executor
662* Apache HTTPD
663* A container to use as a 'static' build node
664
665How they're connected
666=====================
667
668* End Users talk to Gerrit and Apache HTTPD
669* Zuul Scheduler talks to Gerrit
670* Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper
671* Zuul Executor talks to Zuul Scheduler (using Gearman)
672
673Initial provided config
674=======================
675
676* docker-compose has plumbed in basic config ``etc_zuul/zuul.conf``
677 and ``etc_zuul/main.yaml``
678* Gerrit Connection named "gerrit"
679* Zuul user for that connection
680* Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library
681
682Initial tenant
683==============
684
685* Zuul is (always) multi-tenant
686* Example config contains a tenant called ``example-tenant``
687* Three projects in the ``example-tenant`` tenant:
688 ``zuul-config``, ``test1``, ``test2``
689* Three projects are also in gerrit ready to use
690
691zuul.conf
692=========
693
694::
695
696 [gearman]
697 server=scheduler
698
699 [gearman_server]
700 start=true
701
702 [zookeeper]
703 hosts=zk
704
705 [scheduler]
706 tenant_config=/etc/zuul/main.yaml
707
708 [web]
709 listen_address=0.0.0.0
710
711 [executor]
712 private_key_file=/var/ssh/nodepool
713 default_username=root
714
715zuul.conf part 2
716================
717
718::
719
720 [connection "gerrit"]
721 name=gerrit
722 driver=gerrit
723 server=gerrit
724 sshkey=/var/ssh/zuul
725 user=zuul
726 password=secret
727 baseurl=http://gerrit:8080
728 auth_type=basic
729
730 [connection "zuul-ci.org"]
731 name=zuul-ci
732 driver=git
733 baseurl=https://git.zuul-ci.org/
734
735main.yaml
736=========
737
738::
739
740 - tenant:
741 name: example-tenant
742 source:
743 gerrit:
744 config-projects:
745 - zuul-config
746 untrusted-projects:
747 - test1
748 - test2
749 zuul-ci.org:
750 untrusted-projects:
751 - zuul-jobs:
752 include:
753 - job
754
755Gerrit Account
756==============
757
758* Need a user account to interact with Gerrit
759* Gerrit is configured in dev mode - no passwords required
760* Visit http://localhost:8080
761* Click "Become"
762* Click "New Account"
763* Click "Register"
764* Enter Full Name
765* Click "Save Changes"
766* Enter username in Username field (match your local laptop user)
767* Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field
768* Click Continue
769
770Config Repo
771===========
772
773* ``zuul-config`` is a trusted ``config-repo``
774* Security and functionality of system depend on this repo
775* Limit its contents to minimum required
776
777Config Files vs. Directories
778============================
779
780* Zuul reads config from:
781 ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d``
782* For projects with substantial zuul config, like ``zuul-config``
783 ``zuul.d`` directory is likely best.
784* The directories are read run-parts style.
785* Recommended practice is splitting by type of object
786
787Setting up Gating
788=================
789
790* We want to have changes to ``zuul-config`` be gated
791* We need to define pipelines: ``check`` and ``gate``
792* Need to attach ``zuul-config`` to them
793* Start with builtin ``noop`` job (always return success)
794* Use regex to attach all projects to ``check`` and ``gate``
795
796Pipeline Definitions
797====================
798
799* Zuul has no built-in workflow definitions, let's add ``check`` and ``gate``
800
801check pipeline
802==============
803
804::
805
806 - pipeline:
807 name: check
808 description: |
809 Newly uploaded patchsets enter this pipeline to receive an
810 initial +/-1 Verified vote.
811 manager: independent
812 require:
813 gerrit:
814 open: True
815 current-patchset: True
816 trigger:
817 gerrit:
818 - event: patchset-created
819 - event: change-restored
820 success:
821 gerrit:
822 Verified: 1
823 failure:
824 gerrit:
825 Verified: -1
826
827gate pipeline
828=============
829
830::
831 - pipeline:
832 name: gate
833 description: |
834 Changes that have been approved are enqueued in order in this
835 pipeline, and if they pass tests, will be merged.
836 manager: dependent
837 post-review: True
838 require:
839 gerrit:
840 open: True
841 current-patchset: True
842 approval:
843 - Workflow: 1
844 trigger:
845 gerrit:
846 - event: comment-added
847 approval:
848 - Workflow: 1
849 start:
850 gerrit:
851 Verified: 0
852 success:
853 gerrit:
854 Verified: 2
855 submit: true
856 failure:
857 gerrit:
858 Verified: -2
859
860Add the pipeline definitions
861============================
862
863.. code-block:: bash
864
865 git clone http://localhost:8080/zuul-config
866 cd zuul-config
867 mkdir zuul.d
868 cp ../examples/zuul-config/zuul.d/pipelines.yaml .
869
870Shared Project Pipeline Definition
871==================================
872
873In ``examples/zuul-config/zuul.d/projects.yaml``
874
875.. code-block:: yaml
876
877 - project:
878 name: ^.*$
879 check:
880 jobs: []
881 gate:
882 jobs: []
883
884 - project:
885 name: zuul-config
886 check:
887 jobs:
888 - noop
889 gate:
890 jobs:
891 - noop
892
893Attach the projects to the pipelines
894====================================
895
896.. code-block:: bash
897
898 cp ../examples/zuul-config/zuul.d/projects.yaml .
899
900Commit the changes and push up for review
901=========================================
902
903.. code-block:: bash
904
905 git add zuul.d
906 git commit
907 git review
908
909Force merging bootstrap config
910==============================
911
912* Zuul is running with no config, so it won't do anything
913* For this change (and this change only) we will bypass gating
914
915Reviewing normally
916==================
917
918* visit http://localhost:8080/#/c/zuul-config/+/1001/
919* click reply
920* vote +2 Code Review +1 Approved
921
922Verified +2 is Missing
923======================
924
925Verified +2 is what we have zuul configured to do.
926
927::
928 success:
929 gerrit:
930 Verified: 2
931 submit: true
932
933
934Bypassing Gating
935================
936
937* visit http://localhost:8080/
938* click 'switch account'
939* click 'admin'
940* visit http://localhost:8080/#/c/zuul-config/+/1001/
941* click reply
942* vote +2 Verified (normal users do not see this)
943* click submit (normal users do not see this)
944* click 'switch account'
945* click your username
946
947Base Job
948========
949
950* Every Zuul installation must define a ``base`` job
951* Push git repos to build node
952* Publish logs/artifacts
953* Any local specific setup
954* Goes in config repo - because it impacts EVERY job
955
956Add Base Job to zuul-config
957===========================
958
959::
960
961 cp ../examples/zuul-config/zuul.d/jobs.yaml .
962 git add jobs.yaml
963 git commit
964 git review
965
966Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it
967
968Zuul should merge the patch
969===========================
970
971zuul-config is configured to use the ``noop`` job
972
973Zuul tests syntax automatically
974===============================
975
976* Edit jobs.yaml
977* Change ``parent: null`` to ``parent: broken``
978* git commit ; git review
979* Check out the review in gerrit ... there should be errors!
980
981Presentty
982=========
983.. hidetitle::
984.. transition:: pan
985.. figlet:: Presentty
986
987* Console presentations written in reStructuredText
988* Cross-fade, pan, tilt, cut transitions
989* https://pypi.python.org/pypi/presentty
generated by cgit v1.2.3 (git 2.25.1) at 2024-12-25 06:30:44 +0000