diff options
Diffstat (limited to 'src')
-rw-r--r-- | src/zuulv3/overview.rst | 1258 |
1 files changed, 1258 insertions, 0 deletions
diff --git a/src/zuulv3/overview.rst b/src/zuulv3/overview.rst new file mode 100644 index 0000000..5e6b281 --- /dev/null +++ b/src/zuulv3/overview.rst | |||
@@ -0,0 +1,1258 @@ | |||
1 | . display in 68x24 | ||
2 | .. display in 88x24 | ||
3 | |||
4 | .. pygments yaml? (only file breaks (---) tinted) | ||
5 | .. slide on high level v3 changes | ||
6 | .. slide on nodepool | ||
7 | |||
8 | .. transition:: dissolve | ||
9 | :duration: 0.4 | ||
10 | |||
11 | Test Slide | ||
12 | ========== | ||
13 | .. hidetitle:: | ||
14 | |||
15 | .. ansi:: images/testslide.ans | ||
16 | |||
17 | Preshow | ||
18 | ======= | ||
19 | .. hidetitle:: | ||
20 | |||
21 | .. ansi:: images/cursor.ans images/cursor2.ans | ||
22 | |||
23 | Zuul | ||
24 | ==== | ||
25 | .. hidetitle:: | ||
26 | .. ansi:: images/title.ans | ||
27 | |||
28 | Monty Taylor | ||
29 | ============ | ||
30 | |||
31 | * @e_monty | ||
32 | * freenode:mordred | ||
33 | * mordred@inaugust.com | ||
34 | |||
35 | Red Hat | ||
36 | ======= | ||
37 | .. hidetitle:: | ||
38 | .. container:: handout | ||
39 | i work for | ||
40 | |||
41 | .. ansi:: images/redhat.ans | ||
42 | |||
43 | Ansible | ||
44 | ======= | ||
45 | .. hidetitle:: | ||
46 | .. ansi:: images/ansible.ans | ||
47 | |||
48 | OpenDev | ||
49 | ======= | ||
50 | |||
51 | :: | ||
52 | |||
53 | "most insane CI infrastructure I've ever been a part of" | ||
54 | |||
55 | -- Alex Gaynor | ||
56 | |||
57 | "OpenStack Infra are like the SpaceX of CI" | ||
58 | |||
59 | -- Emily Dunham | ||
60 | |||
61 | Zuul | ||
62 | ==== | ||
63 | .. hidetitle:: | ||
64 | .. ansi:: images/zuul.ans | ||
65 | |||
66 | |||
67 | What Zuul Does | ||
68 | ============== | ||
69 | |||
70 | * gated changes | ||
71 | * one or more git repositories | ||
72 | * integrated deliverable | ||
73 | * testing like deployment | ||
74 | |||
75 | Underlying Philosophy | ||
76 | ===================== | ||
77 | |||
78 | * All changes flow through code review | ||
79 | * Changes only land if they pass all tests | ||
80 | * Computers are cheaper than humans | ||
81 | |||
82 | Ramifications of Philosophy | ||
83 | =========================== | ||
84 | |||
85 | * No direct push access for anyone | ||
86 | * Software should be installable from source | ||
87 | * Testing should be automated and repeatable | ||
88 | * Developers write tests with their patches | ||
89 | * Code always works | ||
90 | |||
91 | Getting to Gating | ||
92 | ================= | ||
93 | |||
94 | No Tests / Manual Tests | ||
95 | ======================= | ||
96 | |||
97 | * No test automation exists or ... | ||
98 | * Developer runs test suite before pushing code | ||
99 | * Prone to developer skipping tests for "trivial" changes | ||
100 | * Doesn't scale organizationally | ||
101 | |||
102 | Periodic Testing | ||
103 | ================ | ||
104 | |||
105 | * Developers push changes directly to shared branch | ||
106 | * CI system runs tests from time to time - report if things still work | ||
107 | * "Who broke the build?" | ||
108 | * Leads to hacks like NVIE model | ||
109 | |||
110 | Post-Merge Testing | ||
111 | ================== | ||
112 | |||
113 | * Developers push changes directly to shared branch | ||
114 | * CI system is triggered by push - reports if push broke something | ||
115 | * Frequently batched / rolled up | ||
116 | * Easier to diagnose which change broke things | ||
117 | * Reactive - the bad changes are already in | ||
118 | |||
119 | Pre-Review Testing | ||
120 | ================== | ||
121 | |||
122 | * Changes are pushed to code review (Gerrit Change, GitHub PR, etc) | ||
123 | * CI system is triggered by code review change creation | ||
124 | * Test results inform review decisions | ||
125 | * Proactive - testing code before it lands | ||
126 | * Reviewers can get bored waiting for tests | ||
127 | * Only tests code as written, not potential result of merging code | ||
128 | |||
129 | Gating | ||
130 | ====== | ||
131 | |||
132 | * Changes are pushed to code review | ||
133 | * CI system is triggered by code review approval | ||
134 | * CI system merges code IFF tests pass | ||
135 | * Proactive - testing code before it lands | ||
136 | * Future state resulting from merge of code is tested | ||
137 | * Reviewers can fire-and-forget safely | ||
138 | |||
139 | Mix and Match | ||
140 | ============= | ||
141 | |||
142 | * Zuul supports all of those modes | ||
143 | * Zuul users frequently combine them | ||
144 | * Run pre-review (check) and gating (gate) on each change | ||
145 | * Post-merge/post-tag for release/publication automation | ||
146 | * Periodic for catching bitrot | ||
147 | |||
148 | Multi-repository integration | ||
149 | ============================ | ||
150 | |||
151 | * Multiple source repositories are needed for deliverable | ||
152 | * Future state to be tested is the future state of all involved repos | ||
153 | |||
154 | To test proposed future state | ||
155 | ============================= | ||
156 | |||
157 | * Get tip of each project. Merge appropriate change. Test. | ||
158 | * Changes must be serialized, otherwise state under test is invalid. | ||
159 | * Integrated deliverable repos share serialized queue | ||
160 | |||
161 | Speculative Execution | ||
162 | ===================== | ||
163 | |||
164 | * Correct parallel processing of serialized future states | ||
165 | * Create virtual serial queue of changes for each deliverable | ||
166 | * Assume each change will pass its tests | ||
167 | * Test successive changes with previous changes applied to starting state | ||
168 | |||
169 | Nearest Non-Failing Change | ||
170 | ========================== | ||
171 | |||
172 | (aka 'The Jim Blair Algorithm') | ||
173 | |||
174 | * If a change fails, move it aside | ||
175 | * Cancel all test jobs behind it in the queue | ||
176 | * Reparent queue items on the nearest non-failing change | ||
177 | * Restart tests with new state | ||
178 | |||
179 | Zuul Simulation | ||
180 | =============== | ||
181 | .. transition:: pan | ||
182 | .. container:: handout | ||
183 | |||
184 | * todo | ||
185 | |||
186 | .. ansi:: images/zsim-00.ans | ||
187 | |||
188 | Zuul Simulation | ||
189 | =============== | ||
190 | .. transition:: cut | ||
191 | .. container:: handout | ||
192 | |||
193 | * todo | ||
194 | |||
195 | .. ansi:: images/zsim-01.ans | ||
196 | |||
197 | Zuul Simulation | ||
198 | =============== | ||
199 | .. transition:: cut | ||
200 | .. container:: handout | ||
201 | |||
202 | * todo | ||
203 | |||
204 | .. ansi:: images/zsim-02.ans | ||
205 | |||
206 | Zuul Simulation | ||
207 | =============== | ||
208 | .. transition:: cut | ||
209 | .. container:: handout | ||
210 | |||
211 | * todo | ||
212 | |||
213 | .. ansi:: images/zsim-03.ans | ||
214 | |||
215 | Zuul Simulation | ||
216 | =============== | ||
217 | .. transition:: cut | ||
218 | .. container:: handout | ||
219 | |||
220 | * todo | ||
221 | |||
222 | .. ansi:: images/zsim-04.ans | ||
223 | |||
224 | Zuul Simulation | ||
225 | =============== | ||
226 | .. transition:: cut | ||
227 | .. container:: handout | ||
228 | |||
229 | * todo | ||
230 | |||
231 | .. ansi:: images/zsim-05.ans | ||
232 | |||
233 | Zuul Simulation | ||
234 | =============== | ||
235 | .. transition:: cut | ||
236 | .. container:: handout | ||
237 | |||
238 | * todo | ||
239 | |||
240 | .. ansi:: images/zsim-06.ans | ||
241 | |||
242 | Zuul Simulation | ||
243 | =============== | ||
244 | .. transition:: cut | ||
245 | .. container:: handout | ||
246 | |||
247 | * todo | ||
248 | |||
249 | .. ansi:: images/zsim-07.ans | ||
250 | |||
251 | Zuul Simulation | ||
252 | =============== | ||
253 | .. transition:: cut | ||
254 | .. container:: handout | ||
255 | |||
256 | * todo | ||
257 | |||
258 | .. ansi:: images/zsim-08.ans | ||
259 | |||
260 | Zuul Simulation | ||
261 | =============== | ||
262 | .. transition:: cut | ||
263 | .. container:: handout | ||
264 | |||
265 | * todo | ||
266 | |||
267 | .. ansi:: images/zsim-09.ans | ||
268 | |||
269 | Zuul Simulation | ||
270 | =============== | ||
271 | .. transition:: cut | ||
272 | .. container:: handout | ||
273 | |||
274 | * todo | ||
275 | |||
276 | .. ansi:: images/zsim-10.ans | ||
277 | |||
278 | Zuul Simulation | ||
279 | =============== | ||
280 | .. transition:: cut | ||
281 | .. container:: handout | ||
282 | |||
283 | * todo | ||
284 | |||
285 | .. ansi:: images/zsim-11.ans | ||
286 | |||
287 | Zuul Simulation | ||
288 | =============== | ||
289 | .. transition:: cut | ||
290 | .. container:: handout | ||
291 | |||
292 | * todo | ||
293 | |||
294 | .. ansi:: images/zsim-12.ans | ||
295 | |||
296 | Zuul Simulation | ||
297 | =============== | ||
298 | .. transition:: cut | ||
299 | .. container:: handout | ||
300 | |||
301 | * todo | ||
302 | |||
303 | .. ansi:: images/zsim-13.ans | ||
304 | |||
305 | Zuul Simulation | ||
306 | =============== | ||
307 | .. transition:: cut | ||
308 | .. container:: handout | ||
309 | |||
310 | * todo | ||
311 | |||
312 | .. ansi:: images/zsim-14.ans | ||
313 | |||
314 | Zuul Simulation | ||
315 | =============== | ||
316 | .. transition:: cut | ||
317 | .. container:: handout | ||
318 | |||
319 | * todo | ||
320 | |||
321 | .. ansi:: images/zsim-15.ans | ||
322 | |||
323 | Zuul Simulation | ||
324 | =============== | ||
325 | .. transition:: cut | ||
326 | .. container:: handout | ||
327 | |||
328 | * todo | ||
329 | |||
330 | .. ansi:: images/zsim-16.ans | ||
331 | |||
332 | Zuul Simulation | ||
333 | =============== | ||
334 | .. transition:: cut | ||
335 | .. container:: handout | ||
336 | |||
337 | * todo | ||
338 | |||
339 | .. ansi:: images/zsim-17.ans | ||
340 | |||
341 | Zuul Simulation | ||
342 | =============== | ||
343 | .. transition:: cut | ||
344 | .. container:: handout | ||
345 | |||
346 | * todo | ||
347 | |||
348 | .. ansi:: images/zsim-18.ans | ||
349 | |||
350 | Zuul Simulation | ||
351 | =============== | ||
352 | .. transition:: cut | ||
353 | .. container:: handout | ||
354 | |||
355 | * todo | ||
356 | |||
357 | .. ansi:: images/zsim-19.ans | ||
358 | |||
359 | Zuul Simulation | ||
360 | =============== | ||
361 | .. transition:: cut | ||
362 | .. container:: handout | ||
363 | |||
364 | * todo | ||
365 | |||
366 | .. ansi:: images/zsim-20.ans | ||
367 | |||
368 | Zuul Simulation | ||
369 | =============== | ||
370 | .. transition:: cut | ||
371 | .. container:: handout | ||
372 | |||
373 | * todo | ||
374 | |||
375 | .. ansi:: images/zsim-21.ans | ||
376 | |||
377 | Zuul Simulation | ||
378 | =============== | ||
379 | .. transition:: cut | ||
380 | .. container:: handout | ||
381 | |||
382 | * todo | ||
383 | |||
384 | .. ansi:: images/zsim-22.ans | ||
385 | |||
386 | |||
387 | Cross-Project Dependencies | ||
388 | ========================== | ||
389 | |||
390 | Testing or gating dependencies manually specified by developers | ||
391 | |||
392 | .. container:: progressive | ||
393 | |||
394 | * nodepool https://review.openstack.org/612168 | ||
395 | |||
396 | Make functional src jobs actually install from source | ||
397 | * openstacksdk https://review.openstack.org/612186 | ||
398 | |||
399 | Don't start task managers passed in to Connection | ||
400 | |||
401 | Depends-On: https://review.openstack.org/612168 | ||
402 | * openstacksdk https://review.openstack.org/604521 | ||
403 | |||
404 | Add support for per-service rate limits | ||
405 | |||
406 | (git parent is 612186) | ||
407 | * nodepool https://review.openstack.org/612169 | ||
408 | |||
409 | Consume rate limiting task manager from openstacksdk | ||
410 | |||
411 | Depends-On: https://review.openstack.org/604521 | ||
412 | |||
413 | (nodepool-functional-py35-src should pass, but | ||
414 | nodepool-functional-py35 should not fail until openstacksdk release) | ||
415 | |||
416 | Lock Step Changes | ||
417 | ================= | ||
418 | |||
419 | * Circular Dependencies are not supported on purpose | ||
420 | * Rolling upgrades across interdependent services | ||
421 | * HOWEVER - many valid use cases - support will be coming | ||
422 | |||
423 | Live Configuration Changes | ||
424 | ========================== | ||
425 | |||
426 | .. container:: handout | ||
427 | |||
428 | Zuul is a distributed system, with a distributed configuration. | ||
429 | |||
430 | .. code:: yaml | ||
431 | |||
432 | - tenant: | ||
433 | name: openstack | ||
434 | source: | ||
435 | gerrit: | ||
436 | config-repos: | ||
437 | - opendev/project-config | ||
438 | project-repos: | ||
439 | - opendev/zuul-jobs | ||
440 | - zuul/zuul | ||
441 | - zuul/nodepool | ||
442 | - ansible/ansible | ||
443 | - openstack/openstacksdk | ||
444 | |||
445 | Zuul Startup | ||
446 | ============ | ||
447 | |||
448 | * Read config file | ||
449 | |||
450 | Zuul Startup | ||
451 | ============ | ||
452 | |||
453 | * Read config file | ||
454 | * Ask mergers for branches of each repo | ||
455 | |||
456 | .. ansi:: images/startup1.ans | ||
457 | |||
458 | Zuul Startup | ||
459 | ============ | ||
460 | |||
461 | * Read config file | ||
462 | * Ask mergers for branches of each repo | ||
463 | * Ask mergers for .zuul.yaml for each branch | ||
464 | |||
465 | of each repo | ||
466 | |||
467 | .. ansi:: images/startup2.ans | ||
468 | |||
469 | When .zuul.yaml Changes | ||
470 | ======================= | ||
471 | |||
472 | .. container:: progressive | ||
473 | |||
474 | * Zuul looks for changes to .zuul.yaml | ||
475 | * Asks mergers for updated content | ||
476 | * Splices into configuration used for that change | ||
477 | * Works with cross-repo dependencies | ||
478 | |||
479 | ("This change depends on a change to the job definition") | ||
480 | |||
481 | Zuul Architecture | ||
482 | ================= | ||
483 | |||
484 | .. ansi:: images/architecture.ans | ||
485 | |||
486 | |||
487 | Nodepool | ||
488 | ======== | ||
489 | |||
490 | * A separate program that works very closely with *Zuul* | ||
491 | * Creates and destroys zero or more node resources | ||
492 | * Resources can include VMs, Containers, COE contexts or Bare Metals | ||
493 | * Static driver for allocating pre-existing nodes to jobs | ||
494 | * Optionally periodically builds images and uploads to clouds | ||
495 | |||
496 | Nodepool Launcher | ||
497 | ================= | ||
498 | |||
499 | Where build nodes should come from | ||
500 | |||
501 | * OpenStack | ||
502 | * Static | ||
503 | * Kubernetes | ||
504 | |||
505 | In review: | ||
506 | |||
507 | * OpenShift | ||
508 | * AWS | ||
509 | |||
510 | In work / coming soon: | ||
511 | |||
512 | * Azure | ||
513 | * GCE | ||
514 | * Mac Stadium | ||
515 | |||
516 | What about test/job content? | ||
517 | ============================ | ||
518 | |||
519 | * Written in Ansible | ||
520 | * Ansible is excellent at running one or more tasks in one or more places | ||
521 | * The answer to "how do I" is almost always "Ansible" | ||
522 | |||
523 | What Zuul Does | ||
524 | ============== | ||
525 | |||
526 | * Listens for code events | ||
527 | * Prepares appropriate job config and git repo states | ||
528 | * Allocates nodes for test jobs | ||
529 | * Pushes git repo states to nodes | ||
530 | * Runs user-defined Ansible playbooks | ||
531 | * Collects/reports results | ||
532 | * Potentially merges change | ||
533 | |||
534 | Jobs | ||
535 | ==== | ||
536 | |||
537 | * Jobs run on nodes from nodepool (static or dynamic) | ||
538 | * Metadata defined in Zuul's configuration | ||
539 | * Execution content in Ansible | ||
540 | * Jobs may be defined centrally or in the repo being tested | ||
541 | * Jobs have contextual variants that simplify configuration | ||
542 | |||
543 | Job | ||
544 | === | ||
545 | |||
546 | .. code:: yaml | ||
547 | |||
548 | - job: | ||
549 | name: base | ||
550 | parent: null | ||
551 | description: | | ||
552 | The base job for Zuul. | ||
553 | timeout: 1800 | ||
554 | nodeset: | ||
555 | nodes: | ||
556 | - name: primary | ||
557 | label: centos-7 | ||
558 | pre-run: playbooks/base/pre.yaml | ||
559 | post-run: | ||
560 | - playbooks/base/post-ssh.yaml | ||
561 | - playbooks/base/post-logs.yaml | ||
562 | secrets: | ||
563 | - site_logs | ||
564 | |||
565 | Simple Job | ||
566 | ========== | ||
567 | |||
568 | .. code:: yaml | ||
569 | |||
570 | - job: | ||
571 | name: tox | ||
572 | pre-run: playbooks/setup-tox.yaml | ||
573 | run: playbooks/tox.yaml | ||
574 | post-run: playbooks/fetch-tox-output.yaml | ||
575 | |||
576 | Simple Job Inheritance | ||
577 | ====================== | ||
578 | |||
579 | .. code:: yaml | ||
580 | |||
581 | - job: | ||
582 | name: tox-py36 | ||
583 | parent: tox | ||
584 | vars: | ||
585 | tox_envlist: py36 | ||
586 | |||
587 | Inheritance Works Like An Onion | ||
588 | =============================== | ||
589 | |||
590 | * pre-run playbooks run in order of inheritance | ||
591 | * run playbook of job runs | ||
592 | * post-run playbooks run in reverse order of inheritance | ||
593 | * If pre-run playbooks fail, job is re-tried | ||
594 | * All post-run playbooks run - as far as pre-run playbooks got | ||
595 | |||
596 | Inheritance Example | ||
597 | =================== | ||
598 | |||
599 | For tox-py36 job | ||
600 | |||
601 | * base pre-run playbooks/base/pre.yaml | ||
602 | * tox pre-run playbooks/setup-tox.yaml | ||
603 | * tox run playbooks/tox.yaml | ||
604 | * tox post-run playbooks/fetch-tox-output.yaml | ||
605 | * base post-run playbooks/base/post-ssh.yaml | ||
606 | * base post-run playbooks/base/post-logs.yaml | ||
607 | |||
608 | Simple Job Variant | ||
609 | ================== | ||
610 | |||
611 | .. code:: yaml | ||
612 | |||
613 | - job: | ||
614 | name: tox-py27 | ||
615 | branches: stable/mitaka | ||
616 | nodeset: | ||
617 | - name: ubuntu-trusty | ||
618 | label: ubuntu-trusty | ||
619 | |||
620 | Nodesets for Multi-node Jobs | ||
621 | ============================ | ||
622 | |||
623 | .. code:: yaml | ||
624 | |||
625 | - nodeset: | ||
626 | name: ceph-cluster | ||
627 | nodes: | ||
628 | - name: controller | ||
629 | label: centos-7 | ||
630 | - name: compute1 | ||
631 | label: fedora-28 | ||
632 | - name: compute2 | ||
633 | label: fedora-28 | ||
634 | groups: | ||
635 | - name: ceph-osd | ||
636 | nodes: | ||
637 | - controller | ||
638 | - name: ceph-monitor | ||
639 | nodes: | ||
640 | - controller | ||
641 | - compute1 | ||
642 | - compute2 | ||
643 | |||
644 | Multi-node Job | ||
645 | ============== | ||
646 | |||
647 | * nodesets are provided to Ansible for jobs in inventory | ||
648 | |||
649 | .. code:: yaml | ||
650 | |||
651 | - job: | ||
652 | name: ceph-multinode | ||
653 | nodeset: ceph-cluster | ||
654 | run: playbooks/install-ceph.yaml | ||
655 | |||
656 | |||
657 | Multi-node Ceph Job Content | ||
658 | =========================== | ||
659 | |||
660 | .. code:: yaml | ||
661 | |||
662 | - hosts: all | ||
663 | roles: | ||
664 | - install-ceph | ||
665 | |||
666 | - hosts: ceph-osd | ||
667 | roles: | ||
668 | - start-ceph-osd | ||
669 | |||
670 | - hosts: ceph-monitor | ||
671 | roles: | ||
672 | - start-ceph-monitor | ||
673 | |||
674 | - hosts: all | ||
675 | roles: | ||
676 | - do-something-interesting | ||
677 | |||
678 | Project With Central and Local Config | ||
679 | ===================================== | ||
680 | |||
681 | .. code:: yaml | ||
682 | |||
683 | # In git.openstack.org/openstack-infra/project-config: | ||
684 | - project: | ||
685 | name: openstack/nova | ||
686 | templates: | ||
687 | - openstack-tox-jobs | ||
688 | |||
689 | .. code:: yaml | ||
690 | |||
691 | # In git.openstack.org/openstack/nova/.zuul.yaml: | ||
692 | - project: | ||
693 | check: | ||
694 | - nova-placement-functional-devstack | ||
695 | |||
696 | Project with Job Dependencies | ||
697 | ============================= | ||
698 | |||
699 | .. code:: yaml | ||
700 | |||
701 | - project: | ||
702 | release: | ||
703 | jobs: | ||
704 | - build-artifacts | ||
705 | - upload-tarball: | ||
706 | dependencies: build-artifacts | ||
707 | - upload-pypi: | ||
708 | dependencies: build-artifacts | ||
709 | - notify-mirror: | ||
710 | dependencies: | ||
711 | - upload-tarball | ||
712 | - upload-pypi | ||
713 | |||
714 | Secrets | ||
715 | ======= | ||
716 | |||
717 | * Inspired by Kubernetes Secrets API | ||
718 | * Projects can add named encrypted secrets to their .zuul.yaml file | ||
719 | * Jobs can request to use secrets by name | ||
720 | * Jobs using secrets are not reconfigured speculatively | ||
721 | * Secrets can only be used by the same project they are defined in | ||
722 | * Public key per project: | ||
723 | ``{{ zuul_url }}/{{ tenant }}/{{ project }}.pub`` | ||
724 | |||
725 | :: | ||
726 | GET https://zuul.openstack.org/openstack-infra/shade.pub | ||
727 | |||
728 | Secret Example (note, no admins had to enable this) | ||
729 | =================================================== | ||
730 | |||
731 | .. code:: yaml | ||
732 | |||
733 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
734 | - secret: | ||
735 | name: loci_docker_login | ||
736 | data: | ||
737 | user: loci-username | ||
738 | password: !encrypted/pkcs1-oaep | ||
739 | - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw | ||
740 | hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE | ||
741 | 5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE | ||
742 | 5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo | ||
743 | 4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p | ||
744 | taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA | ||
745 | hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T | ||
746 | IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq | ||
747 | 7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj | ||
748 | /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM= | ||
749 | |||
750 | Secret Example | ||
751 | ============== | ||
752 | |||
753 | .. code:: yaml | ||
754 | |||
755 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
756 | - job: | ||
757 | name: publish-loci-cinder | ||
758 | parent: loci-cinder | ||
759 | post-run: playbooks/push | ||
760 | secrets: | ||
761 | - loci_docker_login | ||
762 | |||
763 | # In git.openstack.org/openstack/loci/playbooks/push.yaml: | ||
764 | - hosts: all | ||
765 | tasks: | ||
766 | - include_vars: vars.yaml | ||
767 | |||
768 | - name: Push project to DockerHub | ||
769 | block: | ||
770 | - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }} | ||
771 | no_log: True | ||
772 | - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} | ||
773 | with_items: "{{ distros }}" | ||
774 | |||
775 | OpenDev - Largest Known Zuul | ||
776 | ==================================== | ||
777 | |||
778 | * 2KJPH (2,000 jobs per hour) | ||
779 | * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds | ||
780 | * Rackspace, Internap, OVH, Vexxhost, CityCloud | ||
781 | * Linaro (ARM), Limestone, Packethost | ||
782 | * 10,000 changes merged per month | ||
783 | |||
784 | Zuul is not New | ||
785 | =============== | ||
786 | |||
787 | * Has been in Production for OpenStack for Six Years | ||
788 | * Zuul is now a top-level effort of OpenStack Foundation | ||
789 | * Zuul v3 first release where not-OpenStack is first-class use case | ||
790 | |||
791 | Not just for OpenStack | ||
792 | ====================== | ||
793 | |||
794 | * BMW (control plane in OpenShift) | ||
795 | * GoDaddy (control plane in Kubernetes) | ||
796 | * Le Bon Coin | ||
797 | * GoodMoney | ||
798 | * Easystack | ||
799 | * TungstenFabric | ||
800 | * OpenLab | ||
801 | * Red Hat | ||
802 | * others ... | ||
803 | |||
804 | Code Review Systems | ||
805 | =================== | ||
806 | |||
807 | * Gerrit | ||
808 | * GitHub (Public and Enterprise) | ||
809 | |||
810 | In work / coming soon: | ||
811 | |||
812 | * GitLab | ||
813 | * Bitbucket | ||
814 | |||
815 | Support for non-git | ||
816 | =================== | ||
817 | |||
818 | .. container:: progressive | ||
819 | |||
820 | * Nope | ||
821 | * helix4git may work for perforce, but is untested | ||
822 | |||
823 | Installation of Software | ||
824 | ======================== | ||
825 | |||
826 | Ways to Install Zuul | ||
827 | ==================== | ||
828 | |||
829 | * Containers: https://hub.docker.com/_/zuul/ | ||
830 | * Windmill: http://git.openstack.org/cgit/openstack/windmill | ||
831 | * Software Factory: https://softwarefactory-project.io/ | ||
832 | * Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul | ||
833 | |||
834 | Zuul Containers | ||
835 | =============== | ||
836 | |||
837 | * Published on every commit | ||
838 | * Application/Process containers | ||
839 | * Config / Data should be bind-mounted in | ||
840 | |||
841 | zuul/zuul-executor | ||
842 | ================== | ||
843 | |||
844 | * In k8s, zuul-executor must be run privileged | ||
845 | * Uses bubblewrap for unprivileged sanboxing | ||
846 | * Restriction may be lifted in the future | ||
847 | |||
848 | Release Management | ||
849 | ================== | ||
850 | |||
851 | * Zuul is a CI system | ||
852 | * C stands for "Continuous" | ||
853 | * It is run Continuously Delivered and Deployed upstream | ||
854 | * Releases are tagged from code run upstream | ||
855 | * There is no intent to have a 'stable' release | ||
856 | * 'stable' is a synonym for "old and buggy" | ||
857 | |||
858 | zuul/zuul-scheduler | ||
859 | =================== | ||
860 | |||
861 | * SPOF | ||
862 | * We're working on it | ||
863 | * Recommend running scheduler from tags | ||
864 | |||
865 | Quick Start | ||
866 | =========== | ||
867 | |||
868 | https://zuul-ci.org/docs/zuul/admin/quick-start.html | ||
869 | |||
870 | Important Links | ||
871 | =============== | ||
872 | |||
873 | * https://zuul-ci.org/ | ||
874 | * https://git.zuul-ci.org/cgit/zuul | ||
875 | * https://zuul-ci.org/docs/zuul | ||
876 | * https://zuul-ci.org/docs/zuul-jobs/ | ||
877 | * freenode:#zuul | ||
878 | |||
879 | Questions | ||
880 | ========= | ||
881 | |||
882 | .. ansi:: images/questions.ans | ||
883 | |||
884 | Quick Start Prereq | ||
885 | ================== | ||
886 | |||
887 | * Install docker, docker-compose, git-review | ||
888 | |||
889 | Debian/Ubuntu: | ||
890 | |||
891 | :: | ||
892 | |||
893 | sudo apt-get install docker-compose git git-review | ||
894 | |||
895 | RHEL / CentOS / Fedora: | ||
896 | |||
897 | :: | ||
898 | |||
899 | sudo yum install docker docker-compose git git-review | ||
900 | |||
901 | OpenSuse: | ||
902 | |||
903 | :: | ||
904 | |||
905 | sudo zypper install docker docker-compose git git-review | ||
906 | |||
907 | RHEL / CentOS / Fedora / OpenSuse | ||
908 | |||
909 | :: | ||
910 | |||
911 | sudo systemctl enable docker.service | ||
912 | sudo systemctl start docker.service | ||
913 | |||
914 | Actual Quick Start | ||
915 | ================== | ||
916 | |||
917 | * git clone https://git.zuul-ci.org/zuul | ||
918 | * cd zuul | ||
919 | * cd doc/source/admin/examples | ||
920 | * docker-compose up | ||
921 | |||
922 | What's Running | ||
923 | ============== | ||
924 | |||
925 | * Zookeeper | ||
926 | * Gerrit | ||
927 | * Nodepool Launcher | ||
928 | * Zuul Scheduler | ||
929 | * Zuul Web Server | ||
930 | * Zuul Executor | ||
931 | * Apache HTTPD | ||
932 | * A container to use as a 'static' build node | ||
933 | |||
934 | How they're connected | ||
935 | ===================== | ||
936 | |||
937 | * End Users talk to Gerrit and Apache HTTPD | ||
938 | * Zuul Scheduler talks to Gerrit | ||
939 | * Nodepool Launcher, Zuul Scheduler, Zuul Web talk to Zookeeper | ||
940 | * Zuul Executor talks to Zuul Scheduler (using Gearman) | ||
941 | |||
942 | Initial provided config | ||
943 | ======================= | ||
944 | |||
945 | * docker-compose has plumbed in basic config ``etc_zuul/zuul.conf`` | ||
946 | and ``etc_zuul/main.yaml`` | ||
947 | * Gerrit Connection named "gerrit" | ||
948 | * Zuul user for that connection | ||
949 | * Git connection named "zuul-ci.org" for ``zuul-jobs`` standard library | ||
950 | |||
951 | Initial tenant | ||
952 | ============== | ||
953 | |||
954 | * Zuul is (always) multi-tenant | ||
955 | * Example config contains a tenant called ``example-tenant`` | ||
956 | * Three projects in the ``example-tenant`` tenant: | ||
957 | ``zuul-config``, ``test1``, ``test2`` | ||
958 | * Three projects are also in gerrit ready to use | ||
959 | |||
960 | zuul.conf | ||
961 | ========= | ||
962 | |||
963 | :: | ||
964 | |||
965 | [gearman] | ||
966 | server=scheduler | ||
967 | |||
968 | [gearman_server] | ||
969 | start=true | ||
970 | |||
971 | [zookeeper] | ||
972 | hosts=zk | ||
973 | |||
974 | [scheduler] | ||
975 | tenant_config=/etc/zuul/main.yaml | ||
976 | |||
977 | [web] | ||
978 | listen_address=0.0.0.0 | ||
979 | |||
980 | [executor] | ||
981 | private_key_file=/var/ssh/nodepool | ||
982 | default_username=root | ||
983 | |||
984 | zuul.conf part 2 | ||
985 | ================ | ||
986 | |||
987 | :: | ||
988 | |||
989 | [connection "gerrit"] | ||
990 | name=gerrit | ||
991 | driver=gerrit | ||
992 | server=gerrit | ||
993 | sshkey=/var/ssh/zuul | ||
994 | user=zuul | ||
995 | password=secret | ||
996 | baseurl=http://gerrit:8080 | ||
997 | auth_type=basic | ||
998 | |||
999 | [connection "zuul-ci.org"] | ||
1000 | name=zuul-ci | ||
1001 | driver=git | ||
1002 | baseurl=https://git.zuul-ci.org/ | ||
1003 | |||
1004 | main.yaml | ||
1005 | ========= | ||
1006 | |||
1007 | :: | ||
1008 | |||
1009 | - tenant: | ||
1010 | name: example-tenant | ||
1011 | source: | ||
1012 | gerrit: | ||
1013 | config-projects: | ||
1014 | - zuul-config | ||
1015 | untrusted-projects: | ||
1016 | - test1 | ||
1017 | - test2 | ||
1018 | zuul-ci.org: | ||
1019 | untrusted-projects: | ||
1020 | - zuul-jobs: | ||
1021 | include: | ||
1022 | - job | ||
1023 | |||
1024 | Gerrit Account | ||
1025 | ============== | ||
1026 | |||
1027 | * Need a user account to interact with Gerrit | ||
1028 | * Gerrit is configured in dev mode - no passwords required | ||
1029 | * Visit http://localhost:8080 | ||
1030 | * Click "Become" | ||
1031 | * Click "New Account" | ||
1032 | * Click "Register" | ||
1033 | * Enter Full Name | ||
1034 | * Click "Save Changes" | ||
1035 | * Enter username in Username field (match your local laptop user) | ||
1036 | * Copy ``~/.ssh/id_rsa.pub`` contents into SSH Key field | ||
1037 | * Click Continue | ||
1038 | |||
1039 | Config Repo | ||
1040 | =========== | ||
1041 | |||
1042 | * ``zuul-config`` is a trusted ``config-repo`` | ||
1043 | * Security and functionality of system depend on this repo | ||
1044 | * Limit its contents to minimum required | ||
1045 | |||
1046 | Config Files vs. Directories | ||
1047 | ============================ | ||
1048 | |||
1049 | * Zuul reads config from: | ||
1050 | ``.zuul.yaml``, ``zuul.yaml``, ``zuul.d`` or ``.zuul.d`` | ||
1051 | * For projects with substantial zuul config, like ``zuul-config`` | ||
1052 | ``zuul.d`` directory is likely best. | ||
1053 | * The directories are read run-parts style. | ||
1054 | * Recommended practice is splitting by type of object | ||
1055 | |||
1056 | Setting up Gating | ||
1057 | ================= | ||
1058 | |||
1059 | * We want to have changes to ``zuul-config`` be gated | ||
1060 | * We need to define pipelines: ``check`` and ``gate`` | ||
1061 | * Need to attach ``zuul-config`` to them | ||
1062 | * Start with builtin ``noop`` job (always return success) | ||
1063 | * Use regex to attach all projects to ``check`` and ``gate`` | ||
1064 | |||
1065 | Pipeline Definitions | ||
1066 | ==================== | ||
1067 | |||
1068 | * Zuul has no built-in workflow definitions, let's add ``check`` and ``gate`` | ||
1069 | |||
1070 | check pipeline | ||
1071 | ============== | ||
1072 | |||
1073 | :: | ||
1074 | |||
1075 | - pipeline: | ||
1076 | name: check | ||
1077 | description: | | ||
1078 | Newly uploaded patchsets enter this pipeline to receive an | ||
1079 | initial +/-1 Verified vote. | ||
1080 | manager: independent | ||
1081 | require: | ||
1082 | gerrit: | ||
1083 | open: True | ||
1084 | current-patchset: True | ||
1085 | trigger: | ||
1086 | gerrit: | ||
1087 | - event: patchset-created | ||
1088 | - event: change-restored | ||
1089 | success: | ||
1090 | gerrit: | ||
1091 | Verified: 1 | ||
1092 | failure: | ||
1093 | gerrit: | ||
1094 | Verified: -1 | ||
1095 | |||
1096 | gate pipeline | ||
1097 | ============= | ||
1098 | |||
1099 | :: | ||
1100 | - pipeline: | ||
1101 | name: gate | ||
1102 | description: | | ||
1103 | Changes that have been approved are enqueued in order in this | ||
1104 | pipeline, and if they pass tests, will be merged. | ||
1105 | manager: dependent | ||
1106 | post-review: True | ||
1107 | require: | ||
1108 | gerrit: | ||
1109 | open: True | ||
1110 | current-patchset: True | ||
1111 | approval: | ||
1112 | - Workflow: 1 | ||
1113 | trigger: | ||
1114 | gerrit: | ||
1115 | - event: comment-added | ||
1116 | approval: | ||
1117 | - Workflow: 1 | ||
1118 | start: | ||
1119 | gerrit: | ||
1120 | Verified: 0 | ||
1121 | success: | ||
1122 | gerrit: | ||
1123 | Verified: 2 | ||
1124 | submit: true | ||
1125 | failure: | ||
1126 | gerrit: | ||
1127 | Verified: -2 | ||
1128 | |||
1129 | Add the pipeline definitions | ||
1130 | ============================ | ||
1131 | |||
1132 | .. code-block:: bash | ||
1133 | |||
1134 | git clone http://localhost:8080/zuul-config | ||
1135 | cd zuul-config | ||
1136 | mkdir zuul.d | ||
1137 | cp ../examples/zuul-config/zuul.d/pipelines.yaml . | ||
1138 | |||
1139 | Shared Project Pipeline Definition | ||
1140 | ================================== | ||
1141 | |||
1142 | In ``examples/zuul-config/zuul.d/projects.yaml`` | ||
1143 | |||
1144 | .. code-block:: yaml | ||
1145 | |||
1146 | - project: | ||
1147 | name: ^.*$ | ||
1148 | check: | ||
1149 | jobs: [] | ||
1150 | gate: | ||
1151 | jobs: [] | ||
1152 | |||
1153 | - project: | ||
1154 | name: zuul-config | ||
1155 | check: | ||
1156 | jobs: | ||
1157 | - noop | ||
1158 | gate: | ||
1159 | jobs: | ||
1160 | - noop | ||
1161 | |||
1162 | Attach the projects to the pipelines | ||
1163 | ==================================== | ||
1164 | |||
1165 | .. code-block:: bash | ||
1166 | |||
1167 | cp ../examples/zuul-config/zuul.d/projects.yaml . | ||
1168 | |||
1169 | Commit the changes and push up for review | ||
1170 | ========================================= | ||
1171 | |||
1172 | .. code-block:: bash | ||
1173 | |||
1174 | git add zuul.d | ||
1175 | git commit | ||
1176 | git review | ||
1177 | |||
1178 | Force merging bootstrap config | ||
1179 | ============================== | ||
1180 | |||
1181 | * Zuul is running with no config, so it won't do anything | ||
1182 | * For this change (and this change only) we will bypass gating | ||
1183 | |||
1184 | Reviewing normally | ||
1185 | ================== | ||
1186 | |||
1187 | * visit http://localhost:8080/#/c/zuul-config/+/1001/ | ||
1188 | * click reply | ||
1189 | * vote +2 Code Review +1 Approved | ||
1190 | |||
1191 | Verified +2 is Missing | ||
1192 | ====================== | ||
1193 | |||
1194 | Verified +2 is what we have zuul configured to do. | ||
1195 | |||
1196 | :: | ||
1197 | success: | ||
1198 | gerrit: | ||
1199 | Verified: 2 | ||
1200 | submit: true | ||
1201 | |||
1202 | |||
1203 | Bypassing Gating | ||
1204 | ================ | ||
1205 | |||
1206 | * visit http://localhost:8080/ | ||
1207 | * click 'switch account' | ||
1208 | * click 'admin' | ||
1209 | * visit http://localhost:8080/#/c/zuul-config/+/1001/ | ||
1210 | * click reply | ||
1211 | * vote +2 Verified (normal users do not see this) | ||
1212 | * click submit (normal users do not see this) | ||
1213 | * click 'switch account' | ||
1214 | * click your username | ||
1215 | |||
1216 | Base Job | ||
1217 | ======== | ||
1218 | |||
1219 | * Every Zuul installation must define a ``base`` job | ||
1220 | * Push git repos to build node | ||
1221 | * Publish logs/artifacts | ||
1222 | * Any local specific setup | ||
1223 | * Goes in config repo - because it impacts EVERY job | ||
1224 | |||
1225 | Add Base Job to zuul-config | ||
1226 | =========================== | ||
1227 | |||
1228 | :: | ||
1229 | |||
1230 | cp ../examples/zuul-config/zuul.d/jobs.yaml . | ||
1231 | git add jobs.yaml | ||
1232 | git commit | ||
1233 | git review | ||
1234 | |||
1235 | Then go to http://localhost:8080/#/c/zuul-config/+/1002/ and approve it | ||
1236 | |||
1237 | Zuul should merge the patch | ||
1238 | =========================== | ||
1239 | |||
1240 | zuul-config is configured to use the ``noop`` job | ||
1241 | |||
1242 | Zuul tests syntax automatically | ||
1243 | =============================== | ||
1244 | |||
1245 | * Edit jobs.yaml | ||
1246 | * Change ``parent: null`` to ``parent: broken`` | ||
1247 | * git commit ; git review | ||
1248 | * Check out the review in gerrit ... there should be errors! | ||
1249 | |||
1250 | Presentty | ||
1251 | ========= | ||
1252 | .. hidetitle:: | ||
1253 | .. transition:: pan | ||
1254 | .. figlet:: Presentty | ||
1255 | |||
1256 | * Console presentations written in reStructuredText | ||
1257 | * Cross-fade, pan, tilt, cut transitions | ||
1258 | * https://pypi.python.org/pypi/presentty | ||