diff options
Diffstat (limited to 'src/gating/gating.rst')
-rw-r--r-- | src/gating/gating.rst | 1044 |
1 files changed, 1044 insertions, 0 deletions
diff --git a/src/gating/gating.rst b/src/gating/gating.rst new file mode 100644 index 0000000..a636e47 --- /dev/null +++ b/src/gating/gating.rst | |||
@@ -0,0 +1,1044 @@ | |||
1 | . display in 68x24 | ||
2 | .. display in 88x24 | ||
3 | |||
4 | .. pygments yaml? (only file breaks (---) tinted) | ||
5 | .. slide on high level v3 changes | ||
6 | .. slide on nodepool | ||
7 | |||
8 | .. transition:: dissolve | ||
9 | :duration: 0.4 | ||
10 | |||
11 | Test Slide | ||
12 | ========== | ||
13 | .. hidetitle:: | ||
14 | |||
15 | .. ansi:: images/testslide.ans | ||
16 | |||
17 | Zuul v3 for Gating | ||
18 | ================== | ||
19 | |||
20 | Who the heck am I? | ||
21 | ================== | ||
22 | |||
23 | * mordred on freenode | ||
24 | * @e_monty on twitter | ||
25 | * I like scuba diving, smoking meat and long walks on the beach ... | ||
26 | |||
27 | Red Hat | ||
28 | ======= | ||
29 | |||
30 | .. hidetitle:: | ||
31 | .. container:: handout | ||
32 | |||
33 | * I work for Red Hat in the CTO Office as the Chief Architect | ||
34 | for CI/CD | ||
35 | |||
36 | .. ansi:: images/redhat.ans | ||
37 | |||
38 | OpenStack | ||
39 | ========= | ||
40 | |||
41 | .. container:: handout | ||
42 | |||
43 | * I work on OpenStack. | ||
44 | * I sit on the Board of Directors. I was on the Technical Committee | ||
45 | |||
46 | .. hidetitle:: | ||
47 | .. ansi:: images/openstack.ans | ||
48 | |||
49 | OpenStack Infra | ||
50 | =============== | ||
51 | |||
52 | .. container:: handout | ||
53 | |||
54 | * My primary technical role with OpenStack is working on the OpenStack CI | ||
55 | system. | ||
56 | |||
57 | :: | ||
58 | |||
59 | "most insane CI infrastructure I've ever been a part of" | ||
60 | |||
61 | -- Alex Gaynor | ||
62 | |||
63 | "OpenStack Infra are like the SpaceX of CI" | ||
64 | |||
65 | -- Emily Dunham | ||
66 | |||
67 | tl;dr | ||
68 | ===== | ||
69 | |||
70 | * multi repo | ||
71 | * integrated deliverable | ||
72 | * gated commits | ||
73 | * open tooling | ||
74 | * nobody is special | ||
75 | * there is no Dana, only Zuul | ||
76 | |||
77 | OpenStack Is | ||
78 | ============ | ||
79 | |||
80 | * Federated | ||
81 | * Distributed | ||
82 | * Large | ||
83 | * Open | ||
84 | * Not Alone | ||
85 | |||
86 | Federated | ||
87 | ========= | ||
88 | |||
89 | * Hundreds of involved companies | ||
90 | * No 'main' company | ||
91 | * "Decisions are made by those who show up" | ||
92 | * Union of priorities/use cases | ||
93 | |||
94 | Impact of being Federated | ||
95 | ========================= | ||
96 | |||
97 | * No company can appoint humans to project positions | ||
98 | * The project cannot fire anyone | ||
99 | * Variable background of contributors | ||
100 | * Heavy reliance on consensus-oriented democracy | ||
101 | |||
102 | Distributed | ||
103 | =========== | ||
104 | |||
105 | * There is no office | ||
106 | * Contributor base is global | ||
107 | * Multitude of contributor backgrounds | ||
108 | |||
109 | Impact of being Distributed | ||
110 | =========================== | ||
111 | |||
112 | * Constantly at odds with American Exceptionalism | ||
113 | * Tooling must empower all contributors, regardless of background, | ||
114 | skill level or cultural context | ||
115 | * Heavy preference for text-based communication | ||
116 | * Cannot assume US-centric needs or solutions | ||
117 | |||
118 | Large numbers of | ||
119 | ================ | ||
120 | |||
121 | * Contributors (\~2k in any given 6 month period) | ||
122 | * Changes | ||
123 | * Code Repositories (1904 as of this morning) | ||
124 | |||
125 | OpenStack Scale Comparison | ||
126 | ========================== | ||
127 | |||
128 | * 2KJPH (2,000 jobs per hour) | ||
129 | * Build Nodes from 13 Regions of 5 Public and 2 Private OpenStack Clouds | ||
130 | * Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone | ||
131 | * 10,000 changes merged per month | ||
132 | |||
133 | OpenStack Scale Comparison | ||
134 | ========================== | ||
135 | |||
136 | * 2KJPH (2,000 jobs per hour) | ||
137 | * Nodes from 12 Regions of 5 Public and 1 Private OpenStack Clouds | ||
138 | * Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone | ||
139 | * 10,000 changes merged per month | ||
140 | |||
141 | By comparison, our friends at the amazing project Ansible received | ||
142 | 13,000 changes and had merged 8,000 of them in its first 4 years. | ||
143 | |||
144 | Four Opens | ||
145 | ========== | ||
146 | |||
147 | * Open Source | ||
148 | (we don't hold back Enterprise features, we don't cripple things) | ||
149 | * Open Design | ||
150 | (design process open to all, decisions are not made inside company doors) | ||
151 | * Open Development | ||
152 | (public source code, public code review, all code is reviewed and gated) | ||
153 | * Open Community | ||
154 | (lazy consensus, democratic leadership from participants, | ||
155 | public logged meetings in IRC, public archived mailing lists) | ||
156 | |||
157 | Nobody is Special | ||
158 | ================= | ||
159 | |||
160 | * No dictators | ||
161 | * Aggressively egalitarian | ||
162 | * No "pay for play" | ||
163 | |||
164 | Free Software Needs Free Tools | ||
165 | ============================== | ||
166 | |||
167 | .. hidetitle:: | ||
168 | |||
169 | :: | ||
170 | |||
171 | Free Software needs Free Tools | ||
172 | - Benjamin Mako Hill | ||
173 | |||
174 | Fifth Open - Four Opens Applied to our Infrastructure | ||
175 | ===================================================== | ||
176 | |||
177 | * All tools must be Open Source | ||
178 | * Any external services must by Open Source | ||
179 | * Strongly avoid single-vendor | ||
180 | |||
181 | All Tools are Open Source | ||
182 | ========================= | ||
183 | |||
184 | .. hidetitle:: | ||
185 | |||
186 | No Developer is ever required to use a proprietary tool | ||
187 | to work on OpenStack. | ||
188 | |||
189 | Sixth Open - Four Opens Applied to Operations | ||
190 | ============================================= | ||
191 | |||
192 | * Ops driven by git/code-review - not by humans running commands | ||
193 | * Run as many things CD as possible | ||
194 | * Infrastructure team operates the same as the project | ||
195 | * Core reviewer status and root access are earned | ||
196 | * Human-initiated ops actions (running commands, clicking a UI) are a bug | ||
197 | * Keys/secrets are not Open :) | ||
198 | |||
199 | We're Not Alone | ||
200 | =============== | ||
201 | |||
202 | * Dependencies (libvirt/kvm/xen, mysql/pg, rabbit, | ||
203 | python/javascript, ceph/gluster, ansible/salt/puppet/chef, ovs/odl) | ||
204 | * Adjacencies (kubernetes, ansible, terraform, opnfv, spinnaker) | ||
205 | * Vendors (plugins, products, services, distros) | ||
206 | |||
207 | Developer Process In a Nutshell | ||
208 | =============================== | ||
209 | |||
210 | * Code Review - nobody has direct commit/push access | ||
211 | * 3rd-Party CI for vendors | ||
212 | * Gated Commits | ||
213 | |||
214 | @jessfraz | ||
215 | ========= | ||
216 | |||
217 | dear @github, | ||
218 | |||
219 | hope you are well, love ya so much | ||
220 | |||
221 | can I get a "squash and merge once | ||
222 | |||
223 | CI passes if no other changes are | ||
224 | |||
225 | pushed" button | ||
226 | |||
227 | k thx | ||
228 | |||
229 | Gated Commits | ||
230 | ============= | ||
231 | |||
232 | :: | ||
233 | |||
234 | Hack Review Test | ||
235 | ========= ========== ========== | ||
236 | |||
237 | push approve | ||
238 | +-------------+ +-------------+ | ||
239 | | | | | | ||
240 | +------+--+ +--v----+--+ +--v-------+ | ||
241 | | | | | | | | ||
242 | | $EDITOR | | Gerrit | | Zuul | | ||
243 | | | | | | | | ||
244 | +------^--+ +--+----^--+ +--+-------+ | ||
245 | | | | | | ||
246 | +-------------+ +-------------+ | ||
247 | clone merge | ||
248 | |||
249 | Gating | ||
250 | ====== | ||
251 | |||
252 | Every change proposed for a repository is tested before it merges. | ||
253 | |||
254 | Co-gating | ||
255 | ========= | ||
256 | |||
257 | Changes to a set of repositories merge monotonically such | ||
258 | that each change is tested with the current state of all | ||
259 | other related repositories before it merges. | ||
260 | |||
261 | Parallel Co-gating | ||
262 | ================== | ||
263 | |||
264 | Changes are serialized such that each change is tested | ||
265 | with all of the changes ahead of it to satisfy the | ||
266 | gating requirement while being able to run tests for | ||
267 | multiple changes simultaneously. | ||
268 | |||
269 | Approve and move on | ||
270 | =================== | ||
271 | |||
272 | * Reviewers approve changes whenever | ||
273 | * Automation correctly deals with the rest | ||
274 | |||
275 | Zuul | ||
276 | ==== | ||
277 | |||
278 | * Multi-repo parallel co-gating engine | ||
279 | * When to run | ||
280 | * Where to run it | ||
281 | * With what git states | ||
282 | * How to respond to results | ||
283 | |||
284 | Zuul Architecture | ||
285 | ================= | ||
286 | |||
287 | .. ansi:: images/architecture.ans | ||
288 | |||
289 | Not just for OpenStack | ||
290 | ====================== | ||
291 | |||
292 | * Zuul v3 is in production for OpenStack (in OpenStack VMs) | ||
293 | |||
294 | Also running at: | ||
295 | |||
296 | * BMW (control plane in OpenShift) | ||
297 | * GoDaddy (control plane in Kubernetes) | ||
298 | * Red Hat | ||
299 | * OpenContrail | ||
300 | * OpenLab | ||
301 | * others ... | ||
302 | |||
303 | Zuul Core Team Corporate Overlords | ||
304 | ================================== | ||
305 | |||
306 | * BMW | ||
307 | * GitHub | ||
308 | * GoDaddy | ||
309 | * OpenStack Foundation | ||
310 | * Red Hat | ||
311 | * SuSE | ||
312 | |||
313 | Zuul is not a general purpose automation framework | ||
314 | ================================================== | ||
315 | |||
316 | Zuul in a nutshell | ||
317 | ================== | ||
318 | |||
319 | * Listens for code events | ||
320 | * Prepares appropriate job config and git repo states | ||
321 | * Allocates nodes for test jobs | ||
322 | * Pushes git repo states to nodes | ||
323 | * Runs user-defined Ansible playbooks | ||
324 | * Collects/reports results | ||
325 | * Potentially merges change | ||
326 | |||
327 | Zuul Simulation | ||
328 | =============== | ||
329 | .. transition:: pan | ||
330 | .. container:: handout | ||
331 | |||
332 | * That was a lot of words - let's walk through it one step at a time | ||
333 | * Here we have two git repos, called nova and keystone, and their | ||
334 | current HEAD state | ||
335 | |||
336 | .. ansi:: images/zsim-00.ans | ||
337 | |||
338 | Zuul Simulation | ||
339 | =============== | ||
340 | .. transition:: cut | ||
341 | .. container:: handout | ||
342 | |||
343 | * A change is approved for Nova | ||
344 | |||
345 | .. ansi:: images/zsim-01.ans | ||
346 | |||
347 | Zuul Simulation | ||
348 | =============== | ||
349 | .. transition:: cut | ||
350 | .. container:: handout | ||
351 | |||
352 | * Zuul starts running jobs for it | ||
353 | * The tests will test the current state of nova and keystone PLUS this nova | ||
354 | change | ||
355 | |||
356 | .. ansi:: images/zsim-02.ans | ||
357 | |||
358 | Zuul Simulation | ||
359 | =============== | ||
360 | .. transition:: cut | ||
361 | .. container:: handout | ||
362 | |||
363 | * A change is approved for Keystone | ||
364 | |||
365 | .. ansi:: images/zsim-03.ans | ||
366 | |||
367 | Zuul Simulation | ||
368 | =============== | ||
369 | .. transition:: cut | ||
370 | .. container:: handout | ||
371 | |||
372 | * The tests will test the current state of nova and keystone PLUS this nova | ||
373 | change | ||
374 | |||
375 | .. ansi:: images/zsim-04.ans | ||
376 | |||
377 | Zuul Simulation | ||
378 | =============== | ||
379 | .. transition:: cut | ||
380 | .. container:: handout | ||
381 | |||
382 | * todo | ||
383 | |||
384 | .. ansi:: images/zsim-05.ans | ||
385 | |||
386 | Zuul Simulation | ||
387 | =============== | ||
388 | .. transition:: cut | ||
389 | .. container:: handout | ||
390 | |||
391 | * todo | ||
392 | |||
393 | .. ansi:: images/zsim-06.ans | ||
394 | |||
395 | Zuul Simulation | ||
396 | =============== | ||
397 | .. transition:: cut | ||
398 | .. container:: handout | ||
399 | |||
400 | * todo | ||
401 | |||
402 | .. ansi:: images/zsim-07.ans | ||
403 | |||
404 | Zuul Simulation | ||
405 | =============== | ||
406 | .. transition:: cut | ||
407 | .. container:: handout | ||
408 | |||
409 | * todo | ||
410 | |||
411 | .. ansi:: images/zsim-08.ans | ||
412 | |||
413 | Zuul Simulation | ||
414 | =============== | ||
415 | .. transition:: cut | ||
416 | .. container:: handout | ||
417 | |||
418 | * todo | ||
419 | |||
420 | .. ansi:: images/zsim-09.ans | ||
421 | |||
422 | Zuul Simulation | ||
423 | =============== | ||
424 | .. transition:: cut | ||
425 | .. container:: handout | ||
426 | |||
427 | * todo | ||
428 | |||
429 | .. ansi:: images/zsim-10.ans | ||
430 | |||
431 | Zuul Simulation | ||
432 | =============== | ||
433 | .. transition:: cut | ||
434 | .. container:: handout | ||
435 | |||
436 | * todo | ||
437 | |||
438 | .. ansi:: images/zsim-11.ans | ||
439 | |||
440 | Zuul Simulation | ||
441 | =============== | ||
442 | .. transition:: cut | ||
443 | .. container:: handout | ||
444 | |||
445 | * todo | ||
446 | |||
447 | .. ansi:: images/zsim-12.ans | ||
448 | |||
449 | Zuul Simulation | ||
450 | =============== | ||
451 | .. transition:: cut | ||
452 | .. container:: handout | ||
453 | |||
454 | * todo | ||
455 | |||
456 | .. ansi:: images/zsim-13.ans | ||
457 | |||
458 | Zuul Simulation | ||
459 | =============== | ||
460 | .. transition:: cut | ||
461 | .. container:: handout | ||
462 | |||
463 | * todo | ||
464 | |||
465 | .. ansi:: images/zsim-14.ans | ||
466 | |||
467 | Zuul Simulation | ||
468 | =============== | ||
469 | .. transition:: cut | ||
470 | .. container:: handout | ||
471 | |||
472 | * todo | ||
473 | |||
474 | .. ansi:: images/zsim-15.ans | ||
475 | |||
476 | Zuul Simulation | ||
477 | =============== | ||
478 | .. transition:: cut | ||
479 | .. container:: handout | ||
480 | |||
481 | * todo | ||
482 | |||
483 | .. ansi:: images/zsim-16.ans | ||
484 | |||
485 | Zuul Simulation | ||
486 | =============== | ||
487 | .. transition:: cut | ||
488 | .. container:: handout | ||
489 | |||
490 | * todo | ||
491 | |||
492 | .. ansi:: images/zsim-17.ans | ||
493 | |||
494 | Zuul Simulation | ||
495 | =============== | ||
496 | .. transition:: cut | ||
497 | .. container:: handout | ||
498 | |||
499 | * todo | ||
500 | |||
501 | .. ansi:: images/zsim-18.ans | ||
502 | |||
503 | Zuul Simulation | ||
504 | =============== | ||
505 | .. transition:: cut | ||
506 | .. container:: handout | ||
507 | |||
508 | * todo | ||
509 | |||
510 | .. ansi:: images/zsim-19.ans | ||
511 | |||
512 | Zuul Simulation | ||
513 | =============== | ||
514 | .. transition:: cut | ||
515 | .. container:: handout | ||
516 | |||
517 | * todo | ||
518 | |||
519 | .. ansi:: images/zsim-20.ans | ||
520 | |||
521 | Zuul Simulation | ||
522 | =============== | ||
523 | .. transition:: cut | ||
524 | .. container:: handout | ||
525 | |||
526 | * todo | ||
527 | |||
528 | .. ansi:: images/zsim-21.ans | ||
529 | |||
530 | Zuul Simulation | ||
531 | =============== | ||
532 | .. transition:: cut | ||
533 | .. container:: handout | ||
534 | |||
535 | * todo | ||
536 | |||
537 | .. ansi:: images/zsim-22.ans | ||
538 | |||
539 | Jobs | ||
540 | ==== | ||
541 | |||
542 | * Jobs run on nodes from nodepool (static or dynamic) | ||
543 | * Metadata defined in Zuul's configuration | ||
544 | * Execution content in Ansible (with live streaming!) | ||
545 | * Jobs may be defined centrally or in the repo being tested | ||
546 | * Jobs have contextual variants that simplify configuration | ||
547 | |||
548 | Shared Job Configs | ||
549 | ================== | ||
550 | |||
551 | * Job config repos are all in git | ||
552 | * Designed to support directly sharing job configurations | ||
553 | * git.zuul-ci.org/zuul-jobs repo is a 'standard library' | ||
554 | to be directly shared between zuul installations | ||
555 | |||
556 | Job | ||
557 | === | ||
558 | |||
559 | .. code:: yaml | ||
560 | |||
561 | - job: | ||
562 | name: base | ||
563 | parent: null | ||
564 | description: | | ||
565 | The base job for Zuul. | ||
566 | timeout: 1800 | ||
567 | nodeset: | ||
568 | nodes: | ||
569 | - name: primary | ||
570 | label: centos-7 | ||
571 | pre-run: playbooks/base/pre.yaml | ||
572 | post-run: | ||
573 | - playbooks/base/post-ssh.yaml | ||
574 | - playbooks/base/post-logs.yaml | ||
575 | secrets: | ||
576 | - site_logs | ||
577 | |||
578 | Simple Job | ||
579 | ========== | ||
580 | |||
581 | .. code:: yaml | ||
582 | |||
583 | - job: | ||
584 | name: tox | ||
585 | pre-run: playbooks/setup-tox.yaml | ||
586 | run: playbooks/tox.yaml | ||
587 | post-run: playbooks/fetch-tox-output.yaml | ||
588 | |||
589 | - job: | ||
590 | name: tox-py27 | ||
591 | parent: tox | ||
592 | vars: | ||
593 | tox_envlist: py27 | ||
594 | |||
595 | Playbooks | ||
596 | ========= | ||
597 | |||
598 | * Jobs run Ansible playbooks | ||
599 | * Playbooks may be defined centrally or in the repo being tested | ||
600 | * Playbooks can use roles from current or other Zuul repos | ||
601 | (or Galaxy, coming soon) | ||
602 | * Playbooks are run on the zuul-executor using bubblewrap | ||
603 | https://github.com/projectatomic/bubblewrap | ||
604 | * Playbooks are not allowed to execute content on 'localhost' | ||
605 | |||
606 | Devstack-gate / Tempest Playbook | ||
607 | ================================ | ||
608 | |||
609 | .. code:: yaml | ||
610 | |||
611 | # devstack-gate / tempest playbook | ||
612 | hosts: all | ||
613 | roles: | ||
614 | - setup-multinode-networking | ||
615 | - partition-swap | ||
616 | - configure-mirrors | ||
617 | - run-devstack | ||
618 | - run-tempest | ||
619 | |||
620 | Simple Shell Playbook | ||
621 | ===================== | ||
622 | |||
623 | .. code:: yaml | ||
624 | |||
625 | hosts: controller | ||
626 | tasks: | ||
627 | - shell: ./run_tests.sh | ||
628 | |||
629 | Test Like Production | ||
630 | ==================== | ||
631 | |||
632 | If you use Ansible for deployment, your test and deployment processes | ||
633 | and playbooks are the same | ||
634 | |||
635 | What if you don't use Ansible? | ||
636 | ============================== | ||
637 | |||
638 | OpenStack Infra Control Plane (currently) uses Puppet | ||
639 | ===================================================== | ||
640 | |||
641 | .. code:: yaml | ||
642 | |||
643 | # In git.openstack.org/openstack-infra/project-config/roles/legacy-install-afs-with-puppet/tasks/main.yaml | ||
644 | - name: Install puppet | ||
645 | shell: ./install_puppet.sh | ||
646 | args: | ||
647 | chdir: "{{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/system-config" | ||
648 | environment: | ||
649 | # Skip setting up pip, our images have already done this. | ||
650 | SETUP_PIP: "false" | ||
651 | become: yes | ||
652 | |||
653 | - name: Copy manifest | ||
654 | copy: | ||
655 | src: manifest.pp | ||
656 | dest: "{{ ansible_user_dir }}/manifest.pp" | ||
657 | |||
658 | - name: Run puppet | ||
659 | puppet: | ||
660 | manifest: "{{ ansible_user_dir }}/manifest.pp" | ||
661 | become: yes | ||
662 | |||
663 | |||
664 | Cross-Project Example Problem | ||
665 | ============================= | ||
666 | |||
667 | * User reports bug in shade - auto_ip is not discovering their NAT properly | ||
668 | * Two fixes, one to detection algorithm, one to config override | ||
669 | * Config override requires adding support to os-client-config | ||
670 | * Once support is added to os-client-config, it can be consumed in shade | ||
671 | * How do we integration test this without releasing os-client-config? | ||
672 | |||
673 | Cross-Project Dependencies | ||
674 | ========================== | ||
675 | |||
676 | Testing or gating dependencies (including jobs) manually specified by | ||
677 | developers | ||
678 | |||
679 | .. container:: progressive | ||
680 | |||
681 | * shade https://review.openstack.org/513913/ | ||
682 | |||
683 | Add unittest tips jobs | ||
684 | |||
685 | Change-ID: I5b411be5c5aa43535fa89a51d6099aadd7a8ea60 | ||
686 | * os-client-config https://review.openstack.org/513915 | ||
687 | |||
688 | Add shade-tox-tips jobs | ||
689 | |||
690 | Change-ID: Ie3e9a4deca1d74b94e810e87e130706fe15fe2c9 | ||
691 | |||
692 | Depends-On: https://review.openstack.org/513913/ | ||
693 | * os-client-config https://review.openstack.org/513751/ | ||
694 | |||
695 | Added nat_source flag for networks | ||
696 | |||
697 | Change-ID: I3d8dd6d734a1013d2d4a43e11c3538c3a345820b | ||
698 | |||
699 | * shade https://review.openstack.org/#/c/513914 | ||
700 | |||
701 | Add support for configured NAT source variable | ||
702 | |||
703 | Change-Id: I4b50c2323a487b5ce90f9d38a48be249cfb739c5 | ||
704 | |||
705 | Depends-On: https://review.openstack.org/513914 | ||
706 | |||
707 | shade: Add unittest tips jobs | ||
708 | ============================= | ||
709 | |||
710 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
711 | |||
712 | .. code:: yaml | ||
713 | |||
714 | - job: | ||
715 | name: shade-tox-py27-tips | ||
716 | parent: openstack-tox-py27 | ||
717 | description: | | ||
718 | Run tox python 27 unittests against master of important libs | ||
719 | required-projects: | ||
720 | - openstack-infra/shade | ||
721 | - openstack/keystoneauth | ||
722 | - openstack/os-client-config | ||
723 | |||
724 | - job: | ||
725 | name: shade-tox-py35-tips | ||
726 | parent: openstack-tox-py35 | ||
727 | description: | | ||
728 | Run tox python 35 unittests against master of important libs | ||
729 | required-projects: | ||
730 | - openstack-infra/shade | ||
731 | - openstack/keystoneauth | ||
732 | - openstack/os-client-config | ||
733 | |||
734 | shade: Add unittest tips project-template | ||
735 | ========================================= | ||
736 | |||
737 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
738 | |||
739 | .. code:: yaml | ||
740 | |||
741 | - project-template: | ||
742 | name: shade-tox-tips | ||
743 | check: | ||
744 | jobs: | ||
745 | - shade-tox-py27-tips | ||
746 | - shade-tox-py35-tips | ||
747 | gate: | ||
748 | jobs: | ||
749 | - shade-tox-py27-tips | ||
750 | - shade-tox-py35-tips | ||
751 | |||
752 | shade: Add unittest tips project-template to project | ||
753 | ==================================================== | ||
754 | |||
755 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
756 | |||
757 | .. code:: yaml | ||
758 | |||
759 | - project: | ||
760 | templates: | ||
761 | - publish-to-pypi | ||
762 | - publish-openstack-sphinx-docs | ||
763 | - shade-tox-tips | ||
764 | |||
765 | os-client-config: Add shade-tox-tips jobs | ||
766 | ========================================= | ||
767 | |||
768 | * In git.openstack.org/openstack/os-client-config/.zuul.yaml: | ||
769 | |||
770 | .. code:: yaml | ||
771 | |||
772 | - project: | ||
773 | templates: | ||
774 | - shade-tox-tips | ||
775 | |||
776 | os-client-config: Add nat_source flag for networks | ||
777 | ================================================== | ||
778 | |||
779 | :: | ||
780 | |||
781 | diff --git a/os_client_config/cloud_config.py b/os_client_config/cloud_config.py | ||
782 | index 2e97629..d1a6983 100644 | ||
783 | --- a/os_client_config/cloud_config.py | ||
784 | +++ b/os_client_config/cloud_config.py | ||
785 | @@ -581,3 +581,10 @@ class CloudConfig(object): | ||
786 | if net['nat_destination']: | ||
787 | return net['name'] | ||
788 | return None | ||
789 | + | ||
790 | + def get_nat_source(self): | ||
791 | + """Get network used for NAT source.""" | ||
792 | + for net in self.config['networks']: | ||
793 | + if net.get('nat_source'): | ||
794 | + return net['name'] | ||
795 | + return None | ||
796 | |||
797 | shade: Add support for configured NAT source variable | ||
798 | ===================================================== | ||
799 | |||
800 | :: | ||
801 | |||
802 | Zuul 10-21 13:57 | ||
803 | Patch Set 5: Verified-1 | ||
804 | Build failed. | ||
805 | openstack-tox-pep8 SUCCESS in 2m 29s | ||
806 | openstack-tox-py27 FAILURE in 2m 34s | ||
807 | build-openstack-releasenotes SUCCESS in 2m 47s | ||
808 | openstack-tox-py35 FAILURE in 2m 41s | ||
809 | openstack-tox-cover POST_FAILURE in 3m 52s (non-voting) | ||
810 | build-openstack-sphinx-docs SUCCESS in 2m 57s | ||
811 | shade-tox-py27-tips SUCCESS in 3m 18s | ||
812 | shade-tox-py35-tips SUCCESS in 2m 28s | ||
813 | |||
814 | OpenStack Github Support for Cross Community Testing | ||
815 | ==================================================== | ||
816 | |||
817 | * OpenStack does not use Github, but other people do | ||
818 | * Github App "OpenStack Zuul" | ||
819 | * App added to github project by project admin | ||
820 | * Project aded to OpenStack's main.yaml | ||
821 | * Test interactions between OpenStack and important adjacent communities | ||
822 | * https://github.com/ansible/ansible/pull/20974 | ||
823 | |||
824 | Cross Source Dependencies | ||
825 | ========================= | ||
826 | |||
827 | .. container:: progressive | ||
828 | |||
829 | * shade https://review.openstack.org/539563 | ||
830 | |||
831 | Shift voting flag and test_matrix_branch for ansible-devel job | ||
832 | |||
833 | Change-ID: Ic9d3983de641dbe618c65b2cbf2dcfa3686575df | ||
834 | |||
835 | * ansible https://github.com/ansible/ansible/pull/34925 | ||
836 | |||
837 | continue fact gathering even without dmidecode | ||
838 | |||
839 | * ansible https://github.com/ansible/ansible/pull/20974 | ||
840 | |||
841 | Make a generalized OpenStack cloud constructor | ||
842 | |||
843 | Depends-On: https://review.openstack.org/539563 | ||
844 | Depends-On: https://github.com/ansible/ansible/pull/34925 | ||
845 | |||
846 | Nodesets for Multi-node Jobs | ||
847 | ============================ | ||
848 | |||
849 | .. code:: yaml | ||
850 | |||
851 | - nodeset: | ||
852 | name: ceph-cluster | ||
853 | nodes: | ||
854 | - name: controller | ||
855 | label: centos-7 | ||
856 | - name: compute1 | ||
857 | label: fedora-26 | ||
858 | - name: compute2 | ||
859 | label: fedora-26 | ||
860 | groups: | ||
861 | - name: ceph-osd | ||
862 | nodes: | ||
863 | - controller | ||
864 | - name: ceph-monitor | ||
865 | nodes: | ||
866 | - controller | ||
867 | - compute1 | ||
868 | - compute2 | ||
869 | |||
870 | Multi-node Job | ||
871 | ============== | ||
872 | |||
873 | * nodesets are provided to Ansible for jobs in inventory | ||
874 | |||
875 | .. code:: yaml | ||
876 | |||
877 | - job: | ||
878 | name: ceph-multinode | ||
879 | nodeset: ceph-cluster | ||
880 | run: playbooks/install-ceph.yaml | ||
881 | |||
882 | Multi-node Ceph Job Content | ||
883 | =========================== | ||
884 | |||
885 | .. code:: yaml | ||
886 | |||
887 | - hosts: all | ||
888 | roles: | ||
889 | - install-ceph | ||
890 | - hosts: ceph-osd | ||
891 | roles: | ||
892 | - start-ceph-osd | ||
893 | - hosts: ceph-monitor | ||
894 | roles: | ||
895 | - start-ceph-monitor | ||
896 | - hosts: all | ||
897 | roles: | ||
898 | - do-something-interesting | ||
899 | |||
900 | Projects | ||
901 | ======== | ||
902 | |||
903 | * Projects are git repositories | ||
904 | * Specify a set of jobs for each pipeline | ||
905 | * golang git repo naming as been adopted: | ||
906 | |||
907 | :: | ||
908 | |||
909 | zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d | ||
910 | /home/zuul/src/git.openstack.org/openstack-infra/shade | ||
911 | /home/zuul/src/git.openstack.org/openstack/keystoneauth | ||
912 | /home/zuul/src/git.openstack.org/openstack/os-client-config | ||
913 | /home/zuul/src/github.com/ansible/ansible | ||
914 | |||
915 | Project with Job Dependencies | ||
916 | ============================= | ||
917 | |||
918 | .. code:: yaml | ||
919 | |||
920 | # In git.openstack.org/openstack-infra/project-config: | ||
921 | - project: | ||
922 | name: openstack/nova | ||
923 | release: | ||
924 | jobs: | ||
925 | - build-artifacts | ||
926 | - upload-tarball: | ||
927 | dependencies: build-artifacts | ||
928 | - upload-pypi: | ||
929 | dependencies: build-artifacts | ||
930 | - notify-mirror: | ||
931 | dependencies: | ||
932 | - upload-tarball | ||
933 | - upload-pypi | ||
934 | |||
935 | Secrets | ||
936 | ======= | ||
937 | |||
938 | * Inspired by Kubernetes Secrets API | ||
939 | * Projects can add named encrypted secrets to their .zuul.yaml file | ||
940 | * Jobs can request to use secrets by name | ||
941 | * Jobs using secrets are not reconfigured speculatively | ||
942 | * Secrets can only be used by the same project they are defined in | ||
943 | * Public key per project: | ||
944 | ``{{ zuul_url }}/tenant/{{ tenant }}/key/{{ project }}.pub`` | ||
945 | |||
946 | Secret Example (note, no admins had to enable this) | ||
947 | =================================================== | ||
948 | |||
949 | .. code:: yaml | ||
950 | |||
951 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
952 | - secret: | ||
953 | name: loci_docker_login | ||
954 | data: | ||
955 | user: !encrypted/pkcs1-oaep | ||
956 | - r8Nbpq5olmfLF035BZ/CUoFLIdhvBi/49KuochOAHbvns+xMiho3C7MEFzYDqJX3IhHde | ||
957 | BICYOgK7qnyINOIZL2e7pl75rEdHQwJjSFUMkpdY6wEP7f9hpolj9xVp0ifHUVQqPHMRn | ||
958 | zoPFd8MEAHxH5GLmc2SWJ98E/QUqGltxBi1YRSZoCcNtq3tHFK5Y+xQlLhIseJ2HkpDs6 | ||
959 | YXOGP9Qt4Va6sdyBcA90H+apSAcYA3Duu962ySZQAsYNui/3NQq3gLA+OZeyTJtcrh4hj | ||
960 | Rb5dBnDWfSrMpxdNkbPXXgbQaxO3T0L4jbaOF8VKEsiI9olBrOeV2M9ddYJjSsHGj4XR8 | ||
961 | 4vwS0+doB7np93fujiDuHVgdG8R40NW2GznyKRlRtzAORla7Mzw1Y1MokcUyY6p1LlLLl | ||
962 | wUuWYCCEuRciOPhZXQ2u42qju/zrK2/dPnO8HfUINSrN0WbNq14ZwPpbj0ro02oGPbtwu | ||
963 | OTw1z+N0Nc+GuLWlwYJGYM/z0UnvDR3WEBc2kXbVev9w4n0cB3RyphML2PDZZWbw8tjnX | ||
964 | h1VsAOJ0Qo4qq1K/ft95ypd+vtjkfepEgHEBmJNwutJa9IHAkGfrkO9VkpUTPpfffnPwz | ||
965 | d0/zaaadNl6MLQUSutRwY23YIIbv+fmukxw2vnJmvn6abkBlMya7KgtifwNA8c= | ||
966 | password: !encrypted/pkcs1-oaep | ||
967 | - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw | ||
968 | hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE | ||
969 | 5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE | ||
970 | 5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo | ||
971 | 4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p | ||
972 | taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA | ||
973 | hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T | ||
974 | IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq | ||
975 | 7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj | ||
976 | /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM= | ||
977 | |||
978 | Secret Example | ||
979 | ============== | ||
980 | |||
981 | .. code:: yaml | ||
982 | |||
983 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
984 | - job: | ||
985 | name: publish-loci-cinder | ||
986 | parent: loci-cinder | ||
987 | post-run: playbooks/push | ||
988 | secrets: | ||
989 | - loci_docker_login | ||
990 | |||
991 | # In git.openstack.org/openstack/loci/playbooks/push.yaml: | ||
992 | - hosts: all | ||
993 | tasks: | ||
994 | - include_vars: vars.yaml | ||
995 | |||
996 | - name: Push project to DockerHub | ||
997 | block: | ||
998 | - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }} | ||
999 | no_log: True | ||
1000 | - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} | ||
1001 | with_items: "{{ distros }}" | ||
1002 | |||
1003 | |||
1004 | What's Next? | ||
1005 | ============ | ||
1006 | |||
1007 | * MQTT publisher | ||
1008 | * node providers | ||
1009 | |||
1010 | * kuberenetes | ||
1011 | * OCI/docker | ||
1012 | * Mac Stadium (for our Ansible friends) | ||
1013 | * ec2/gce/azure | ||
1014 | |||
1015 | * native container/kubernetes job execution | ||
1016 | |||
1017 | Important Links | ||
1018 | =============== | ||
1019 | |||
1020 | * https://zuul-ci.org/ | ||
1021 | * https://git.zuul-ci.org/cgit/zuul | ||
1022 | * https://zuul-ci.org/docs/zuul | ||
1023 | * https://zuul-ci.org/docs/zuul-jobs/ | ||
1024 | * https://docs.openstack.org/infra/manual/zuulv3.html | ||
1025 | * https://docs.openstack.org/infra/openstack-zuul-jobs/ | ||
1026 | * https://storyboard.openstack.org/#!/project/679 | ||
1027 | * https://storyboard.openstack.org/#!/board/41 | ||
1028 | * freenode:#zuul | ||
1029 | |||
1030 | Questions | ||
1031 | ========= | ||
1032 | |||
1033 | .. ansi:: images/questions.ans | ||
1034 | |||
1035 | Presentty | ||
1036 | ========= | ||
1037 | .. hidetitle:: | ||
1038 | .. transition:: pan | ||
1039 | .. figlet:: Presentty | ||
1040 | |||
1041 | * Console presentations written in reStructuredText | ||
1042 | * Cross-fade, pan, tilt, cut transitions | ||
1043 | * Figlet, cowsay! | ||
1044 | * https://pypi.python.org/pypi/presentty | ||