diff options
Diffstat (limited to 'src/infra/infra.rst')
-rw-r--r-- | src/infra/infra.rst | 1239 |
1 files changed, 1239 insertions, 0 deletions
diff --git a/src/infra/infra.rst b/src/infra/infra.rst new file mode 100644 index 0000000..4b199d3 --- /dev/null +++ b/src/infra/infra.rst | |||
@@ -0,0 +1,1239 @@ | |||
1 | . display in 68x24 | ||
2 | .. display in 88x24 | ||
3 | |||
4 | .. pygments yaml? (only file breaks (---) tinted) | ||
5 | .. slide on high level v3 changes | ||
6 | .. slide on nodepool | ||
7 | |||
8 | .. transition:: dissolve | ||
9 | :duration: 0.4 | ||
10 | |||
11 | Test Slide | ||
12 | ========== | ||
13 | .. hidetitle:: | ||
14 | |||
15 | .. ansi:: images/testslide.ans | ||
16 | |||
17 | OpenStack | ||
18 | ========= | ||
19 | |||
20 | tl;dr | ||
21 | ===== | ||
22 | |||
23 | * multi repo | ||
24 | * integrated deliverable | ||
25 | * gated commits | ||
26 | * open tooling | ||
27 | * nobody is special | ||
28 | * there is no Dana, only Zuul | ||
29 | |||
30 | OpenStack Is | ||
31 | ============ | ||
32 | |||
33 | * Federated | ||
34 | * Distributed | ||
35 | * Large | ||
36 | * Open | ||
37 | * Not Alone | ||
38 | |||
39 | Federated | ||
40 | ========= | ||
41 | |||
42 | * Hundreds of involved companies | ||
43 | * No 'main' company | ||
44 | * "Decisions are made by those who show up" | ||
45 | * Union of priorities/use cases | ||
46 | |||
47 | Impact of being Federated | ||
48 | ========================= | ||
49 | |||
50 | * No company can appoint humans to project positions | ||
51 | * The project cannot fire anyone | ||
52 | * Variable background of contributors | ||
53 | * Heavy reliance on consensus-oriented democracy | ||
54 | |||
55 | Distributed | ||
56 | =========== | ||
57 | |||
58 | * There is no office | ||
59 | * Contributor base is global | ||
60 | * Multitude of contributor backgrounds | ||
61 | |||
62 | Impact of being Distributed | ||
63 | =========================== | ||
64 | |||
65 | * Constantly at odds with American Exceptionalism | ||
66 | * Tooling must empower all contributors, regardless of background, | ||
67 | skill level or cultural context | ||
68 | * Heavy preference for text-based communication | ||
69 | * Cannot assume US-centric needs or solutions | ||
70 | |||
71 | Large numbers of | ||
72 | ================ | ||
73 | |||
74 | * Contributors (\~2k in any given 6 month period) | ||
75 | * Changes | ||
76 | * Code Repositories (1889 as of this morning) | ||
77 | |||
78 | OpenStack Scale Comparison | ||
79 | ========================== | ||
80 | |||
81 | * 2KJPH (2,000 jobs per hour) | ||
82 | * Nodes from 13 Regions of 5 Public and 2 Private OpenStack Clouds | ||
83 | * Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone | ||
84 | * 10,000 changes merged per month | ||
85 | |||
86 | OpenStack Scale Comparison | ||
87 | ========================== | ||
88 | |||
89 | * 2KJPH (2,000 jobs per hour) | ||
90 | * Nodes from 12 Regions of 5 Public and 1 Private OpenStack Clouds | ||
91 | * Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro, Limestone | ||
92 | * 10,000 changes merged per month | ||
93 | |||
94 | By comparison, our friends at the amazing project Ansible received | ||
95 | 13,000 changes and had merged 8,000 of them in its first 4 years. | ||
96 | |||
97 | Four Opens | ||
98 | ========== | ||
99 | |||
100 | * Open Source | ||
101 | (we don't hold back Enterprise features, we don't cripple things) | ||
102 | * Open Design | ||
103 | (design process open to all, decisions are not made inside company doors) | ||
104 | * Open Development | ||
105 | (public source code, public code review, all code is reviewed and gated) | ||
106 | * Open Community | ||
107 | (lazy consensus, democratic leadership from participants, | ||
108 | public logged meetings in IRC, public archived mailing lists) | ||
109 | |||
110 | Nobody is Special | ||
111 | ================= | ||
112 | |||
113 | * No dictators | ||
114 | * Aggressively egalitarian | ||
115 | * No "pay for play" | ||
116 | |||
117 | Fifth Open - Four Opens Applied to the Infrastructure | ||
118 | ===================================================== | ||
119 | |||
120 | * All tools must be Open | ||
121 | * Any external services must by Open | ||
122 | * Strongly avoid single-vendor | ||
123 | |||
124 | Multi-Vendor Open Tooling | ||
125 | ========================= | ||
126 | |||
127 | * Philosophical | ||
128 | * Pragmatic | ||
129 | |||
130 | Sixth Open - Four Opens Applied to Operations | ||
131 | ============================================= | ||
132 | |||
133 | * Ops driven by git/code-review - not by humans running commands | ||
134 | * Run as many things CD as possible | ||
135 | * Infrastructure team operates the same as the project | ||
136 | * Core reviewer status and root access are earned | ||
137 | * Human-initiated ops actions (running commands, clicking a UI) are a bug | ||
138 | * Keys/secrets are not Open :) | ||
139 | |||
140 | We're Not Alone | ||
141 | =============== | ||
142 | |||
143 | * Dependencies (libvirt/kvm/xen, mysql/pg, rabbit, | ||
144 | python/javascript, ceph/gluster, ansible/salt/puppet/chef, ovs/odl) | ||
145 | * Adjacencies (kubernetes, ansible, terraform, opnfv, spinnaker) | ||
146 | * Vendors (plugins, products, services, distros) | ||
147 | |||
148 | In a Nutshell | ||
149 | ============= | ||
150 | |||
151 | * Code Review - nobody has commit/push access | ||
152 | * Gated Commits | ||
153 | * 3rd-Party CI for vendors | ||
154 | |||
155 | Infra Responsibilities | ||
156 | ====================== | ||
157 | |||
158 | * Communication and Planning Systems | ||
159 | * Source Code (code hosting, code review) | ||
160 | * Contributor Feedback and Reporting | ||
161 | * Content Publication | ||
162 | * Automated Gating | ||
163 | |||
164 | All of our systems run in OpenStack Public Clouds | ||
165 | ================================================= | ||
166 | |||
167 | Communication and Planning Systems | ||
168 | ================================== | ||
169 | |||
170 | * Etherpad http://etherpad.openstack.org | ||
171 | * Pastebin http://paste.openstack.org | ||
172 | * Issue/Feature Tracking (storyboard) http://storyboard.openstack.org | ||
173 | * Forum (askbot) http://ask.openstack.org | ||
174 | * Mailing Lists (mailman) http://lists.openstack.org | ||
175 | * IRC registration, meetings, bots and logging http://eavesdrop.openstack.org | ||
176 | * Meeting planning (yaml2ical) | ||
177 | * PBX (asterisk) | ||
178 | |||
179 | Source Code | ||
180 | =========== | ||
181 | |||
182 | * Code Review (gerrit) http://review.openstack.org | ||
183 | * git replica farm (git/cgit) http://git.openstack.org | ||
184 | * Code Search (hound) http://codesearch.openstack.org | ||
185 | |||
186 | Contributor Feedback and Reporting | ||
187 | ================================== | ||
188 | |||
189 | * Build Logs (apache) http://logs.openstack.org | ||
190 | * Log Indexing (ELK) http://logstash.openstack.org | ||
191 | * Known Failure Identification http://status.openstack.org/elastic-recheck/ | ||
192 | * Test Trending http://status.openstack.org/openstack-health/#/ | ||
193 | * Metrics/Statistics http://grafana.openstack.org/ | ||
194 | |||
195 | Content Publication | ||
196 | =================== | ||
197 | |||
198 | * Documentation http://docs.openstack.org | ||
199 | * Tag-driven releases to PyPI, NPM, Maven, etc | ||
200 | |||
201 | Gated Commits | ||
202 | ============= | ||
203 | |||
204 | :: | ||
205 | |||
206 | Hack Review Test | ||
207 | ========= ========== ========== | ||
208 | |||
209 | push approve | ||
210 | +-------------+ +-------------+ | ||
211 | | | | | | ||
212 | +------+--+ +--v----+--+ +--v-------+ | ||
213 | | | | | | | | ||
214 | | $EDITOR | | Gerrit | | Zuul | | ||
215 | | | | | | | | ||
216 | +------^--+ +--+----^--+ +--+-------+ | ||
217 | | | | | | ||
218 | +-------------+ +-------------+ | ||
219 | clone merge | ||
220 | |||
221 | Gating | ||
222 | ====== | ||
223 | |||
224 | Every change proposed for a repository is tested before it merges. | ||
225 | |||
226 | Co-gating | ||
227 | ========= | ||
228 | |||
229 | Changes to a set of repositories merge monotonically such | ||
230 | that each change is tested with the current state of all | ||
231 | other related repositories before it merges. | ||
232 | |||
233 | Parallel Co-gating | ||
234 | ================== | ||
235 | |||
236 | Changes are serialized such that each change is tested | ||
237 | with all of the changes ahead of it to satisfy the | ||
238 | gating requirement while being able to run tests for | ||
239 | multiple changes simultaneously. | ||
240 | |||
241 | Zuul | ||
242 | ==== | ||
243 | |||
244 | * Custom-written multi-repo parallel co-gating engine | ||
245 | * When to run | ||
246 | * Where to run it | ||
247 | * With what git states | ||
248 | * How to respond to results | ||
249 | |||
250 | Not just for OpenStack | ||
251 | ====================== | ||
252 | |||
253 | * Zuul v3 is in production for OpenStack (in OpenStack VMs) | ||
254 | |||
255 | Also running at: | ||
256 | |||
257 | * BMW (control plane in OpenShift) | ||
258 | * Godaddy (control plane in kuberenetes) | ||
259 | * OpenContrail (just learned about that yesterday) | ||
260 | * OpenLab | ||
261 | |||
262 | Zuul is not a general purpose automation framework | ||
263 | ================================================== | ||
264 | |||
265 | Zuul in a nutshell | ||
266 | ================== | ||
267 | |||
268 | * Listens for code events | ||
269 | * Prepares appropriate job config and git repo states | ||
270 | * Allocates nodes for test jobs | ||
271 | * Pushes git repo states to nodes | ||
272 | * Runs user-defined Ansible playbooks | ||
273 | * Collects/reports results | ||
274 | * Potentially merges change | ||
275 | |||
276 | Zuul Simulation | ||
277 | =============== | ||
278 | .. transition:: pan | ||
279 | .. container:: handout | ||
280 | |||
281 | * That was a lot of words - let's walk through it one step at a time | ||
282 | * Here we have two git repos, called nova and keystone, and their | ||
283 | current HEAD state | ||
284 | |||
285 | .. ansi:: images/zsim-00.ans | ||
286 | |||
287 | Zuul Simulation | ||
288 | =============== | ||
289 | .. transition:: cut | ||
290 | .. container:: handout | ||
291 | |||
292 | * A change is approved for Nova | ||
293 | |||
294 | .. ansi:: images/zsim-01.ans | ||
295 | |||
296 | Zuul Simulation | ||
297 | =============== | ||
298 | .. transition:: cut | ||
299 | .. container:: handout | ||
300 | |||
301 | * Zuul starts running jobs for it | ||
302 | * The tests will test the current state of nova and keystone PLUS this nova | ||
303 | change | ||
304 | |||
305 | .. ansi:: images/zsim-02.ans | ||
306 | |||
307 | Zuul Simulation | ||
308 | =============== | ||
309 | .. transition:: cut | ||
310 | .. container:: handout | ||
311 | |||
312 | * A change is approved for Keystone | ||
313 | |||
314 | .. ansi:: images/zsim-03.ans | ||
315 | |||
316 | Zuul Simulation | ||
317 | =============== | ||
318 | .. transition:: cut | ||
319 | .. container:: handout | ||
320 | |||
321 | * The tests will test the current state of nova and keystone PLUS this nova | ||
322 | change | ||
323 | |||
324 | .. ansi:: images/zsim-04.ans | ||
325 | |||
326 | Zuul Simulation | ||
327 | =============== | ||
328 | .. transition:: cut | ||
329 | .. container:: handout | ||
330 | |||
331 | * todo | ||
332 | |||
333 | .. ansi:: images/zsim-05.ans | ||
334 | |||
335 | Zuul Simulation | ||
336 | =============== | ||
337 | .. transition:: cut | ||
338 | .. container:: handout | ||
339 | |||
340 | * todo | ||
341 | |||
342 | .. ansi:: images/zsim-06.ans | ||
343 | |||
344 | Zuul Simulation | ||
345 | =============== | ||
346 | .. transition:: cut | ||
347 | .. container:: handout | ||
348 | |||
349 | * todo | ||
350 | |||
351 | .. ansi:: images/zsim-07.ans | ||
352 | |||
353 | Zuul Simulation | ||
354 | =============== | ||
355 | .. transition:: cut | ||
356 | .. container:: handout | ||
357 | |||
358 | * todo | ||
359 | |||
360 | .. ansi:: images/zsim-08.ans | ||
361 | |||
362 | Zuul Simulation | ||
363 | =============== | ||
364 | .. transition:: cut | ||
365 | .. container:: handout | ||
366 | |||
367 | * todo | ||
368 | |||
369 | .. ansi:: images/zsim-09.ans | ||
370 | |||
371 | Zuul Simulation | ||
372 | =============== | ||
373 | .. transition:: cut | ||
374 | .. container:: handout | ||
375 | |||
376 | * todo | ||
377 | |||
378 | .. ansi:: images/zsim-10.ans | ||
379 | |||
380 | Zuul Simulation | ||
381 | =============== | ||
382 | .. transition:: cut | ||
383 | .. container:: handout | ||
384 | |||
385 | * todo | ||
386 | |||
387 | .. ansi:: images/zsim-11.ans | ||
388 | |||
389 | Zuul Simulation | ||
390 | =============== | ||
391 | .. transition:: cut | ||
392 | .. container:: handout | ||
393 | |||
394 | * todo | ||
395 | |||
396 | .. ansi:: images/zsim-12.ans | ||
397 | |||
398 | Zuul Simulation | ||
399 | =============== | ||
400 | .. transition:: cut | ||
401 | .. container:: handout | ||
402 | |||
403 | * todo | ||
404 | |||
405 | .. ansi:: images/zsim-13.ans | ||
406 | |||
407 | Zuul Simulation | ||
408 | =============== | ||
409 | .. transition:: cut | ||
410 | .. container:: handout | ||
411 | |||
412 | * todo | ||
413 | |||
414 | .. ansi:: images/zsim-14.ans | ||
415 | |||
416 | Zuul Simulation | ||
417 | =============== | ||
418 | .. transition:: cut | ||
419 | .. container:: handout | ||
420 | |||
421 | * todo | ||
422 | |||
423 | .. ansi:: images/zsim-15.ans | ||
424 | |||
425 | Zuul Simulation | ||
426 | =============== | ||
427 | .. transition:: cut | ||
428 | .. container:: handout | ||
429 | |||
430 | * todo | ||
431 | |||
432 | .. ansi:: images/zsim-16.ans | ||
433 | |||
434 | Zuul Simulation | ||
435 | =============== | ||
436 | .. transition:: cut | ||
437 | .. container:: handout | ||
438 | |||
439 | * todo | ||
440 | |||
441 | .. ansi:: images/zsim-17.ans | ||
442 | |||
443 | Zuul Simulation | ||
444 | =============== | ||
445 | .. transition:: cut | ||
446 | .. container:: handout | ||
447 | |||
448 | * todo | ||
449 | |||
450 | .. ansi:: images/zsim-18.ans | ||
451 | |||
452 | Zuul Simulation | ||
453 | =============== | ||
454 | .. transition:: cut | ||
455 | .. container:: handout | ||
456 | |||
457 | * todo | ||
458 | |||
459 | .. ansi:: images/zsim-19.ans | ||
460 | |||
461 | Zuul Simulation | ||
462 | =============== | ||
463 | .. transition:: cut | ||
464 | .. container:: handout | ||
465 | |||
466 | * todo | ||
467 | |||
468 | .. ansi:: images/zsim-20.ans | ||
469 | |||
470 | Zuul Simulation | ||
471 | =============== | ||
472 | .. transition:: cut | ||
473 | .. container:: handout | ||
474 | |||
475 | * todo | ||
476 | |||
477 | .. ansi:: images/zsim-21.ans | ||
478 | |||
479 | Zuul Simulation | ||
480 | =============== | ||
481 | .. transition:: cut | ||
482 | .. container:: handout | ||
483 | |||
484 | * todo | ||
485 | |||
486 | .. ansi:: images/zsim-22.ans | ||
487 | |||
488 | Zuul Architecture | ||
489 | ================= | ||
490 | |||
491 | .. ansi:: images/architecture.ans | ||
492 | |||
493 | Nodepool | ||
494 | ======== | ||
495 | |||
496 | .. container:: handout | ||
497 | |||
498 | nodepool builds nodes for zuul | ||
499 | Remember that 2,000 jobs per hour number? | ||
500 | Each job gets a fresh VM - that's 2,000 VMs per hours | ||
501 | Treats our 12 regions across 6 clouds as one REALLY big cloud | ||
502 | |||
503 | :: | ||
504 | |||
505 | * A separate program that works very closely with *zuul* | ||
506 | * Builds images daily and uploads to clouds | ||
507 | * Creates and destroys zero or more VMs for every job | ||
508 | * Supports using pre-existing nodes (static provider) | ||
509 | |||
510 | Nodepool can use pre-existing images, BUT ... | ||
511 | ============================================= | ||
512 | |||
513 | * Clouds have 'helpful' differences between base images | ||
514 | * Cloud images have 'helpful' software pre-installed | ||
515 | * Distros have 'helpful' different user names | ||
516 | * Most clouds use DHCP for networking, but some don't | ||
517 | * We can add pre-cached content | ||
518 | |||
519 | Cross-Project Example Problem | ||
520 | ============================= | ||
521 | |||
522 | * User reports bug in shade - auto_ip is not discovering their NAT properly | ||
523 | * Two fixes, one to detection algorithm, one to config override | ||
524 | * Config override requires adding support to os-client-config | ||
525 | * Once support is added to os-client-config, it can be consumed in shade | ||
526 | * How do we integration test this without releasing os-client-config? | ||
527 | |||
528 | Cross-Project Dependencies | ||
529 | ========================== | ||
530 | |||
531 | Testing or gating dependencies (including jobs) manually specified by | ||
532 | developers | ||
533 | |||
534 | .. container:: progressive | ||
535 | |||
536 | * shade https://review.openstack.org/513913/ | ||
537 | |||
538 | Add unittest tips jobs | ||
539 | |||
540 | Change-ID: I5b411be5c5aa43535fa89a51d6099aadd7a8ea60 | ||
541 | * os-client-config https://review.openstack.org/513915 | ||
542 | |||
543 | Add shade-tox-tips jobs | ||
544 | |||
545 | Change-ID: Ie3e9a4deca1d74b94e810e87e130706fe15fe2c9 | ||
546 | |||
547 | Depends-On: https://review.openstack.org/513913/ | ||
548 | * os-client-config https://review.openstack.org/513751/ | ||
549 | |||
550 | Added nat_source flag for networks | ||
551 | |||
552 | Change-ID: I3d8dd6d734a1013d2d4a43e11c3538c3a345820b | ||
553 | |||
554 | * shade https://review.openstack.org/#/c/513914 | ||
555 | |||
556 | Add support for configured NAT source variable | ||
557 | |||
558 | Change-Id: I4b50c2323a487b5ce90f9d38a48be249cfb739c5 | ||
559 | |||
560 | Depends-On: https://review.openstack.org/513914 | ||
561 | |||
562 | shade: Add unittest tips jobs | ||
563 | ============================= | ||
564 | |||
565 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
566 | |||
567 | .. code:: yaml | ||
568 | |||
569 | - job: | ||
570 | name: shade-tox-py27-tips | ||
571 | parent: openstack-tox-py27 | ||
572 | description: | | ||
573 | Run tox python 27 unittests against master of important libs | ||
574 | required-projects: | ||
575 | - openstack-infra/shade | ||
576 | - openstack/keystoneauth | ||
577 | - openstack/os-client-config | ||
578 | |||
579 | - job: | ||
580 | name: shade-tox-py35-tips | ||
581 | parent: openstack-tox-py35 | ||
582 | description: | | ||
583 | Run tox python 35 unittests against master of important libs | ||
584 | required-projects: | ||
585 | - openstack-infra/shade | ||
586 | - openstack/keystoneauth | ||
587 | - openstack/os-client-config | ||
588 | |||
589 | shade: Add unittest tips project-template | ||
590 | ========================================= | ||
591 | |||
592 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
593 | |||
594 | .. code:: yaml | ||
595 | |||
596 | - project-template: | ||
597 | name: shade-tox-tips | ||
598 | check: | ||
599 | jobs: | ||
600 | - shade-tox-py27-tips | ||
601 | - shade-tox-py35-tips | ||
602 | gate: | ||
603 | jobs: | ||
604 | - shade-tox-py27-tips | ||
605 | - shade-tox-py35-tips | ||
606 | |||
607 | shade: Add unittest tips project-template to project | ||
608 | ==================================================== | ||
609 | |||
610 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
611 | |||
612 | .. code:: yaml | ||
613 | |||
614 | - project: | ||
615 | templates: | ||
616 | - publish-to-pypi | ||
617 | - publish-openstack-sphinx-docs | ||
618 | - shade-tox-tips | ||
619 | |||
620 | os-client-config: Add shade-tox-tips jobs | ||
621 | ========================================= | ||
622 | |||
623 | * In git.openstack.org/openstack/os-client-config/.zuul.yaml: | ||
624 | |||
625 | .. code:: yaml | ||
626 | |||
627 | - project: | ||
628 | templates: | ||
629 | - shade-tox-tips | ||
630 | check: | ||
631 | jobs: | ||
632 | - legacy-osc-dsvm-functional-tips: | ||
633 | voting: false | ||
634 | |||
635 | os-client-config: Add nat_source flag for networks | ||
636 | ================================================== | ||
637 | |||
638 | :: | ||
639 | |||
640 | diff --git a/os_client_config/cloud_config.py b/os_client_config/cloud_config.py | ||
641 | index 2e97629..d1a6983 100644 | ||
642 | --- a/os_client_config/cloud_config.py | ||
643 | +++ b/os_client_config/cloud_config.py | ||
644 | @@ -581,3 +581,10 @@ class CloudConfig(object): | ||
645 | if net['nat_destination']: | ||
646 | return net['name'] | ||
647 | return None | ||
648 | + | ||
649 | + def get_nat_source(self): | ||
650 | + """Get network used for NAT source.""" | ||
651 | + for net in self.config['networks']: | ||
652 | + if net.get('nat_source'): | ||
653 | + return net['name'] | ||
654 | + return None | ||
655 | |||
656 | shade: Add support for configured NAT source variable | ||
657 | ===================================================== | ||
658 | |||
659 | :: | ||
660 | |||
661 | Zuul 10-21 13:57 | ||
662 | Patch Set 5: Verified-1 | ||
663 | Build failed. | ||
664 | openstack-tox-pep8 SUCCESS in 2m 29s | ||
665 | openstack-tox-py27 FAILURE in 2m 34s | ||
666 | build-openstack-releasenotes SUCCESS in 2m 47s | ||
667 | openstack-tox-py35 FAILURE in 2m 41s | ||
668 | openstack-tox-cover POST_FAILURE in 3m 52s (non-voting) | ||
669 | build-openstack-sphinx-docs SUCCESS in 2m 57s | ||
670 | shade-tox-py27-tips SUCCESS in 3m 18s | ||
671 | shade-tox-py35-tips SUCCESS in 2m 28s | ||
672 | |||
673 | Live Configuration Changes | ||
674 | ========================== | ||
675 | |||
676 | .. container:: handout | ||
677 | |||
678 | Zuul is a distributed system, with a distributed configuration. | ||
679 | |||
680 | .. code:: yaml | ||
681 | |||
682 | - tenant: | ||
683 | name: openstack | ||
684 | source: | ||
685 | gerrit: | ||
686 | config-projects: | ||
687 | - project-config | ||
688 | untrusted-projects: | ||
689 | - openstack-infra/zuul-jobs: | ||
690 | shadow: openstack-infra/project-config | ||
691 | - openstack-infra/openstack-zuul-jobs | ||
692 | - openstack-infra/nodepool | ||
693 | - openstack-infra/shade | ||
694 | - openstack-infra/zuul | ||
695 | - openstack/requirements | ||
696 | |||
697 | Zuul Startup | ||
698 | ============ | ||
699 | |||
700 | * Read config file | ||
701 | |||
702 | Zuul Startup | ||
703 | ============ | ||
704 | |||
705 | * Read config file | ||
706 | * Ask mergers for branches of each repo | ||
707 | |||
708 | .. ansi:: images/startup1.ans | ||
709 | |||
710 | Zuul Startup | ||
711 | ============ | ||
712 | |||
713 | * Read config file | ||
714 | * Ask mergers for branches of each repo | ||
715 | * Ask mergers for .zuul.yaml file for each branch of each repo | ||
716 | |||
717 | ``.zuul.yaml`` can be ``^\.?zuul.yaml$`` file or ``^\.?zuul.d$`` run-parts | ||
718 | directory. | ||
719 | |||
720 | .. ansi:: images/startup2.ans | ||
721 | |||
722 | When .zuul.yaml Changes | ||
723 | ======================= | ||
724 | |||
725 | .. container:: progressive | ||
726 | |||
727 | * Zuul looks for changes to .zuul.yaml | ||
728 | * Asks mergers for updated content | ||
729 | * Splices into configuration used for that change | ||
730 | * Works with cross-repo dependencies | ||
731 | |||
732 | ("This change depends on a change to the job definition") | ||
733 | |||
734 | OpenStack Github Support for Cross Community Testing | ||
735 | ==================================================== | ||
736 | |||
737 | * OpenStack does not use Github, but other people do | ||
738 | * Github App "OpenStack Zuul" | ||
739 | * App added to github project by project admin | ||
740 | * Project aded to OpenStack's main.yaml | ||
741 | * Test interactions between OpenStack and important adjacent communities | ||
742 | * https://github.com/ansible/ansible/pull/20974 | ||
743 | |||
744 | OpenLab for ecosystem testing | ||
745 | ============================= | ||
746 | |||
747 | * Hey Melvin | ||
748 | * Separate from but friendly with Infra | ||
749 | * Joint effort by Huawei and Intel | ||
750 | * Provides resources and support to test things that consume OpenStack | ||
751 | * gophercloud/terraform, ansible/openstack | ||
752 | |||
753 | Cross Source Dependencies | ||
754 | ========================= | ||
755 | |||
756 | .. container:: progressive | ||
757 | |||
758 | * shade https://review.openstack.org/539563 | ||
759 | |||
760 | Shift voting flag and test_matrix_branch for ansible-devel job | ||
761 | |||
762 | Change-ID: Ic9d3983de641dbe618c65b2cbf2dcfa3686575df | ||
763 | |||
764 | * ansible https://github.com/ansible/ansible/pull/34925 | ||
765 | |||
766 | continue fact gathering even without dmidecode | ||
767 | |||
768 | * ansible https://github.com/ansible/ansible/pull/20974 | ||
769 | |||
770 | Make a generalized OpenStack cloud constructor | ||
771 | |||
772 | Depends-On: https://review.openstack.org/539563 | ||
773 | Depends-On: https://github.com/ansible/ansible/pull/34925 | ||
774 | |||
775 | Jobs | ||
776 | ==== | ||
777 | |||
778 | * Jobs run on nodes from nodepool (static or dynamic) | ||
779 | * Metadata defined in Zuul's configuration | ||
780 | * Execution content in Ansible (with live streaming!) | ||
781 | * Jobs may be defined centrally or in the repo being tested | ||
782 | * Jobs have contextual variants that simplify configuration | ||
783 | |||
784 | Shared Job Configs | ||
785 | ================== | ||
786 | |||
787 | * Job config repos are all in git | ||
788 | * Designed to support directly sharing job configurations | ||
789 | * git.openstack.org/openstack-infra/zuul-jobs repo is a 'standard library' | ||
790 | to be directly shared between zuul installations | ||
791 | |||
792 | Job | ||
793 | === | ||
794 | |||
795 | .. code:: yaml | ||
796 | |||
797 | - job: | ||
798 | name: base | ||
799 | parent: null | ||
800 | description: | | ||
801 | The base job for Zuul. | ||
802 | timeout: 1800 | ||
803 | nodeset: | ||
804 | nodes: | ||
805 | - name: primary | ||
806 | label: centos-7 | ||
807 | pre-run: playbooks/base/pre.yaml | ||
808 | post-run: | ||
809 | - playbooks/base/post-ssh.yaml | ||
810 | - playbooks/base/post-logs.yaml | ||
811 | secrets: | ||
812 | - site_logs | ||
813 | |||
814 | Simple Job | ||
815 | ========== | ||
816 | |||
817 | .. code:: yaml | ||
818 | |||
819 | - job: | ||
820 | name: tox | ||
821 | pre-run: playbooks/setup-tox.yaml | ||
822 | run: playbooks/tox.yaml | ||
823 | post-run: playbooks/fetch-tox-output.yaml | ||
824 | |||
825 | - job: | ||
826 | name: tox-py27 | ||
827 | parent: tox | ||
828 | vars: | ||
829 | tox_envlist: py27 | ||
830 | |||
831 | |||
832 | Simple Job Variant | ||
833 | ================== | ||
834 | |||
835 | .. code:: yaml | ||
836 | |||
837 | - job: | ||
838 | name: tox-py27 | ||
839 | branches: stable/mitaka | ||
840 | nodeset: | ||
841 | - name: ubuntu-trusty | ||
842 | label: ubuntu-trusty | ||
843 | |||
844 | Nodesets for Multi-node Jobs | ||
845 | ============================ | ||
846 | |||
847 | .. code:: yaml | ||
848 | |||
849 | - nodeset: | ||
850 | name: ceph-cluster | ||
851 | nodes: | ||
852 | - name: controller | ||
853 | label: centos-7 | ||
854 | - name: compute1 | ||
855 | label: fedora-26 | ||
856 | - name: compute2 | ||
857 | label: fedora-26 | ||
858 | groups: | ||
859 | - name: ceph-osd | ||
860 | nodes: | ||
861 | - controller | ||
862 | - name: ceph-monitor | ||
863 | nodes: | ||
864 | - controller | ||
865 | - compute1 | ||
866 | - compute2 | ||
867 | |||
868 | Multi-node Job | ||
869 | ============== | ||
870 | |||
871 | * nodesets are provided to Ansible for jobs in inventory | ||
872 | |||
873 | .. code:: yaml | ||
874 | |||
875 | - job: | ||
876 | name: ceph-multinode | ||
877 | nodeset: ceph-cluster | ||
878 | run: playbooks/install-ceph.yaml | ||
879 | |||
880 | Multi-node Ceph Job Content | ||
881 | =========================== | ||
882 | |||
883 | .. code:: yaml | ||
884 | |||
885 | - hosts: all | ||
886 | roles: | ||
887 | - install-ceph | ||
888 | - hosts: ceph-osd | ||
889 | roles: | ||
890 | - start-ceph-osd | ||
891 | - hosts: ceph-monitor | ||
892 | roles: | ||
893 | - start-ceph-monitor | ||
894 | - hosts: all | ||
895 | roles: | ||
896 | - do-something-interesting | ||
897 | |||
898 | Projects | ||
899 | ======== | ||
900 | |||
901 | * Projects are git repositories | ||
902 | * Specify a set of jobs for each pipeline | ||
903 | * golang git repo naming as been adopted: | ||
904 | |||
905 | :: | ||
906 | |||
907 | zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d | ||
908 | /home/zuul/src/git.openstack.org/openstack-infra/shade | ||
909 | /home/zuul/src/git.openstack.org/openstack/keystoneauth | ||
910 | /home/zuul/src/git.openstack.org/openstack/os-client-config | ||
911 | /home/zuul/src/github.com/ansible/ansible | ||
912 | |||
913 | Project | ||
914 | ======= | ||
915 | |||
916 | .. code:: yaml | ||
917 | |||
918 | - project: | ||
919 | check: | ||
920 | jobs: | ||
921 | - openstack-tox-py27 | ||
922 | - openstack-tox-py35 | ||
923 | - openstack-doc-build | ||
924 | |||
925 | Project with Local Variant | ||
926 | ========================== | ||
927 | |||
928 | .. code:: yaml | ||
929 | |||
930 | - project: | ||
931 | check: | ||
932 | jobs: | ||
933 | - openstack-tox-py27 | ||
934 | - openstack-tox-py35 | ||
935 | - openstack-doc-build | ||
936 | - openstack-tox-pypy: | ||
937 | voting: false | ||
938 | |||
939 | Project with More Local Variants | ||
940 | ================================ | ||
941 | |||
942 | .. code:: yaml | ||
943 | |||
944 | - project: | ||
945 | name: openstack/nova | ||
946 | check: | ||
947 | jobs: | ||
948 | - openstack-tox-py27 | ||
949 | - openstack-tox-py35 | ||
950 | - openstack-doc-build: | ||
951 | files: '^docs/.*$' | ||
952 | - openstack-tox-pypy: | ||
953 | voting: false | ||
954 | |||
955 | Project with Many Local Variants | ||
956 | ================================ | ||
957 | |||
958 | .. code:: yaml | ||
959 | |||
960 | - project: | ||
961 | name: openstack/nova | ||
962 | check: | ||
963 | jobs: | ||
964 | - openstack-tox-py27 | ||
965 | nodeset: | ||
966 | - name: centos-7 | ||
967 | label: centos-7 | ||
968 | - openstack-tox-py27 | ||
969 | branches: stable/newton | ||
970 | nodeset: | ||
971 | - name: ubuntu-trusty | ||
972 | label: ubuntu-trusty | ||
973 | - openstack-doc-build: | ||
974 | files: '^docs/.*$' | ||
975 | - openstack-tox-pypy: | ||
976 | voting: false | ||
977 | |||
978 | Project With Central and Local Config | ||
979 | ===================================== | ||
980 | |||
981 | .. code:: yaml | ||
982 | |||
983 | # In git.openstack.org/openstack-infra/project-config: | ||
984 | - project: | ||
985 | name: openstack/nova | ||
986 | templates: | ||
987 | - openstack-tox-jobs | ||
988 | |||
989 | .. code:: yaml | ||
990 | |||
991 | # In git.openstack.org/openstack/nova/.zuul.yaml: | ||
992 | - project: | ||
993 | check: | ||
994 | - nova-placement-functional-devstack | ||
995 | |||
996 | Project with Job Dependencies | ||
997 | ============================= | ||
998 | |||
999 | .. code:: yaml | ||
1000 | |||
1001 | # In git.openstack.org/openstack-infra/project-config: | ||
1002 | - project: | ||
1003 | name: openstack/nova | ||
1004 | release: | ||
1005 | jobs: | ||
1006 | - build-artifacts | ||
1007 | - upload-tarball: | ||
1008 | dependencies: build-artifacts | ||
1009 | - upload-pypi: | ||
1010 | dependencies: build-artifacts | ||
1011 | - notify-mirror: | ||
1012 | dependencies: | ||
1013 | - upload-tarball | ||
1014 | - upload-pypi | ||
1015 | |||
1016 | Playbooks | ||
1017 | ========= | ||
1018 | |||
1019 | * Jobs run Ansible playbooks | ||
1020 | * Playbooks may be defined centrally or in the repo being tested | ||
1021 | * Playbooks can use roles from current or other Zuul repos | ||
1022 | (or Galaxy, coming soon) | ||
1023 | * Playbooks are run on the zuul-executor using bubblewrap | ||
1024 | https://github.com/projectatomic/bubblewrap | ||
1025 | * Playbooks are not allowed to execute content on 'localhost' | ||
1026 | |||
1027 | Job with Roles | ||
1028 | ============== | ||
1029 | |||
1030 | .. code:: yaml | ||
1031 | |||
1032 | - job: | ||
1033 | name: zuul-integration | ||
1034 | description: | | ||
1035 | Multi-node Zuul installation and integration test | ||
1036 | nodeset: zuul-cluster | ||
1037 | roles: | ||
1038 | - zuul: openstack-infra/ansible-role-zuul | ||
1039 | run: playbooks/zuul-integration | ||
1040 | |||
1041 | Job with Multiple Projects | ||
1042 | ========================== | ||
1043 | |||
1044 | .. code:: yaml | ||
1045 | |||
1046 | - job: | ||
1047 | name: tox-py35-on-zuul | ||
1048 | parent: tox-py35 | ||
1049 | description: | | ||
1050 | Run zuul's py35 unittests on patches to zuul-jobs | ||
1051 | vars: | ||
1052 | zuul_work_dir: src/git.openstack.org/openstack-infra/zuul | ||
1053 | required-projects: | ||
1054 | - openstack-infra/zuul | ||
1055 | |||
1056 | - project: | ||
1057 | name: openstack-infra/zuul-jobs | ||
1058 | check: | ||
1059 | jobs: | ||
1060 | - tox-py35-on-zuul | ||
1061 | |||
1062 | Devstack-gate / Tempest Playbook | ||
1063 | ================================ | ||
1064 | |||
1065 | .. code:: yaml | ||
1066 | |||
1067 | # devstack-gate / tempest playbook | ||
1068 | hosts: all | ||
1069 | roles: | ||
1070 | - setup-multinode-networking | ||
1071 | - partition-swap | ||
1072 | - configure-mirrors | ||
1073 | - run-devstack | ||
1074 | - run-tempest | ||
1075 | |||
1076 | Simple Shell Playbook | ||
1077 | ===================== | ||
1078 | |||
1079 | .. code:: yaml | ||
1080 | |||
1081 | hosts: controller | ||
1082 | tasks: | ||
1083 | - shell: ./run_tests.sh | ||
1084 | |||
1085 | Test Like Production | ||
1086 | ==================== | ||
1087 | |||
1088 | If you use Ansible for deployment, your test and deployment processes | ||
1089 | and playbooks are the same | ||
1090 | |||
1091 | What if you don't use Ansible? | ||
1092 | ============================== | ||
1093 | |||
1094 | OpenStack Infra Control Plane uses Puppet | ||
1095 | ========================================= | ||
1096 | |||
1097 | .. code:: yaml | ||
1098 | |||
1099 | # In git.openstack.org/openstack-infra/project-config/roles/legacy-install-afs-with-puppet/tasks/main.yaml | ||
1100 | - name: Install puppet | ||
1101 | shell: ./install_puppet.sh | ||
1102 | args: | ||
1103 | chdir: "{{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/system-config" | ||
1104 | environment: | ||
1105 | # Skip setting up pip, our images have already done this. | ||
1106 | SETUP_PIP: "false" | ||
1107 | become: yes | ||
1108 | |||
1109 | - name: Copy manifest | ||
1110 | copy: | ||
1111 | src: manifest.pp | ||
1112 | dest: "{{ ansible_user_dir }}/manifest.pp" | ||
1113 | |||
1114 | - name: Run puppet | ||
1115 | puppet: | ||
1116 | manifest: "{{ ansible_user_dir }}/manifest.pp" | ||
1117 | become: yes | ||
1118 | |||
1119 | Secrets | ||
1120 | ======= | ||
1121 | |||
1122 | * Inspired by Kubernetes Secrets API | ||
1123 | * Projects can add named encrypted secrets to their .zuul.yaml file | ||
1124 | * Jobs can request to use secrets by name | ||
1125 | * Jobs using secrets are not reconfigured speculatively | ||
1126 | * Secrets can only be used by the same project they are defined in | ||
1127 | * Public key per project: | ||
1128 | ``{{ zuul_url }}/{{ tenant }}/{{ project }}.pub`` | ||
1129 | |||
1130 | :: | ||
1131 | GET http://zuul.openstack.org/openstack-infra/shade.pub | ||
1132 | |||
1133 | Secret Example (note, no admins had to enable this) | ||
1134 | =================================================== | ||
1135 | |||
1136 | .. code:: yaml | ||
1137 | |||
1138 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
1139 | - secret: | ||
1140 | name: loci_docker_login | ||
1141 | data: | ||
1142 | user: !encrypted/pkcs1-oaep | ||
1143 | - r8Nbpq5olmfLF035BZ/CUoFLIdhvBi/49KuochOAHbvns+xMiho3C7MEFzYDqJX3IhHde | ||
1144 | BICYOgK7qnyINOIZL2e7pl75rEdHQwJjSFUMkpdY6wEP7f9hpolj9xVp0ifHUVQqPHMRn | ||
1145 | zoPFd8MEAHxH5GLmc2SWJ98E/QUqGltxBi1YRSZoCcNtq3tHFK5Y+xQlLhIseJ2HkpDs6 | ||
1146 | YXOGP9Qt4Va6sdyBcA90H+apSAcYA3Duu962ySZQAsYNui/3NQq3gLA+OZeyTJtcrh4hj | ||
1147 | Rb5dBnDWfSrMpxdNkbPXXgbQaxO3T0L4jbaOF8VKEsiI9olBrOeV2M9ddYJjSsHGj4XR8 | ||
1148 | 4vwS0+doB7np93fujiDuHVgdG8R40NW2GznyKRlRtzAORla7Mzw1Y1MokcUyY6p1LlLLl | ||
1149 | wUuWYCCEuRciOPhZXQ2u42qju/zrK2/dPnO8HfUINSrN0WbNq14ZwPpbj0ro02oGPbtwu | ||
1150 | OTw1z+N0Nc+GuLWlwYJGYM/z0UnvDR3WEBc2kXbVev9w4n0cB3RyphML2PDZZWbw8tjnX | ||
1151 | h1VsAOJ0Qo4qq1K/ft95ypd+vtjkfepEgHEBmJNwutJa9IHAkGfrkO9VkpUTPpfffnPwz | ||
1152 | d0/zaaadNl6MLQUSutRwY23YIIbv+fmukxw2vnJmvn6abkBlMya7KgtifwNA8c= | ||
1153 | password: !encrypted/pkcs1-oaep | ||
1154 | - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw | ||
1155 | hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE | ||
1156 | 5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE | ||
1157 | 5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo | ||
1158 | 4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p | ||
1159 | taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA | ||
1160 | hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T | ||
1161 | IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq | ||
1162 | 7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj | ||
1163 | /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM= | ||
1164 | |||
1165 | Secret Example | ||
1166 | ============== | ||
1167 | |||
1168 | .. code:: yaml | ||
1169 | |||
1170 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
1171 | - job: | ||
1172 | name: publish-loci-cinder | ||
1173 | parent: loci-cinder | ||
1174 | post-run: playbooks/push | ||
1175 | secrets: | ||
1176 | - loci_docker_login | ||
1177 | |||
1178 | # In git.openstack.org/openstack/loci/playbooks/push.yaml: | ||
1179 | - hosts: all | ||
1180 | tasks: | ||
1181 | - include_vars: vars.yaml | ||
1182 | |||
1183 | - name: Push project to DockerHub | ||
1184 | block: | ||
1185 | - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }} | ||
1186 | no_log: True | ||
1187 | - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} | ||
1188 | with_items: "{{ distros }}" | ||
1189 | |||
1190 | Status | ||
1191 | ====== | ||
1192 | |||
1193 | * Zuul v3 is in production for OpenStack (in OpenStack VMs) | ||
1194 | * Zuul v3 also runing at BMW (in OpenShift) and Godaddy (in kuberenetes) | ||
1195 | and Huawei's OpenLab. | ||
1196 | * Software Factory updated to v3 | ||
1197 | https://softwarefactory-project.io/sf/welcome.html | ||
1198 | * will tag and release v3.0 once we're satisfied it's good for other people | ||
1199 | (within the next few weeks) | ||
1200 | |||
1201 | What's Next? | ||
1202 | ============ | ||
1203 | |||
1204 | * shared job doc generation | ||
1205 | * node providers | ||
1206 | * kuberenetes | ||
1207 | * OCI/docker | ||
1208 | * Mac Stadium (for our Ansible friends) | ||
1209 | * ec2 | ||
1210 | * native container/kubernetes job execution | ||
1211 | |||
1212 | Important Links | ||
1213 | =============== | ||
1214 | |||
1215 | * https://zuul-ci.org/ | ||
1216 | * https://git.openstack.org/cgit/openstack-infra/zuul | ||
1217 | * https://docs.openstack.org/infra/zuul | ||
1218 | * https://docs.openstack.org/infra/manual/zuulv3.html | ||
1219 | * https://docs.openstack.org/infra/zuul-jobs/ | ||
1220 | * https://docs.openstack.org/infra/openstack-zuul-jobs/ | ||
1221 | * https://storyboard.openstack.org/#!/project/679 | ||
1222 | * https://storyboard.openstack.org/#!/board/41 | ||
1223 | * freenode:#zuul | ||
1224 | |||
1225 | Questions | ||
1226 | ========= | ||
1227 | |||
1228 | .. ansi:: images/questions.ans | ||
1229 | |||
1230 | Presentty | ||
1231 | ========= | ||
1232 | .. hidetitle:: | ||
1233 | .. transition:: pan | ||
1234 | .. figlet:: Presentty | ||
1235 | |||
1236 | * Console presentations written in reStructuredText | ||
1237 | * Cross-fade, pan, tilt, cut transitions | ||
1238 | * Figlet, cowsay! | ||
1239 | * https://pypi.python.org/pypi/presentty | ||