diff options
Diffstat (limited to 'src/zuulv3/zuul.rst')
-rw-r--r-- | src/zuulv3/zuul.rst | 1326 |
1 files changed, 1326 insertions, 0 deletions
diff --git a/src/zuulv3/zuul.rst b/src/zuulv3/zuul.rst new file mode 100644 index 0000000..2070ad6 --- /dev/null +++ b/src/zuulv3/zuul.rst | |||
@@ -0,0 +1,1326 @@ | |||
1 | . display in 68x24 | ||
2 | .. display in 88x24 | ||
3 | |||
4 | .. pygments yaml? (only file breaks (---) tinted) | ||
5 | .. slide on high level v3 changes | ||
6 | .. slide on nodepool | ||
7 | |||
8 | .. transition:: dissolve | ||
9 | :duration: 0.4 | ||
10 | |||
11 | Test Slide | ||
12 | ========== | ||
13 | .. hidetitle:: | ||
14 | |||
15 | .. ansi:: images/testslide.ans | ||
16 | |||
17 | Preshow | ||
18 | ======= | ||
19 | .. hidetitle:: | ||
20 | |||
21 | .. ansi:: images/cursor.ans images/cursor2.ans | ||
22 | |||
23 | Zuul | ||
24 | ==== | ||
25 | .. hidetitle:: | ||
26 | .. ansi:: images/zuul.ans | ||
27 | |||
28 | Red Hat | ||
29 | ======= | ||
30 | |||
31 | .. hidetitle:: | ||
32 | .. container:: handout | ||
33 | |||
34 | * I work for Red Hat in the Office of Technology as the Chief Architect | ||
35 | for CI/CD | ||
36 | |||
37 | .. ansi:: images/redhat.ans | ||
38 | |||
39 | OpenStack | ||
40 | ========= | ||
41 | .. container:: handout | ||
42 | |||
43 | * I work on OpenStack. | ||
44 | * I sit on the Technical Committee. I was on the Board of Directors | ||
45 | |||
46 | .. hidetitle:: | ||
47 | .. ansi:: images/openstack.ans | ||
48 | |||
49 | OpenStack Infra | ||
50 | =============== | ||
51 | .. container:: handout | ||
52 | |||
53 | * My primary technical role with OpenStack is working on the OpenStack CI | ||
54 | system. | ||
55 | |||
56 | :: | ||
57 | |||
58 | "most insane CI infrastructure I've ever been a part of" | ||
59 | |||
60 | -- Alex Gaynor | ||
61 | |||
62 | Zuul | ||
63 | ==== | ||
64 | .. container:: handout | ||
65 | |||
66 | * As part of working on OpenStack Infra I work on Zuul | ||
67 | |||
68 | .. hidetitle:: | ||
69 | .. ansi:: images/zuul.ans | ||
70 | |||
71 | |||
72 | Ansible | ||
73 | ======= | ||
74 | |||
75 | .. container:: handout | ||
76 | |||
77 | * And as part of working on Zuul and on OpenStack I work on Ansible | ||
78 | * I maintain the OpenStack modules for Ansible as well as the shade library | ||
79 | |||
80 | .. hidetitle:: | ||
81 | .. ansi:: images/ansible.ans | ||
82 | |||
83 | Presentation Checklist | ||
84 | ====================== | ||
85 | |||
86 | .. container:: handout | ||
87 | |||
88 | * Every good presentation needs logos, so we're starting well | ||
89 | |||
90 | :: | ||
91 | |||
92 | [X] Logos | ||
93 | |||
94 | |||
95 | Spoilers | ||
96 | ======== | ||
97 | |||
98 | * What the old version of Zuul (v2) was | ||
99 | |||
100 | * a nifty project gating system | ||
101 | |||
102 | * What the new version of Zuul (v3) is | ||
103 | |||
104 | * multinode support | ||
105 | * live configuration changes | ||
106 | * better job definition | ||
107 | * sharable job definition | ||
108 | * testing like deployment | ||
109 | |||
110 | What do I mean by Massive Scale? | ||
111 | ================================ | ||
112 | |||
113 | * Contributors (~2k / 6 month period) | ||
114 | * Companies | ||
115 | * Changes | ||
116 | * Code Repositories (1827 as of this morning) | ||
117 | * Communities | ||
118 | |||
119 | OpenStack Scale | ||
120 | =============== | ||
121 | |||
122 | * 2,000 git repositories | ||
123 | * 2KJPH (2,000 jobs per hour) | ||
124 | * Nodes 14 Regions off 5 OpenStack Public Clouds and 2 Private Clouds | ||
125 | (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud, HPE, Red Hat) | ||
126 | * 10,000 changes merged per month | ||
127 | |||
128 | OpenStack Scale | ||
129 | =============== | ||
130 | |||
131 | * 2,000 git repositories | ||
132 | * 2KJPH (2,000 jobs per hour) | ||
133 | * Nodes 14 Regions off 5 OpenStack Public Clouds and 2 Private Clouds | ||
134 | (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud, HPE, Red Hat) | ||
135 | * 10,000 changes merged per month | ||
136 | |||
137 | * By comparison, our friends at the amazing project Ansible received | ||
138 | 13,000 changes and had merged 8,000 of them in its first 4 years. | ||
139 | |||
140 | Pretty Things to Look for Scale | ||
141 | =============================== | ||
142 | |||
143 | * http://grafana.openstack.org/dashboard/db/zuul-status | ||
144 | * http://grafana.openstack.org/dashboard/db/nodepool | ||
145 | * http://zuulv3.openstack.org/ | ||
146 | |||
147 | Dealing With Scale | ||
148 | ================== | ||
149 | |||
150 | * Egalitarian Process | ||
151 | * Balance Centralized vs Distributed | ||
152 | * Code Review plus Enforced Testing | ||
153 | |||
154 | OpenStack Developer Workflow | ||
155 | ============================ | ||
156 | .. container:: handout | ||
157 | |||
158 | :: | ||
159 | |||
160 | Hack Review Test | ||
161 | ========= ========== ========== | ||
162 | |||
163 | push approve | ||
164 | +-------------+ +-------------+ | ||
165 | | | | | | ||
166 | +------+--+ +--v----+--+ +--v-------+ | ||
167 | | | | | | | | ||
168 | | $EDITOR | | Gerrit | | Zuul | | ||
169 | | | | | | | | ||
170 | +------^--+ +--+----^--+ +--+-------+ | ||
171 | | | | | | ||
172 | +-------------+ +-------------+ | ||
173 | clone merge | ||
174 | |||
175 | Gerrit | ||
176 | ====== | ||
177 | .. hidetitle:: | ||
178 | .. container:: handout | ||
179 | |||
180 | the primary interface for our developers is the code review system | ||
181 | gerrit. No matter how complex zuul becomes, this is still primary | ||
182 | focus we want the developers to have. | ||
183 | |||
184 | explain patch upload, zuul runs, test results displayed in gerrit | ||
185 | this is all the interface to zuul users need to see | ||
186 | |||
187 | but zuul is doing a lot of work behind the scenes, and if you look | ||
188 | closer, this is what you see | ||
189 | |||
190 | .. ansi:: images/color-gertty.ans | ||
191 | |||
192 | Github Developer Workflow | ||
193 | ========================= | ||
194 | .. container:: handout | ||
195 | |||
196 | :: | ||
197 | |||
198 | Hack Review Test | ||
199 | ========= ========== ========== | ||
200 | |||
201 | push approve | ||
202 | +-------------+ +-------------+ | ||
203 | | | | | | ||
204 | +------+--+ +--v----+--+ +--v-------+ | ||
205 | | | | | | | | ||
206 | | $EDITOR | | Github | | Zuul | | ||
207 | | | | | | | | ||
208 | +------^--+ +--+----^--+ +--+-------+ | ||
209 | | | | | | ||
210 | +-------------+ +-------------+ | ||
211 | clone merge | ||
212 | |||
213 | Zuul Architecture | ||
214 | ================= | ||
215 | |||
216 | .. ansi:: images/architecture.ans | ||
217 | |||
218 | Presentation Checklist | ||
219 | ====================== | ||
220 | |||
221 | :: | ||
222 | |||
223 | [x] Logos | ||
224 | [x] Architecture diagram | ||
225 | |||
226 | Nodepool | ||
227 | ======== | ||
228 | |||
229 | .. container:: handout | ||
230 | |||
231 | nodepool builds nodes for zuul | ||
232 | Remember that 2,000 jobs per hour number? | ||
233 | Each job gets a fresh VM - that's 2,000 VMs per hours | ||
234 | Treats our 20 regions across 9 clouds as one REALLY big cloud | ||
235 | |||
236 | :: | ||
237 | |||
238 | * A separate program that works very closely with *zuul* | ||
239 | * Builds images daily and uploads to clouds | ||
240 | * Creates and destroys (at least) a VM for every job | ||
241 | |||
242 | Nodepool can use pre-existing images, BUT ... | ||
243 | ============================================= | ||
244 | |||
245 | * Clouds have 'helpful' differences between base images | ||
246 | * Cloud images have 'helpful' software pre-installed | ||
247 | * Distros have 'helpful' different user names | ||
248 | * Most clouds use DHCP for networking, but some don't | ||
249 | * We can add pre-cached content | ||
250 | |||
251 | Gating | ||
252 | ====== | ||
253 | |||
254 | .. cowsay:: Every change proposed for a repository is tested before | ||
255 | it merges. | ||
256 | |||
257 | Co-gating | ||
258 | ========= | ||
259 | |||
260 | .. cowsay:: Changes to a set of repositories merge monotonically such | ||
261 | that each change is tested with the current state of all | ||
262 | other related repositories before it merges. | ||
263 | |||
264 | Parallel Co-gating | ||
265 | ================== | ||
266 | |||
267 | .. cowsay:: Changes are serialized such that each change is tested | ||
268 | with all of the changes ahead of it to satisfy the | ||
269 | gating requirement while being able to run tests for | ||
270 | multiple changes simultaneously. | ||
271 | |||
272 | Presentation Checklist | ||
273 | ====================== | ||
274 | |||
275 | :: | ||
276 | |||
277 | [x] Logos | ||
278 | [x] Architecture diagram | ||
279 | [x] Cows | ||
280 | |||
281 | Zuul Simulation | ||
282 | =============== | ||
283 | .. transition:: pan | ||
284 | .. container:: handout | ||
285 | |||
286 | * That was a lot of words - let's walk through it one step at a time | ||
287 | * Here we have two git repos, called nova and keystone, and their | ||
288 | current HEAD state | ||
289 | |||
290 | .. ansi:: images/zsim-00.ans | ||
291 | |||
292 | Zuul Simulation | ||
293 | =============== | ||
294 | .. transition:: cut | ||
295 | .. container:: handout | ||
296 | |||
297 | * A change is approved for Nova | ||
298 | |||
299 | .. ansi:: images/zsim-01.ans | ||
300 | |||
301 | Zuul Simulation | ||
302 | =============== | ||
303 | .. transition:: cut | ||
304 | .. container:: handout | ||
305 | |||
306 | * Zuul starts running jobs for it | ||
307 | * The tests will test the current state of nova and keystone PLUS this nova | ||
308 | change | ||
309 | |||
310 | .. ansi:: images/zsim-02.ans | ||
311 | |||
312 | Zuul Simulation | ||
313 | =============== | ||
314 | .. transition:: cut | ||
315 | .. container:: handout | ||
316 | |||
317 | * A change is approved for Keystone | ||
318 | |||
319 | .. ansi:: images/zsim-03.ans | ||
320 | |||
321 | Zuul Simulation | ||
322 | =============== | ||
323 | .. transition:: cut | ||
324 | .. container:: handout | ||
325 | |||
326 | * The tests will test the current state of nova and keystone PLUS this nova | ||
327 | change | ||
328 | |||
329 | .. ansi:: images/zsim-04.ans | ||
330 | |||
331 | Zuul Simulation | ||
332 | =============== | ||
333 | .. transition:: cut | ||
334 | .. container:: handout | ||
335 | |||
336 | * todo | ||
337 | |||
338 | .. ansi:: images/zsim-05.ans | ||
339 | |||
340 | Zuul Simulation | ||
341 | =============== | ||
342 | .. transition:: cut | ||
343 | .. container:: handout | ||
344 | |||
345 | * todo | ||
346 | |||
347 | .. ansi:: images/zsim-06.ans | ||
348 | |||
349 | Zuul Simulation | ||
350 | =============== | ||
351 | .. transition:: cut | ||
352 | .. container:: handout | ||
353 | |||
354 | * todo | ||
355 | |||
356 | .. ansi:: images/zsim-07.ans | ||
357 | |||
358 | Zuul Simulation | ||
359 | =============== | ||
360 | .. transition:: cut | ||
361 | .. container:: handout | ||
362 | |||
363 | * todo | ||
364 | |||
365 | .. ansi:: images/zsim-08.ans | ||
366 | |||
367 | Zuul Simulation | ||
368 | =============== | ||
369 | .. transition:: cut | ||
370 | .. container:: handout | ||
371 | |||
372 | * todo | ||
373 | |||
374 | .. ansi:: images/zsim-09.ans | ||
375 | |||
376 | Zuul Simulation | ||
377 | =============== | ||
378 | .. transition:: cut | ||
379 | .. container:: handout | ||
380 | |||
381 | * todo | ||
382 | |||
383 | .. ansi:: images/zsim-10.ans | ||
384 | |||
385 | Zuul Simulation | ||
386 | =============== | ||
387 | .. transition:: cut | ||
388 | .. container:: handout | ||
389 | |||
390 | * todo | ||
391 | |||
392 | .. ansi:: images/zsim-11.ans | ||
393 | |||
394 | Zuul Simulation | ||
395 | =============== | ||
396 | .. transition:: cut | ||
397 | .. container:: handout | ||
398 | |||
399 | * todo | ||
400 | |||
401 | .. ansi:: images/zsim-12.ans | ||
402 | |||
403 | Zuul Simulation | ||
404 | =============== | ||
405 | .. transition:: cut | ||
406 | .. container:: handout | ||
407 | |||
408 | * todo | ||
409 | |||
410 | .. ansi:: images/zsim-13.ans | ||
411 | |||
412 | Zuul Simulation | ||
413 | =============== | ||
414 | .. transition:: cut | ||
415 | .. container:: handout | ||
416 | |||
417 | * todo | ||
418 | |||
419 | .. ansi:: images/zsim-14.ans | ||
420 | |||
421 | Zuul Simulation | ||
422 | =============== | ||
423 | .. transition:: cut | ||
424 | .. container:: handout | ||
425 | |||
426 | * todo | ||
427 | |||
428 | .. ansi:: images/zsim-15.ans | ||
429 | |||
430 | Zuul Simulation | ||
431 | =============== | ||
432 | .. transition:: cut | ||
433 | .. container:: handout | ||
434 | |||
435 | * todo | ||
436 | |||
437 | .. ansi:: images/zsim-16.ans | ||
438 | |||
439 | Zuul Simulation | ||
440 | =============== | ||
441 | .. transition:: cut | ||
442 | .. container:: handout | ||
443 | |||
444 | * todo | ||
445 | |||
446 | .. ansi:: images/zsim-17.ans | ||
447 | |||
448 | Zuul Simulation | ||
449 | =============== | ||
450 | .. transition:: cut | ||
451 | .. container:: handout | ||
452 | |||
453 | * todo | ||
454 | |||
455 | .. ansi:: images/zsim-18.ans | ||
456 | |||
457 | Zuul Simulation | ||
458 | =============== | ||
459 | .. transition:: cut | ||
460 | .. container:: handout | ||
461 | |||
462 | * todo | ||
463 | |||
464 | .. ansi:: images/zsim-19.ans | ||
465 | |||
466 | Zuul Simulation | ||
467 | =============== | ||
468 | .. transition:: cut | ||
469 | .. container:: handout | ||
470 | |||
471 | * todo | ||
472 | |||
473 | .. ansi:: images/zsim-20.ans | ||
474 | |||
475 | Zuul Simulation | ||
476 | =============== | ||
477 | .. transition:: cut | ||
478 | .. container:: handout | ||
479 | |||
480 | * todo | ||
481 | |||
482 | .. ansi:: images/zsim-21.ans | ||
483 | |||
484 | Zuul Simulation | ||
485 | =============== | ||
486 | .. transition:: cut | ||
487 | .. container:: handout | ||
488 | |||
489 | * todo | ||
490 | |||
491 | .. ansi:: images/zsim-22.ans | ||
492 | |||
493 | Cross-Project Problem | ||
494 | ===================== | ||
495 | |||
496 | * User reports bug in shade - auto_ip is not discovering their NAT properly | ||
497 | * Two fixes, one to detection algorithm, one to config override | ||
498 | * Config override requires adding support to os-client-config | ||
499 | * Once support is added to os-client-config, it can be consumed in shade | ||
500 | * How do we integration test this without releasing os-client-config? | ||
501 | |||
502 | Cross-Project Dependencies | ||
503 | ========================== | ||
504 | |||
505 | Testing or gating dependencies (including jobs) manually specified by | ||
506 | developers | ||
507 | |||
508 | .. container:: progressive | ||
509 | |||
510 | * shade https://review.openstack.org/#/c/513913/ | ||
511 | |||
512 | Add unittest tips jobs | ||
513 | |||
514 | Change-ID: I5b411be5c5aa43535fa89a51d6099aadd7a8ea60 | ||
515 | * os-client-config https://review.openstack.org/#/c/513915 | ||
516 | |||
517 | Add shade-tox-tips jobs | ||
518 | |||
519 | Change-ID: Ie3e9a4deca1d74b94e810e87e130706fe15fe2c9 | ||
520 | |||
521 | Depends-On: I5b411be5c5aa43535fa89a51d6099aadd7a8ea60 | ||
522 | * os-client-config https://review.openstack.org/#/c/513751/ | ||
523 | |||
524 | Added nat_source flag for networks | ||
525 | |||
526 | Change-ID: I3d8dd6d734a1013d2d4a43e11c3538c3a345820b | ||
527 | |||
528 | * shade https://review.openstack.org/#/c/513914 | ||
529 | |||
530 | Add support for configured NAT source variable | ||
531 | |||
532 | Change-Id: I4b50c2323a487b5ce90f9d38a48be249cfb739c5 | ||
533 | |||
534 | Depends-On: I3d8dd6d734a1013d2d4a43e11c3538c3a345820b | ||
535 | |||
536 | shade: Add unittest tips jobs | ||
537 | ============================= | ||
538 | |||
539 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
540 | |||
541 | .. code:: yaml | ||
542 | |||
543 | - job: | ||
544 | name: shade-tox-py27-tips | ||
545 | parent: openstack-tox-py27 | ||
546 | description: | | ||
547 | Run tox python 27 unittests against master of important libs | ||
548 | required-projects: | ||
549 | - openstack-infra/shade | ||
550 | - openstack/os-client-config | ||
551 | |||
552 | - job: | ||
553 | name: shade-tox-py35-tips | ||
554 | parent: openstack-tox-py35 | ||
555 | description: | | ||
556 | Run tox python 35 unittests against master of important libs | ||
557 | required-projects: | ||
558 | - openstack-infra/shade | ||
559 | - openstack/keystoneauth | ||
560 | - openstack/os-client-config | ||
561 | |||
562 | shade: Add unittest tips project-template | ||
563 | ========================================= | ||
564 | |||
565 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
566 | |||
567 | .. code:: yaml | ||
568 | |||
569 | - project-template: | ||
570 | name: shade-tox-tips | ||
571 | check: | ||
572 | jobs: | ||
573 | - shade-tox-py27-tips | ||
574 | - shade-tox-py35-tips | ||
575 | gate: | ||
576 | jobs: | ||
577 | - shade-tox-py27-tips | ||
578 | - shade-tox-py35-tips | ||
579 | |||
580 | shade: Add unittest tips project-template to project | ||
581 | ==================================================== | ||
582 | |||
583 | * In git.openstack.org/openstack-infra/shade/.zuul.yaml: | ||
584 | |||
585 | .. code:: yaml | ||
586 | |||
587 | - project: | ||
588 | name: openstack-infra/shade | ||
589 | templates: | ||
590 | - publish-to-pypi | ||
591 | - publish-openstack-sphinx-docs | ||
592 | - shade-tox-tips | ||
593 | |||
594 | os-client-config: Add shade-tox-tips jobs | ||
595 | ========================================= | ||
596 | |||
597 | * In git.openstack.org/openstack/os-client-config/.zuul.yaml: | ||
598 | |||
599 | .. code:: yaml | ||
600 | |||
601 | - project: | ||
602 | name: openstack/os-client-config | ||
603 | templates: | ||
604 | - shade-tox-tips | ||
605 | check: | ||
606 | jobs: | ||
607 | - legacy-osc-dsvm-functional-tips: | ||
608 | voting: false | ||
609 | |||
610 | os-client-config: Add nat_source flag for networks | ||
611 | ================================================== | ||
612 | |||
613 | :: | ||
614 | |||
615 | diff --git a/os_client_config/cloud_config.py b/os_client_config/cloud_config.py | ||
616 | index 2e97629..d1a6983 100644 | ||
617 | --- a/os_client_config/cloud_config.py | ||
618 | +++ b/os_client_config/cloud_config.py | ||
619 | @@ -581,3 +581,10 @@ class CloudConfig(object): | ||
620 | if net['nat_destination']: | ||
621 | return net['name'] | ||
622 | return None | ||
623 | + | ||
624 | + def get_nat_source(self): | ||
625 | + """Get network used for NAT source.""" | ||
626 | + for net in self.config['networks']: | ||
627 | + if net.get('nat_source'): | ||
628 | + return net['name'] | ||
629 | + return None | ||
630 | |||
631 | shade: Add support for configured NAT source variable | ||
632 | ===================================================== | ||
633 | |||
634 | :: | ||
635 | |||
636 | Zuul 10-21 13:57 | ||
637 | Patch Set 5: Verified-1 | ||
638 | Build failed. | ||
639 | openstack-tox-pep8 SUCCESS in 2m 29s | ||
640 | openstack-tox-py27 FAILURE in 2m 34s | ||
641 | build-openstack-releasenotes SUCCESS in 2m 47s | ||
642 | openstack-tox-py35 FAILURE in 2m 41s | ||
643 | openstack-tox-cover POST_FAILURE in 3m 52s (non-voting) | ||
644 | build-openstack-sphinx-docs SUCCESS in 2m 57s | ||
645 | shade-tox-py27-tips SUCCESS in 3m 18s | ||
646 | shade-tox-py35-tips SUCCESS in 2m 28s | ||
647 | |||
648 | Live Configuration Changes | ||
649 | ========================== | ||
650 | |||
651 | .. container:: handout | ||
652 | |||
653 | Zuul is a distributed system, with a distributed configuration. | ||
654 | |||
655 | .. code:: yaml | ||
656 | |||
657 | - tenant: | ||
658 | name: openstack | ||
659 | source: | ||
660 | gerrit: | ||
661 | config-projects: | ||
662 | - project-config | ||
663 | untrusted-projects: | ||
664 | - openstack-infra/zuul-jobs: | ||
665 | shadow: openstack-infra/project-config | ||
666 | - openstack-infra/openstack-zuul-jobs | ||
667 | - openstack-infra/nodepool | ||
668 | - openstack-infra/shade | ||
669 | - openstack-infra/zuul | ||
670 | - openstack/requirements | ||
671 | |||
672 | Zuul Startup | ||
673 | ============ | ||
674 | |||
675 | * Read config file | ||
676 | |||
677 | Zuul Startup | ||
678 | ============ | ||
679 | |||
680 | * Read config file | ||
681 | * Ask mergers for branches of each repo | ||
682 | |||
683 | .. ansi:: images/startup1.ans | ||
684 | |||
685 | Zuul Startup | ||
686 | ============ | ||
687 | |||
688 | * Read config file | ||
689 | * Ask mergers for branches of each repo | ||
690 | * Ask mergers for .zuul.yaml file for each branch of each repo | ||
691 | |||
692 | ``.zuul.yaml`` can be ``^\.?zuul.ya?ml$`` file or ``^\.?zuul.d$`` run-parts | ||
693 | directory. | ||
694 | |||
695 | .. ansi:: images/startup2.ans | ||
696 | |||
697 | When .zuul.yaml Changes | ||
698 | ======================= | ||
699 | |||
700 | .. container:: progressive | ||
701 | |||
702 | * Zuul looks for changes to .zuul.yaml | ||
703 | * Asks mergers for updated content | ||
704 | * Splices into configuration used for that change | ||
705 | * Works with cross-repo dependencies | ||
706 | |||
707 | ("This change depends on a change to the job definition") | ||
708 | |||
709 | How do you use this thing? | ||
710 | ========================== | ||
711 | .. transition:: tilt | ||
712 | .. hidetitle:: | ||
713 | .. figlet:: Configuration | ||
714 | |||
715 | Pipelines | ||
716 | ========= | ||
717 | |||
718 | * Describes the process flow and lifecycle **for a change** | ||
719 | * A process definition that connects git repositories, jobs, and | ||
720 | reporting mechanisms. | ||
721 | * A context to fix a set of jobs to each project. | ||
722 | |||
723 | Check Pipeline | ||
724 | ============== | ||
725 | |||
726 | .. code:: yaml | ||
727 | |||
728 | - pipeline: | ||
729 | name: check | ||
730 | manager: independent | ||
731 | source: gerrit | ||
732 | trigger: | ||
733 | gerrit: | ||
734 | - event: patchset-created | ||
735 | - event: change-restored | ||
736 | success: | ||
737 | gerrit: | ||
738 | verified: 1 | ||
739 | |||
740 | Gate Pipeline | ||
741 | ============= | ||
742 | |||
743 | .. code:: yaml | ||
744 | |||
745 | - pipeline: | ||
746 | name: gate | ||
747 | manager: dependent | ||
748 | trigger: | ||
749 | gerrit: | ||
750 | - event: comment-added | ||
751 | approval: | ||
752 | - workflow: 1 | ||
753 | success: | ||
754 | gerrit: | ||
755 | verified: 2 | ||
756 | submit: true | ||
757 | |||
758 | Zuul Github Support | ||
759 | =================== | ||
760 | |||
761 | .. code:: yaml | ||
762 | |||
763 | - pipeline: | ||
764 | name: check | ||
765 | manager: independent | ||
766 | trigger: | ||
767 | github: | ||
768 | - event: pull_request | ||
769 | action: | ||
770 | - opened | ||
771 | - changed | ||
772 | - reopened | ||
773 | success: | ||
774 | github: | ||
775 | status: 'success' | ||
776 | failure: | ||
777 | github: | ||
778 | status: 'failure' | ||
779 | |||
780 | OpenStack Github Support for Cross Community Testing | ||
781 | ==================================================== | ||
782 | |||
783 | * Github App "OpenStack Zuul" | ||
784 | * App added to github project by project admin | ||
785 | * Project aded to OpenStack's main.yaml | ||
786 | * Test interactions between OpenStack and important adjacent communities | ||
787 | |||
788 | * Ansible | ||
789 | * Kubernetes | ||
790 | * Ceph? | ||
791 | * Open vSwitch? | ||
792 | |||
793 | Cross Community Testing | ||
794 | ======================= | ||
795 | |||
796 | .. code:: yaml | ||
797 | |||
798 | - pipeline: | ||
799 | name: check | ||
800 | description: | | ||
801 | Newly uploaded patchsets enter this pipeline to receive an | ||
802 | initial +/-1 Verified vote. | ||
803 | manager: independent | ||
804 | trigger: | ||
805 | gerrit: | ||
806 | - event: patchset-created | ||
807 | - event: change-restored | ||
808 | - event: comment-added | ||
809 | comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*recheck | ||
810 | - event: comment-added | ||
811 | require-approval: | ||
812 | - Verified: [-1, -2] | ||
813 | username: zuul | ||
814 | approval: | ||
815 | - Workflow: 1 | ||
816 | github: | ||
817 | - event: pull_request | ||
818 | action: | ||
819 | - opened | ||
820 | - changed | ||
821 | - reopened | ||
822 | - event: pull_request | ||
823 | action: comment | ||
824 | comment: (?i)^\s*recheck\s*$ | ||
825 | |||
826 | Cross Community Support cont. | ||
827 | ============================= | ||
828 | |||
829 | .. code:: yaml | ||
830 | |||
831 | start: | ||
832 | github: | ||
833 | status: pending | ||
834 | comment: false | ||
835 | success: | ||
836 | gerrit: | ||
837 | # Note that gerrit keywords are case-sensitive. | ||
838 | Verified: 1 | ||
839 | github: | ||
840 | status: 'success' | ||
841 | mysql: | ||
842 | failure: | ||
843 | gerrit: | ||
844 | Verified: -1 | ||
845 | github: | ||
846 | status: 'failure' | ||
847 | mysql: | ||
848 | |||
849 | Cross Community Depends-On (coming soon) | ||
850 | ======================================== | ||
851 | |||
852 | .. container:: progressive | ||
853 | |||
854 | * shade https://review.openstack.org/#/c/613914/ | ||
855 | |||
856 | Add support for server groups | ||
857 | |||
858 | Change-ID: I5b411be5c5aa43535fa89a51d6099aadd7a8ea61 | ||
859 | |||
860 | * ansible https://github.com/ansible/ansible/pull/32159 | ||
861 | |||
862 | Add os_server_group module | ||
863 | |||
864 | Depends-On: https://review.openstack.org/#/613914/ | ||
865 | |||
866 | Jobs | ||
867 | ==== | ||
868 | |||
869 | * Jobs run on nodes from nodepool (static or dynamic) | ||
870 | * Metadata defined in Zuul's configuration | ||
871 | * Execution content in Ansible (with live streaming!) | ||
872 | * Jobs may be defined centrally or in the repo being tested | ||
873 | * Jobs have contextual variants that simplify configuration | ||
874 | * git.openstack.org/openstack-infra/zuul-jobs repo can be directly shared | ||
875 | between zuul installations | ||
876 | |||
877 | Job | ||
878 | === | ||
879 | |||
880 | .. code:: yaml | ||
881 | |||
882 | - job: | ||
883 | name: base | ||
884 | parent: null | ||
885 | description: | | ||
886 | The base job for Zuul. | ||
887 | timeout: 1800 | ||
888 | nodeset: | ||
889 | nodes: | ||
890 | - name: primary | ||
891 | label: centos-7 | ||
892 | pre-run: playbooks/base/pre | ||
893 | post-run: | ||
894 | - playbooks/base/post-ssh | ||
895 | - playbooks/base/post-logs | ||
896 | secrets: | ||
897 | - site_logs | ||
898 | |||
899 | Simple Job | ||
900 | ========== | ||
901 | |||
902 | .. code:: yaml | ||
903 | |||
904 | - job: | ||
905 | name: tox | ||
906 | pre-run: playbooks/setup-tox | ||
907 | run: playbooks/tox | ||
908 | post-run: playbooks/fetch-tox-output | ||
909 | |||
910 | - job: | ||
911 | name: tox-py27 | ||
912 | parent: tox | ||
913 | vars: | ||
914 | tox_envlist: py27 | ||
915 | |||
916 | |||
917 | Simple Job Variant | ||
918 | ================== | ||
919 | |||
920 | .. code:: yaml | ||
921 | |||
922 | - job: | ||
923 | name: tox-py27 | ||
924 | branch: stable/mitaka | ||
925 | nodeset: | ||
926 | - name: ubuntu-trusty | ||
927 | label: ubuntu-trusty | ||
928 | |||
929 | Nodesets for Multi-node Jobs | ||
930 | ============================ | ||
931 | |||
932 | .. code:: yaml | ||
933 | |||
934 | - nodeset: | ||
935 | name: ceph-cluster | ||
936 | nodes: | ||
937 | - name: controller | ||
938 | label: centos-7 | ||
939 | - name: compute1 | ||
940 | label: fedora-26 | ||
941 | - name: compute2 | ||
942 | label: fedora-26 | ||
943 | groups: | ||
944 | - name: ceph-osd | ||
945 | nodes: | ||
946 | - controller | ||
947 | - name: ceph-monitor | ||
948 | nodes: | ||
949 | - controller | ||
950 | - compute1 | ||
951 | - compute2 | ||
952 | |||
953 | Multi-node Job | ||
954 | ============== | ||
955 | |||
956 | * nodesets are provided to Ansible for jobs in inventory | ||
957 | |||
958 | .. code:: yaml | ||
959 | |||
960 | - job: | ||
961 | name: ceph-multinode | ||
962 | nodeset: ceph-cluster | ||
963 | run: playbooks/install-ceph | ||
964 | |||
965 | Multi-node Ceph Job Content | ||
966 | =========================== | ||
967 | |||
968 | .. code:: yaml | ||
969 | |||
970 | - hosts: all | ||
971 | roles: | ||
972 | - install-ceph | ||
973 | - hosts: ceph-osd | ||
974 | roles: | ||
975 | - start-ceph-osd | ||
976 | - hosts: ceph-monitor | ||
977 | roles: | ||
978 | - start-ceph-monitor | ||
979 | - hosts: all | ||
980 | roles: | ||
981 | - do-something-interesting | ||
982 | |||
983 | Projects | ||
984 | ======== | ||
985 | |||
986 | * Projects are git repositories | ||
987 | * Specify a set of jobs for each pipeline | ||
988 | * golang git repo naming as been adopted: | ||
989 | |||
990 | :: | ||
991 | |||
992 | zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d | ||
993 | src/git.openstack.org/openstack-infra/shade | ||
994 | src/git.openstack.org/openstack/os-client-config | ||
995 | src/github.com/ansible/ansible | ||
996 | |||
997 | Project | ||
998 | ======= | ||
999 | |||
1000 | .. code:: yaml | ||
1001 | |||
1002 | - project: | ||
1003 | # Needing to name the project in that project's .zuul.yaml is going away | ||
1004 | name: openstack/nova | ||
1005 | check: | ||
1006 | jobs: | ||
1007 | - openstack-tox-py27 | ||
1008 | - openstack-tox-py35 | ||
1009 | - openstack-doc-build | ||
1010 | |||
1011 | Project with Local Variant | ||
1012 | ========================== | ||
1013 | |||
1014 | .. code:: yaml | ||
1015 | |||
1016 | - project: | ||
1017 | name: openstack/nova | ||
1018 | check: | ||
1019 | jobs: | ||
1020 | - openstack-tox-py27 | ||
1021 | - openstack-tox-py35 | ||
1022 | - openstack-doc-build | ||
1023 | - openstack-tox-pypy: | ||
1024 | voting: false | ||
1025 | |||
1026 | Project with More Local Variants | ||
1027 | ================================ | ||
1028 | |||
1029 | .. code:: yaml | ||
1030 | |||
1031 | - project: | ||
1032 | name: openstack/nova | ||
1033 | check: | ||
1034 | jobs: | ||
1035 | - openstack-tox-py27 | ||
1036 | - openstack-tox-py35 | ||
1037 | - openstack-doc-build: | ||
1038 | files: '^docs/.*$' | ||
1039 | - openstack-tox-pypy: | ||
1040 | voting: false | ||
1041 | |||
1042 | Project with Many Local Variants | ||
1043 | ================================ | ||
1044 | |||
1045 | .. code:: yaml | ||
1046 | |||
1047 | - project: | ||
1048 | name: openstack/nova | ||
1049 | check: | ||
1050 | jobs: | ||
1051 | - openstack-tox-py27 | ||
1052 | nodeset: | ||
1053 | - name: centos-7 | ||
1054 | label: centos-7 | ||
1055 | - openstack-tox-py27 | ||
1056 | branch: stable/newton | ||
1057 | nodeset: | ||
1058 | - name: ubuntu-trusty | ||
1059 | label: ubuntu-trusty | ||
1060 | - openstack-doc-build: | ||
1061 | files: '^docs/.*$' | ||
1062 | - openstack-tox-pypy: | ||
1063 | voting: false | ||
1064 | |||
1065 | Project With Central and Local Config | ||
1066 | ===================================== | ||
1067 | |||
1068 | .. code:: yaml | ||
1069 | |||
1070 | # In git.openstack.org/openstack-infra/project-config: | ||
1071 | - project: | ||
1072 | name: openstack/nova | ||
1073 | templates: | ||
1074 | - openstack-tox-jobs | ||
1075 | |||
1076 | .. code:: yaml | ||
1077 | |||
1078 | # In git.openstack.org/openstack/nova/.zuul.yaml: | ||
1079 | - project: | ||
1080 | name: openstack/nova | ||
1081 | check: | ||
1082 | - nova-placement-functional-devstack | ||
1083 | |||
1084 | Project with Job Dependencies | ||
1085 | ============================= | ||
1086 | |||
1087 | .. code:: yaml | ||
1088 | |||
1089 | - project: | ||
1090 | name: openstack/nova | ||
1091 | release: | ||
1092 | jobs: | ||
1093 | - build-artifacts | ||
1094 | - upload-tarball: | ||
1095 | dependencies: build-artifacts | ||
1096 | - upload-pypi: | ||
1097 | dependencies: build-artifacts | ||
1098 | - notify-mirror: | ||
1099 | dependencies: | ||
1100 | - upload-tarball | ||
1101 | - upload-pypi | ||
1102 | |||
1103 | Playbooks | ||
1104 | ========= | ||
1105 | |||
1106 | * Jobs run Ansible playbooks | ||
1107 | * Playbooks may be defined centrally or in the repo being tested | ||
1108 | * Playbooks can use roles from current or other Zuul repos (or Galaxy, coming soon) | ||
1109 | * Playbooks are run on the zuul-executor using bubblewrap https://github.com/projectatomic/bubblewrap | ||
1110 | * Playbooks are not allowed to execute content on 'localhost' | ||
1111 | |||
1112 | Job with Roles | ||
1113 | ============== | ||
1114 | |||
1115 | .. code:: yaml | ||
1116 | |||
1117 | - job: | ||
1118 | name: zuul-integration | ||
1119 | description: | | ||
1120 | Multi-node Zuul installation and integration test | ||
1121 | nodeset: zuul-cluster | ||
1122 | roles: | ||
1123 | - zuul: openstack/ansible-role-zuul | ||
1124 | run: playbooks/zuul-integration | ||
1125 | |||
1126 | Job with Multiple Projects | ||
1127 | ========================== | ||
1128 | |||
1129 | .. code:: yaml | ||
1130 | |||
1131 | - job: | ||
1132 | name: tox-py35-on-zuul | ||
1133 | parent: tox-py35 | ||
1134 | description: | | ||
1135 | Run zuul's py35 unittests on patches to zuul-jobs | ||
1136 | vars: | ||
1137 | zuul_work_dir: src/git.openstack.org/openstack-infra/zuul | ||
1138 | required-projects: | ||
1139 | - name: openstack-infra/zuul | ||
1140 | override-branch: feature/zuulv3 | ||
1141 | |||
1142 | - project: openstack-infra/zuul-jobs | ||
1143 | check: | ||
1144 | jobs: | ||
1145 | - tox-py35-on-zuul | ||
1146 | |||
1147 | Devstack-gate / Tempest Playbook | ||
1148 | ================================ | ||
1149 | |||
1150 | .. code:: yaml | ||
1151 | |||
1152 | # devstack-gate / tempest playbook | ||
1153 | hosts: all | ||
1154 | roles: | ||
1155 | - setup-multinode-networking | ||
1156 | - partition-swap | ||
1157 | - configure-mirrors | ||
1158 | - run-devstack | ||
1159 | - run-tempest | ||
1160 | |||
1161 | Simple Shell Playbook | ||
1162 | ===================== | ||
1163 | |||
1164 | .. code:: yaml | ||
1165 | |||
1166 | hosts: controller | ||
1167 | tasks: | ||
1168 | - shell: ./run_tests.sh | ||
1169 | |||
1170 | Test Like Production | ||
1171 | ==================== | ||
1172 | |||
1173 | If you use Ansible for deployment, your test and deployment processes | ||
1174 | and playbooks are the same | ||
1175 | |||
1176 | What if you don't use Ansible? | ||
1177 | ============================== | ||
1178 | |||
1179 | OpenStack Infra Control Plane uses Puppet | ||
1180 | ========================================= | ||
1181 | |||
1182 | .. code:: yaml | ||
1183 | |||
1184 | # In git.openstack.org/openstack-infra/project-config/roles/legacy-install-afs-with-puppet/tasks/main.yaml | ||
1185 | - name: Install puppet | ||
1186 | shell: ./install_puppet.sh | ||
1187 | args: | ||
1188 | chdir: "{{ ansible_user_dir }}/src/git.openstack.org/openstack-infra/system-config" | ||
1189 | environment: | ||
1190 | # Skip setting up pip, our images have already done this. | ||
1191 | SETUP_PIP: "false" | ||
1192 | become: yes | ||
1193 | |||
1194 | - name: Copy manifest | ||
1195 | copy: | ||
1196 | src: manifest.pp | ||
1197 | dest: "{{ ansible_user_dir }}/manifest.pp" | ||
1198 | |||
1199 | - name: Run puppet | ||
1200 | puppet: | ||
1201 | manifest: "{{ ansible_user_dir }}/manifest.pp" | ||
1202 | become: yes | ||
1203 | |||
1204 | Secrets | ||
1205 | ======= | ||
1206 | |||
1207 | * Inspired by Kubernetes Secrets API | ||
1208 | * Projects can add named encrypted secrets to their .zuul.yaml file | ||
1209 | * Jobs can request to use secrets by name | ||
1210 | * Jobs using secrets are not reconfigured speculatively | ||
1211 | * Secrets can only be used by the same project they are defined in | ||
1212 | * Public key per project: | ||
1213 | ``{{ zuul_url }}/{{ tenant }}/keys/{{ project }}.pub`` | ||
1214 | |||
1215 | :: | ||
1216 | GET http://zuulv3.openstack.org/keys/openstack-infra/shade.pub | ||
1217 | |||
1218 | Secret Example (note, no admins had to enable this) | ||
1219 | =================================================== | ||
1220 | |||
1221 | .. code:: yaml | ||
1222 | |||
1223 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
1224 | - secret: | ||
1225 | name: loci_docker_login | ||
1226 | data: | ||
1227 | user: !encrypted/pkcs1-oaep | ||
1228 | - r8Nbpq5olmfLF035BZ/CUoFLIdhvBi/49KuochOAHbvns+xMiho3C7MEFzYDqJX3IhHde | ||
1229 | BICYOgK7qnyINOIZL2e7pl75rEdHQwJjSFUMkpdY6wEP7f9hpolj9xVp0ifHUVQqPHMRn | ||
1230 | zoPFd8MEAHxH5GLmc2SWJ98E/QUqGltxBi1YRSZoCcNtq3tHFK5Y+xQlLhIseJ2HkpDs6 | ||
1231 | YXOGP9Qt4Va6sdyBcA90H+apSAcYA3Duu962ySZQAsYNui/3NQq3gLA+OZeyTJtcrh4hj | ||
1232 | Rb5dBnDWfSrMpxdNkbPXXgbQaxO3T0L4jbaOF8VKEsiI9olBrOeV2M9ddYJjSsHGj4XR8 | ||
1233 | 4vwS0+doB7np93fujiDuHVgdG8R40NW2GznyKRlRtzAORla7Mzw1Y1MokcUyY6p1LlLLl | ||
1234 | wUuWYCCEuRciOPhZXQ2u42qju/zrK2/dPnO8HfUINSrN0WbNq14ZwPpbj0ro02oGPbtwu | ||
1235 | OTw1z+N0Nc+GuLWlwYJGYM/z0UnvDR3WEBc2kXbVev9w4n0cB3RyphML2PDZZWbw8tjnX | ||
1236 | h1VsAOJ0Qo4qq1K/ft95ypd+vtjkfepEgHEBmJNwutJa9IHAkGfrkO9VkpUTPpfffnPwz | ||
1237 | d0/zaaadNl6MLQUSutRwY23YIIbv+fmukxw2vnJmvn6abkBlMya7KgtifwNA8c= | ||
1238 | password: !encrypted/pkcs1-oaep | ||
1239 | - gUEX4eY3JAk/Xt7Evmf/hF7xr6HpNRXTibZjrKTbmI4QYHlzEBrBbHey27Pt/eYvKKeKw | ||
1240 | hk8MDQ4rNX7ZK1v+CKTilUfOf4AkKYbe6JFDd4z+zIZ2PAA7ZedO5FY/OnqrG7nhLvQHE | ||
1241 | 5nQrYwmxRp4O8eU5qG1dSrM9X+bzri8UnsI7URjqmEsIvlUqtybQKB9qQXT4d6mOeaKGE | ||
1242 | 5h6Ydkb9Zdi4Qh+GpCGDYwHZKu1mBgVK5M1G6NFMy1DYz+4NJNkTRe9J+0TmWhQ/KZSqo | ||
1243 | 4ck0x7Tb0Nr7hQzV8SxlwkaCTLDzvbiqmsJPLmzXY2jry6QsaRCpthS01vnj47itoZ/7p | ||
1244 | taH9CoJ0Gl7AkaxsrDSVjWSjatTQpsy1ub2fuzWHH4ASJFCiu83Lb2xwYts++r8ZSn+mA | ||
1245 | hbEs0GzPI6dIWg0u7aUsRWMOB4A+6t2IOJibVYwmwkG8TjHRXxVCLH5sY+i3MR+NicR9T | ||
1246 | IZFdY/AyH6vt5uHLQDU35+5n91pUG3F2lyiY5aeMOvBL05p27GTMuixR5ZoHcvSoHHtCq | ||
1247 | 7Wnk21iHqmv/UnEzqUfXZOque9YP386RBWkshrHd0x3OHUfBK/WrpivxvIGBzGwMr2qAj | ||
1248 | /AhJsfDXKBBbhGOGk1u5oBLjeC4SRnAcIVh1+RWzR4/cAhOuy2EcbzxaGb6VTM= | ||
1249 | |||
1250 | Secret Example | ||
1251 | ============== | ||
1252 | |||
1253 | .. code:: yaml | ||
1254 | |||
1255 | # In git.openstack.org/openstack/loci/.zuul.yaml: | ||
1256 | - job: | ||
1257 | name: publish-loci-cinder | ||
1258 | parent: loci-cinder | ||
1259 | post-run: playbooks/push | ||
1260 | secrets: | ||
1261 | - loci_docker_login | ||
1262 | |||
1263 | # In git.openstack.org/openstack/loci/playbooks/push.yaml: | ||
1264 | - hosts: all | ||
1265 | tasks: | ||
1266 | - include_vars: vars.yaml | ||
1267 | |||
1268 | - name: Push project to DockerHub | ||
1269 | block: | ||
1270 | - command: docker login -u {{ loci_docker_login.user }} -p {{ loci_docker_login.password }} | ||
1271 | no_log: True | ||
1272 | - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} | ||
1273 | with_items: "{{ distros }}" | ||
1274 | |||
1275 | Status | ||
1276 | ====== | ||
1277 | |||
1278 | * zuulv3 is running for OpenStack | ||
1279 | * zuulv3 also runing at BMW (in OpenShift) | ||
1280 | * automation job migration sucked (I wrote the script, whoops) | ||
1281 | * cleaning up fixing automation and bugs found running at scale | ||
1282 | * reference documentation exists and is complete | ||
1283 | * pre-repository job documentation | ||
1284 | * will release v3.0 once we're satisfied it's good for other people | ||
1285 | |||
1286 | What's Next? | ||
1287 | ============ | ||
1288 | |||
1289 | * dashboard + REST API | ||
1290 | * user and deployer *documentation* - getting started walkthroughs | ||
1291 | * node providers: | ||
1292 | * static | ||
1293 | * OCI/docker | ||
1294 | * Mac Stadium (for our Ansible friends) | ||
1295 | * bifrost | ||
1296 | * support for galaxy roles | ||
1297 | * in-line code-review comments from Zuul | ||
1298 | * shared job doc generation | ||
1299 | * native container/kubernetes execution? | ||
1300 | |||
1301 | Important Links | ||
1302 | =============== | ||
1303 | |||
1304 | * https://git.openstack.org/cgit/openstack-infra/zuul/log/?h=feature/zuulv3 | ||
1305 | * https://docs.openstack.org/infra/zuul/feature/zuulv3/ | ||
1306 | * https://docs.openstack.org/infra/manual/zuulv3.html | ||
1307 | * https://docs.openstack.org/infra/openstack-zuul-jobs/ | ||
1308 | * https://storyboard.openstack.org/#!/project/679 | ||
1309 | * https://storyboard.openstack.org/#!/board/41 | ||
1310 | * freenode:#zuul | ||
1311 | |||
1312 | Questions | ||
1313 | ========= | ||
1314 | |||
1315 | .. ansi:: images/questions.ans | ||
1316 | |||
1317 | Presentty | ||
1318 | ========= | ||
1319 | .. hidetitle:: | ||
1320 | .. transition:: pan | ||
1321 | .. figlet:: Presentty | ||
1322 | |||
1323 | * Console presentations written in reStructuredText | ||
1324 | * Cross-fade, pan, tilt, cut transitions | ||
1325 | * Figlet, cowsay! | ||
1326 | * https://pypi.python.org/pypi/presentty | ||