summaryrefslogtreecommitdiff
path: root/src/zuulv3
diff options
context:
space:
mode:
Diffstat (limited to 'src/zuulv3')
-rw-r--r--src/zuulv3/overview.rst200
1 files changed, 136 insertions, 64 deletions
diff --git a/src/zuulv3/overview.rst b/src/zuulv3/overview.rst
index a14e671..f1b86dc 100644
--- a/src/zuulv3/overview.rst
+++ b/src/zuulv3/overview.rst
@@ -70,7 +70,7 @@ OpenDev
70 70
71 -- Alex Gaynor 71 -- Alex Gaynor
72 72
73 "OpenStack Infra are like the SpaceX of CI" 73 "like the SpaceX of CI"
74 74
75 -- Emily Dunham 75 -- Emily Dunham
76 76
@@ -83,6 +83,7 @@ Zuul
83What Zuul Does 83What Zuul Does
84============== 84==============
85 85
86 * "Speculative Future State"
86 * gated changes 87 * gated changes
87 * one or more git repositories 88 * one or more git repositories
88 * integrated deliverable 89 * integrated deliverable
@@ -93,6 +94,7 @@ Underlying Philosophy
93 94
94 * All changes flow through code review 95 * All changes flow through code review
95 * Changes only land if they pass all tests 96 * Changes only land if they pass all tests
97 * End-to-end integration testing is essential
96 * Computers are cheaper than humans 98 * Computers are cheaper than humans
97 99
98Ramifications of Philosophy 100Ramifications of Philosophy
@@ -146,8 +148,8 @@ Gating
146====== 148======
147 149
148 * Changes are pushed to code review 150 * Changes are pushed to code review
149 * CI system is triggered by code review approval 151 * Gating system is triggered by code review approval
150 * CI system merges code IFF tests pass 152 * Gating system merges code IFF tests pass
151 * Proactive - testing code before it lands 153 * Proactive - testing code before it lands
152 * Future state resulting from merge of code is tested 154 * Future state resulting from merge of code is tested
153 * Reviewers can fire-and-forget safely 155 * Reviewers can fire-and-forget safely
@@ -400,35 +402,6 @@ Zuul Simulation
400.. ansi:: images/zsim-22.ans 402.. ansi:: images/zsim-22.ans
401 403
402 404
403Cross-Project Dependencies
404==========================
405
406Testing or gating dependencies manually specified by developers
407
408.. container:: progressive
409
410 * nodepool https://review.openstack.org/612168
411
412 Make functional src jobs actually install from source
413 * openstacksdk https://review.openstack.org/612186
414
415 Don't start task managers passed in to Connection
416
417 Depends-On: https://review.openstack.org/612168
418 * openstacksdk https://review.openstack.org/604521
419
420 Add support for per-service rate limits
421
422 (git parent is 612186)
423 * nodepool https://review.openstack.org/612169
424
425 Consume rate limiting task manager from openstacksdk
426
427 Depends-On: https://review.openstack.org/604521
428
429 (nodepool-functional-py35-src should pass, but
430 nodepool-functional-py35 should not fail until openstacksdk release)
431
432Lock Step Changes 405Lock Step Changes
433================= 406=================
434 407
@@ -452,7 +425,7 @@ Live Configuration Changes
452 config-repos: 425 config-repos:
453 - opendev/project-config 426 - opendev/project-config
454 project-repos: 427 project-repos:
455 - opendev/zuul-jobs 428 - zuul/zuul-jobs
456 - zuul/zuul 429 - zuul/zuul
457 - zuul/nodepool 430 - zuul/nodepool
458 - ansible/ansible 431 - ansible/ansible
@@ -494,16 +467,89 @@ When .zuul.yaml Changes
494 467
495 ("This change depends on a change to the job definition") 468 ("This change depends on a change to the job definition")
496 469
470Explicit Cross-Project Dependencies
471===================================
472
473 * Developers can mark changes as being dependent
474 * Depends-On: footer - in commit or PR
475 * Zuul uses depends-on when constructing virtual serial queue
476 * Will not merge changes in gate before depends-on changes
477 * Works cross-repo AND cross-source
478
479Depends-On Example
480==================
481
482 * Service 'nova' talks to service 'ironic'
483 * Currently using 'python-ironicclient'
484 * Want to replace python-ironicclient with openstacksdk:
485 * https://review.openstack.org/643664
486 * Need some plumbing in nova first:
487 * https://review.openstack.org/642899
488 * That change "Depends-On" a change to openstacksdk
489
490Depends-On Example - openstacksdk
491=================================
492
493 * In openstacksdk, need a new method to extract config differently
494 * https://review.openstack.org/643601
495 * The nova plumbing change adds this:
496
497::
498
499 Depends-On: https://review.openstack.org/643601
500
501Depends-On Example - keystoneauth
502=================================
503
504 * openstacksdk uses 'keystoneauth' library to make REST calls
505 * Config extraction change wants a new helper method in keystoneauth
506 * https://review.openstack.org/644251
507 * openstacksdk change adds:
508
509::
510
511 Depends-On: https://review.openstack.org/644251
512
513Depends-On Example - In the Gate
514================================
515
516 * When Zuul prepares git repos for the Ironic nova change:
517 * Tip of nova, plus nova plumbing change, plus nova ironic change
518 * Tip of openstacksdk, plus config method change
519 * Tip of keystoneauth, plus helper method change
520 * Developers iterate on the nova service change
521 * BEFORE finalizing and releasing keystoneauth and openstacksdk changes
522
523Zuul Architecture
524=================
525
526We used to call "microservices" "distributed"
527
528 * Zuul is comprised of several services (mostly python3)
529 * zuul-scheduler
530 * zuul-executor
531 * zuul-merger
532 * zuul-web
533 * zuul-dashboard (javascript/react)
534 * zuul-fingergw
535 * zuul-proxy (c++)
536 * RDBMS
537 * Gearman
538 * Zookeeper
539
497Zuul Architecture 540Zuul Architecture
498================= 541=================
499 542
500.. ansi:: images/architecture.ans 543.. ansi:: images/architecture.ans
501 544
545Where Does Job Content Run?
546===========================
502 547
503Nodepool 548Nodepool
504======== 549========
505 550
506 * A separate program that works very closely with *Zuul* 551 * A separate program that works very closely with *Zuul*
552 * *Zuul* requires *Nodepool* but *Nodepool* can be used independently
507 * Creates and destroys zero or more node resources 553 * Creates and destroys zero or more node resources
508 * Resources can include VMs, Containers, COE contexts or Bare Metals 554 * Resources can include VMs, Containers, COE contexts or Bare Metals
509 * Static driver for allocating pre-existing nodes to jobs 555 * Static driver for allocating pre-existing nodes to jobs
@@ -517,17 +563,12 @@ Nodepool Launcher
517 * OpenStack 563 * OpenStack
518 * Static 564 * Static
519 * Kubernetes 565 * Kubernetes
520
521 In review:
522
523 * OpenShift
524 * AWS 566 * AWS
525 567
526 In work / coming soon: 568 In work / coming soon:
527 569
528 * Azure 570 * Azure
529 * GCE 571 * GCE
530 * Mac Stadium
531 572
532What about test/job content? 573What about test/job content?
533============================ 574============================
@@ -541,16 +582,15 @@ What Zuul Does
541 582
542 * Listens for code events 583 * Listens for code events
543 * Prepares appropriate job config and git repo states 584 * Prepares appropriate job config and git repo states
544 * Allocates nodes for test jobs 585 * Requests nodes for test jobs from *Nodepool*
545 * Pushes git repo states to nodes 586 * Runs user-defined Ansible playbooks with nodes in an inventory
546 * Runs user-defined Ansible playbooks
547 * Collects/reports results 587 * Collects/reports results
548 * Potentially merges change 588 * Potentially merges change
549 589
550Jobs 590Jobs
551==== 591====
552 592
553* Jobs run on nodes from nodepool (static or dynamic) 593* Jobs define test node needs
554* Metadata defined in Zuul's configuration 594* Metadata defined in Zuul's configuration
555* Execution content in Ansible 595* Execution content in Ansible
556* Jobs may be defined centrally or in the repo being tested 596* Jobs may be defined centrally or in the repo being tested
@@ -570,7 +610,7 @@ Job
570 nodeset: 610 nodeset:
571 nodes: 611 nodes:
572 - name: primary 612 - name: primary
573 label: centos-7 613 label: ubuntu-bionic
574 pre-run: playbooks/base/pre.yaml 614 pre-run: playbooks/base/pre.yaml
575 post-run: 615 post-run:
576 - playbooks/base/post-ssh.yaml 616 - playbooks/base/post-ssh.yaml
@@ -669,6 +709,21 @@ Multi-node Job
669 nodeset: ceph-cluster 709 nodeset: ceph-cluster
670 run: playbooks/install-ceph.yaml 710 run: playbooks/install-ceph.yaml
671 711
712* Creates ansible inventory:
713
714.. code:: yaml
715
716 controller ansible_host=1.2.3.4
717 compute1 ansible_host=1.2.3.5
718 compute2 ansible_host=1.2.3.6
719
720 [ceph-osd]
721 controller
722
723 [ceph-monitor]
724 controller
725 compute1
726 compute2
672 727
673Multi-node Ceph Job Content 728Multi-node Ceph Job Content
674=========================== 729===========================
@@ -696,7 +751,7 @@ Project With Central and Local Config
696 751
697.. code:: yaml 752.. code:: yaml
698 753
699 # In git.openstack.org/openstack-infra/project-config: 754 # In opendev.org/openstack-infra/project-config:
700 - project: 755 - project:
701 name: openstack/nova 756 name: openstack/nova
702 templates: 757 templates:
@@ -704,11 +759,18 @@ Project With Central and Local Config
704 759
705.. code:: yaml 760.. code:: yaml
706 761
707 # In git.openstack.org/openstack/nova/.zuul.yaml: 762 # In opendev.org/openstack/nova/.zuul.yaml:
708 - project: 763 - project:
709 check: 764 check:
710 - nova-placement-functional-devstack 765 - nova-placement-functional-devstack
711 766
767zuul-jobs standard library
768==========================
769
770* https://opendev.org/openstack-infra/zuul-jobs
771* Repo containing general purpose job definitions
772* Add the git repo directly to a local Zuul config
773
712Project with Job Dependencies 774Project with Job Dependencies
713============================= 775=============================
714 776
@@ -746,7 +808,7 @@ Secret Example (note, no admins had to enable this)
746 808
747.. code:: yaml 809.. code:: yaml
748 810
749 # In git.openstack.org/openstack/loci/.zuul.yaml: 811 # In opendev.org/openstack/loci/.zuul.yaml:
750 - secret: 812 - secret:
751 name: loci_docker_login 813 name: loci_docker_login
752 data: 814 data:
@@ -768,7 +830,7 @@ Secret Example
768 830
769.. code:: yaml 831.. code:: yaml
770 832
771 # In git.openstack.org/openstack/loci/.zuul.yaml: 833 # In opendev.org/openstack/loci/.zuul.yaml:
772 - job: 834 - job:
773 name: publish-loci-cinder 835 name: publish-loci-cinder
774 parent: loci-cinder 836 parent: loci-cinder
@@ -776,7 +838,7 @@ Secret Example
776 secrets: 838 secrets:
777 - loci_docker_login 839 - loci_docker_login
778 840
779 # In git.openstack.org/openstack/loci/playbooks/push.yaml: 841 # In opendev.org/openstack/loci/playbooks/push.yaml:
780 - hosts: all 842 - hosts: all
781 tasks: 843 tasks:
782 - include_vars: vars.yaml 844 - include_vars: vars.yaml
@@ -788,14 +850,14 @@ Secret Example
788 - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }} 850 - command: docker push openstackloci/{{ project }}:{{ branch }}-{{ item.name }}
789 with_items: "{{ distros }}" 851 with_items: "{{ distros }}"
790 852
791OpenDev - Largest Known Zuul 853Speculative Conatiner Images
792==================================== 854============================
793 855
794 * 2KJPH (2,000 jobs per hour) 856 * Gating applied to continuously deployed container images
795 * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds 857 * Build and test images that depend on other images
796 * Rackspace, Internap, OVH, Vexxhost, CityCloud 858 * Build and test deployments comprising multiple images
797 * Linaro (ARM), Limestone, Packethost 859 * Without publishing to final location
798 * 10,000 changes merged per month 860 * Publish the actual image that was built in the gate
799 861
800Zuul is not New 862Zuul is not New
801=============== 863===============
@@ -804,13 +866,22 @@ Zuul is not New
804 * Zuul is now a top-level effort of OpenStack Foundation 866 * Zuul is now a top-level effort of OpenStack Foundation
805 * Zuul v3 first release where not-OpenStack is first-class use case 867 * Zuul v3 first release where not-OpenStack is first-class use case
806 868
869OpenDev - Largest Known Zuul
870============================
871
872 * 2KJPH (2,000 jobs per hour)
873 * Build Nodes from 16 Regions of 5 Public and 3 Private OpenStack Clouds
874 * Rackspace, Internap, OVH, Vexxhost, CityCloud
875 * Linaro (ARM), Limestone, Packethost
876 * 10,000 changes merged per month
877
807Not just for OpenStack 878Not just for OpenStack
808====================== 879======================
809 880
810 * BMW (control plane in OpenShift) 881 * BMW (control plane in OpenShift)
811 * GoDaddy (control plane in Kubernetes) 882 * GoDaddy (control plane in private Kubernetes)
883 * GoodMoney (control plane in EKS, adding GKE)
812 * Le Bon Coin 884 * Le Bon Coin
813 * GoodMoney
814 * Easystack 885 * Easystack
815 * TungstenFabric 886 * TungstenFabric
816 * OpenLab 887 * OpenLab
@@ -826,8 +897,10 @@ Code Review Systems
826 In work / coming soon: 897 In work / coming soon:
827 898
828 * Pagure 899 * Pagure
900 * Gitea
829 901
830 Commonly Requested: 902 Commonly Requested:
903
831 * GitLab 904 * GitLab
832 * Bitbucket 905 * Bitbucket
833 906
@@ -846,9 +919,9 @@ Ways to Install Zuul
846==================== 919====================
847 920
848* Containers: https://hub.docker.com/_/zuul/ 921* Containers: https://hub.docker.com/_/zuul/
849* Windmill: http://git.openstack.org/cgit/openstack/windmill 922* Windmill: http://opendev.org/openstack/windmill
850* Software Factory: https://softwarefactory-project.io/ 923* Software Factory: https://softwarefactory-project.io/
851* Puppet: http://git.openstack.org/cgit/openstack-infra/puppet-zuul 924* Puppet: http://opendev.org/openstack-infra/puppet-zuul
852 925
853Zuul Containers 926Zuul Containers
854=============== 927===============
@@ -867,10 +940,9 @@ zuul/zuul-executor
867Release Management 940Release Management
868================== 941==================
869 942
870* Zuul is a CI system 943* Zuul is run Continuously Delivered and Deployed upstream
871* C stands for "Continuous" 944* Some users deploy Zuul with Zuul
872* It is run Continuously Delivered and Deployed upstream 945* Releases are tagged from code run for OpenDev
873* Releases are tagged from code run upstream
874* There is no intent to have a 'stable' release 946* There is no intent to have a 'stable' release
875* 'stable' is a synonym for "old and buggy" 947* 'stable' is a synonym for "old and buggy"
876 948
@@ -890,10 +962,10 @@ Important Links
890=============== 962===============
891 963
892* https://zuul-ci.org/ 964* https://zuul-ci.org/
893* https://git.zuul-ci.org/cgit/zuul
894* https://zuul-ci.org/docs/zuul 965* https://zuul-ci.org/docs/zuul
895* https://zuul-ci.org/docs/zuul-jobs/ 966* https://zuul-ci.org/docs/zuul-jobs/
896* freenode:#zuul 967* freenode:#zuul
968* https://opendev.org/zuul (https://git.zuul-ci.org/cgit/zuul)
897 969
898Questions 970Questions
899========= 971=========