summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMonty Taylor <mordred@inaugust.com>2018-02-21 12:12:45 -0600
committerMonty Taylor <mordred@inaugust.com>2018-02-21 12:43:26 -0600
commit0560476234c83a1a2ec36e0ed803de1b60008502 (patch)
tree93b217c76d5f86f703de5f88466d380733079d91
parent4ac60cf090af9618bd52b4b30cac4ee31b5d5967 (diff)
Update slides for changes since the fall
-rw-r--r--src/zuulv3/zuul.rst158
1 files changed, 84 insertions, 74 deletions
diff --git a/src/zuulv3/zuul.rst b/src/zuulv3/zuul.rst
index 870c373..3ce2437 100644
--- a/src/zuulv3/zuul.rst
+++ b/src/zuulv3/zuul.rst
@@ -67,6 +67,10 @@ OpenStack Infra
67 67
68 -- Alex Gaynor 68 -- Alex Gaynor
69 69
70 "OpenStack Infra are like the SpaceX of CI"
71
72 -- Emily Dunham
73
70Zuul 74Zuul
71==== 75====
72.. container:: handout 76.. container:: handout
@@ -121,25 +125,23 @@ What do I mean by Massive Scale?
121 * Contributors (~2k / 6 month period) 125 * Contributors (~2k / 6 month period)
122 * Companies 126 * Companies
123 * Changes 127 * Changes
124 * Code Repositories (1827 as of this morning) 128 * Code Repositories (1868 as of this morning)
125 * Communities 129 * Communities
126 130
127OpenStack Scale 131OpenStack Scale Comparison
128=============== 132===========================
129 133
130 * 2,000 git repositories
131 * 2KJPH (2,000 jobs per hour) 134 * 2KJPH (2,000 jobs per hour)
132 * Nodes 14 Regions off 5 OpenStack Public Clouds and 2 Private Clouds 135 * Nodes from 12 Regions of 5 Public and 1 Private OpenStack Clouds
133 (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud, HPE, Red Hat) 136 (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro)
134 * 10,000 changes merged per month 137 * 10,000 changes merged per month
135 138
136OpenStack Scale 139OpenStack Scale Comparison
137=============== 140==========================
138 141
139 * 2,000 git repositories
140 * 2KJPH (2,000 jobs per hour) 142 * 2KJPH (2,000 jobs per hour)
141 * Nodes 14 Regions off 5 OpenStack Public Clouds and 2 Private Clouds 143 * Nodes from 12 Regions of 5 Public and 1 Private OpenStack Clouds
142 (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud, HPE, Red Hat) 144 (Thanks Rackspace, Internap, OVH, Vexxhost, CityCloud and Linaro)
143 * 10,000 changes merged per month 145 * 10,000 changes merged per month
144 146
145 * By comparison, our friends at the amazing project Ansible received 147 * By comparison, our friends at the amazing project Ansible received
@@ -150,7 +152,7 @@ Pretty Things to Look for Scale
150 152
151 * http://grafana.openstack.org/dashboard/db/zuul-status 153 * http://grafana.openstack.org/dashboard/db/zuul-status
152 * http://grafana.openstack.org/dashboard/db/nodepool 154 * http://grafana.openstack.org/dashboard/db/nodepool
153 * http://zuulv3.openstack.org/ 155 * http://zuul.openstack.org/
154 156
155Dealing With Scale 157Dealing With Scale
156================== 158==================
@@ -239,13 +241,14 @@ Nodepool
239 nodepool builds nodes for zuul 241 nodepool builds nodes for zuul
240 Remember that 2,000 jobs per hour number? 242 Remember that 2,000 jobs per hour number?
241 Each job gets a fresh VM - that's 2,000 VMs per hours 243 Each job gets a fresh VM - that's 2,000 VMs per hours
242 Treats our 20 regions across 9 clouds as one REALLY big cloud 244 Treats our 12 regions across 6 clouds as one REALLY big cloud
243 245
244:: 246::
245 247
246 * A separate program that works very closely with *zuul* 248 * A separate program that works very closely with *zuul*
247 * Builds images daily and uploads to clouds 249 * Builds images daily and uploads to clouds
248 * Creates and destroys (at least) a VM for every job 250 * Creates and destroys (at least) a VM for every job
251 * Supports using pre-existing nodes (static provider)
249 252
250Nodepool can use pre-existing images, BUT ... 253Nodepool can use pre-existing images, BUT ...
251============================================= 254=============================================
@@ -555,6 +558,7 @@ shade: Add unittest tips jobs
555 Run tox python 27 unittests against master of important libs 558 Run tox python 27 unittests against master of important libs
556 required-projects: 559 required-projects:
557 - openstack-infra/shade 560 - openstack-infra/shade
561 - openstack/keystoneauth
558 - openstack/os-client-config 562 - openstack/os-client-config
559 563
560 - job: 564 - job:
@@ -593,7 +597,6 @@ shade: Add unittest tips project-template to project
593.. code:: yaml 597.. code:: yaml
594 598
595 - project: 599 - project:
596 name: openstack-infra/shade
597 templates: 600 templates:
598 - publish-to-pypi 601 - publish-to-pypi
599 - publish-openstack-sphinx-docs 602 - publish-openstack-sphinx-docs
@@ -607,7 +610,6 @@ os-client-config: Add shade-tox-tips jobs
607.. code:: yaml 610.. code:: yaml
608 611
609 - project: 612 - project:
610 name: openstack/os-client-config
611 templates: 613 templates:
612 - shade-tox-tips 614 - shade-tox-tips
613 check: 615 check:
@@ -697,7 +699,7 @@ Zuul Startup
697* Ask mergers for branches of each repo 699* Ask mergers for branches of each repo
698* Ask mergers for .zuul.yaml file for each branch of each repo 700* Ask mergers for .zuul.yaml file for each branch of each repo
699 701
700``.zuul.yaml`` can be ``^\.?zuul.ya?ml$`` file or ``^\.?zuul.d$`` run-parts 702``.zuul.yaml`` can be ``^\.?zuul.yaml$`` file or ``^\.?zuul.d$`` run-parts
701directory. 703directory.
702 704
703.. ansi:: images/startup2.ans 705.. ansi:: images/startup2.ans
@@ -792,11 +794,7 @@ OpenStack Github Support for Cross Community Testing
792 * App added to github project by project admin 794 * App added to github project by project admin
793 * Project aded to OpenStack's main.yaml 795 * Project aded to OpenStack's main.yaml
794 * Test interactions between OpenStack and important adjacent communities 796 * Test interactions between OpenStack and important adjacent communities
795 797 * https://github.com/ansible/ansible/pull/20974
796 * Ansible
797 * Kubernetes
798 * Ceph?
799 * Open vSwitch?
800 798
801Cross Community Testing 799Cross Community Testing
802======================= 800=======================
@@ -854,22 +852,27 @@ Cross Community Support cont.
854 status: 'failure' 852 status: 'failure'
855 mysql: 853 mysql:
856 854
857Cross Community Depends-On (coming soon) 855Cross Source Dependencies
858======================================== 856=========================
859 857
860.. container:: progressive 858.. container:: progressive
861 859
862 * shade https://review.openstack.org/#/c/613914/ 860 * shade https://review.openstack.org/539563
863 861
864 Add support for server groups 862 Shift voting flag and test_matrix_branch for ansible-devel job
865 863
866 Change-ID: I5b411be5c5aa43535fa89a51d6099aadd7a8ea61 864 Change-ID: Ic9d3983de641dbe618c65b2cbf2dcfa3686575df
867 865
868 * ansible https://github.com/ansible/ansible/pull/32159 866 * ansible https://github.com/ansible/ansible/pull/34925
869 867
870 Add os_server_group module 868 continue fact gathering even without dmidecode
871 869
872 Depends-On: https://review.openstack.org/#/613914/ 870 * ansible https://github.com/ansible/ansible/pull/20974
871
872 Make a generalized OpenStack cloud constructor
873
874 Depends-On: https://review.openstack.org/539563
875 Depends-On: https://github.com/ansible/ansible/pull/34925
873 876
874Jobs 877Jobs
875==== 878====
@@ -879,8 +882,14 @@ Jobs
879* Execution content in Ansible (with live streaming!) 882* Execution content in Ansible (with live streaming!)
880* Jobs may be defined centrally or in the repo being tested 883* Jobs may be defined centrally or in the repo being tested
881* Jobs have contextual variants that simplify configuration 884* Jobs have contextual variants that simplify configuration
882* git.openstack.org/openstack-infra/zuul-jobs repo can be directly shared 885
883 between zuul installations 886Shared Job Configs
887==================
888
889* Job config repos are all in git
890* Designed to support directly sharing job configurations
891* git.openstack.org/openstack-infra/zuul-jobs repo is a 'standard library'
892 to be directly shared between zuul installations
884 893
885Job 894Job
886=== 895===
@@ -897,10 +906,10 @@ Job
897 nodes: 906 nodes:
898 - name: primary 907 - name: primary
899 label: centos-7 908 label: centos-7
900 pre-run: playbooks/base/pre 909 pre-run: playbooks/base/pre.yaml
901 post-run: 910 post-run:
902 - playbooks/base/post-ssh 911 - playbooks/base/post-ssh.yaml
903 - playbooks/base/post-logs 912 - playbooks/base/post-logs.yaml
904 secrets: 913 secrets:
905 - site_logs 914 - site_logs
906 915
@@ -911,9 +920,9 @@ Simple Job
911 920
912 - job: 921 - job:
913 name: tox 922 name: tox
914 pre-run: playbooks/setup-tox 923 pre-run: playbooks/setup-tox.yaml
915 run: playbooks/tox 924 run: playbooks/tox.yaml
916 post-run: playbooks/fetch-tox-output 925 post-run: playbooks/fetch-tox-output.yaml
917 926
918 - job: 927 - job:
919 name: tox-py27 928 name: tox-py27
@@ -929,7 +938,7 @@ Simple Job Variant
929 938
930 - job: 939 - job:
931 name: tox-py27 940 name: tox-py27
932 branch: stable/mitaka 941 branches: stable/mitaka
933 nodeset: 942 nodeset:
934 - name: ubuntu-trusty 943 - name: ubuntu-trusty
935 label: ubuntu-trusty 944 label: ubuntu-trusty
@@ -968,7 +977,7 @@ Multi-node Job
968 - job: 977 - job:
969 name: ceph-multinode 978 name: ceph-multinode
970 nodeset: ceph-cluster 979 nodeset: ceph-cluster
971 run: playbooks/install-ceph 980 run: playbooks/install-ceph.yaml
972 981
973Multi-node Ceph Job Content 982Multi-node Ceph Job Content
974=========================== 983===========================
@@ -998,9 +1007,10 @@ Projects
998:: 1007::
999 1008
1000 zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d 1009 zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d
1001 src/git.openstack.org/openstack-infra/shade 1010 /home/zuul/src/git.openstack.org/openstack-infra/shade
1002 src/git.openstack.org/openstack/os-client-config 1011 /home/zuul/src/git.openstack.org/openstack/keystoneauth
1003 src/github.com/ansible/ansible 1012 /home/zuul/src/git.openstack.org/openstack/os-client-config
1013 /home/zuul/src/github.com/ansible/ansible
1004 1014
1005Project 1015Project
1006======= 1016=======
@@ -1008,8 +1018,6 @@ Project
1008.. code:: yaml 1018.. code:: yaml
1009 1019
1010 - project: 1020 - project:
1011 # Needing to name the project in that project's .zuul.yaml is going away
1012 name: openstack/nova
1013 check: 1021 check:
1014 jobs: 1022 jobs:
1015 - openstack-tox-py27 1023 - openstack-tox-py27
@@ -1022,7 +1030,6 @@ Project with Local Variant
1022.. code:: yaml 1030.. code:: yaml
1023 1031
1024 - project: 1032 - project:
1025 name: openstack/nova
1026 check: 1033 check:
1027 jobs: 1034 jobs:
1028 - openstack-tox-py27 1035 - openstack-tox-py27
@@ -1061,7 +1068,7 @@ Project with Many Local Variants
1061 - name: centos-7 1068 - name: centos-7
1062 label: centos-7 1069 label: centos-7
1063 - openstack-tox-py27 1070 - openstack-tox-py27
1064 branch: stable/newton 1071 branches: stable/newton
1065 nodeset: 1072 nodeset:
1066 - name: ubuntu-trusty 1073 - name: ubuntu-trusty
1067 label: ubuntu-trusty 1074 label: ubuntu-trusty
@@ -1085,7 +1092,6 @@ Project With Central and Local Config
1085 1092
1086 # In git.openstack.org/openstack/nova/.zuul.yaml: 1093 # In git.openstack.org/openstack/nova/.zuul.yaml:
1087 - project: 1094 - project:
1088 name: openstack/nova
1089 check: 1095 check:
1090 - nova-placement-functional-devstack 1096 - nova-placement-functional-devstack
1091 1097
@@ -1094,6 +1100,7 @@ Project with Job Dependencies
1094 1100
1095.. code:: yaml 1101.. code:: yaml
1096 1102
1103 # In git.openstack.org/openstack-infra/project-config:
1097 - project: 1104 - project:
1098 name: openstack/nova 1105 name: openstack/nova
1099 release: 1106 release:
@@ -1113,8 +1120,10 @@ Playbooks
1113 1120
1114* Jobs run Ansible playbooks 1121* Jobs run Ansible playbooks
1115* Playbooks may be defined centrally or in the repo being tested 1122* Playbooks may be defined centrally or in the repo being tested
1116* Playbooks can use roles from current or other Zuul repos (or Galaxy, coming soon) 1123* Playbooks can use roles from current or other Zuul repos
1117* Playbooks are run on the zuul-executor using bubblewrap https://github.com/projectatomic/bubblewrap 1124 (or Galaxy, coming soon)
1125* Playbooks are run on the zuul-executor using bubblewrap
1126 https://github.com/projectatomic/bubblewrap
1118* Playbooks are not allowed to execute content on 'localhost' 1127* Playbooks are not allowed to execute content on 'localhost'
1119 1128
1120Job with Roles 1129Job with Roles
@@ -1128,7 +1137,7 @@ Job with Roles
1128 Multi-node Zuul installation and integration test 1137 Multi-node Zuul installation and integration test
1129 nodeset: zuul-cluster 1138 nodeset: zuul-cluster
1130 roles: 1139 roles:
1131 - zuul: openstack/ansible-role-zuul 1140 - zuul: openstack-infra/ansible-role-zuul
1132 run: playbooks/zuul-integration 1141 run: playbooks/zuul-integration
1133 1142
1134Job with Multiple Projects 1143Job with Multiple Projects
@@ -1144,13 +1153,13 @@ Job with Multiple Projects
1144 vars: 1153 vars:
1145 zuul_work_dir: src/git.openstack.org/openstack-infra/zuul 1154 zuul_work_dir: src/git.openstack.org/openstack-infra/zuul
1146 required-projects: 1155 required-projects:
1147 - name: openstack-infra/zuul 1156 - openstack-infra/zuul
1148 override-branch: feature/zuulv3
1149 1157
1150 - project: openstack-infra/zuul-jobs 1158 - project:
1151 check: 1159 name: openstack-infra/zuul-jobs
1152 jobs: 1160 check:
1153 - tox-py35-on-zuul 1161 jobs:
1162 - tox-py35-on-zuul
1154 1163
1155Devstack-gate / Tempest Playbook 1164Devstack-gate / Tempest Playbook
1156================================ 1165================================
@@ -1218,10 +1227,10 @@ Secrets
1218* Jobs using secrets are not reconfigured speculatively 1227* Jobs using secrets are not reconfigured speculatively
1219* Secrets can only be used by the same project they are defined in 1228* Secrets can only be used by the same project they are defined in
1220* Public key per project: 1229* Public key per project:
1221 ``{{ zuul_url }}/{{ tenant }}/keys/{{ project }}.pub`` 1230 ``{{ zuul_url }}/{{ tenant }}/{{ project }}.pub``
1222 1231
1223:: 1232::
1224 GET http://zuulv3.openstack.org/keys/openstack-infra/shade.pub 1233 GET http://zuul.openstack.org/openstack-infra/shade.pub
1225 1234
1226Secret Example (note, no admins had to enable this) 1235Secret Example (note, no admins had to enable this)
1227=================================================== 1236===================================================
@@ -1283,35 +1292,36 @@ Secret Example
1283Status 1292Status
1284====== 1293======
1285 1294
1286* zuulv3 is running for OpenStack 1295* Zuul v3 is in production for OpenStack (in OpenStack VMs)
1287* zuulv3 also runing at BMW (in OpenShift) 1296* Zuul v3 also runing at BMW (in OpenShift) and Godaddy (in kuberenetes)
1288* automation job migration sucked (I wrote the script, whoops) 1297 and Huawei's OpenLab.
1289* cleaning up fixing automation and bugs found running at scale 1298* Software Factory updated to v3
1290* reference documentation exists and is complete 1299 https://softwarefactory-project.io/sf/welcome.html
1291* pre-repository job documentation 1300* will tag and release v3.0 once we're satisfied it's good for other people
1292* will release v3.0 once we're satisfied it's good for other people 1301 (within the next few weeks)
1293 1302
1294What's Next? 1303What's Next?
1295============ 1304============
1296 1305
1297* dashboard + REST API 1306* shared job doc generation
1298* user and deployer *documentation* - getting started walkthroughs 1307* node providers
1299* node providers: 1308 * kuberenetes
1300 * static
1301 * OCI/docker 1309 * OCI/docker
1302 * Mac Stadium (for our Ansible friends) 1310 * Mac Stadium (for our Ansible friends)
1303 * bifrost 1311 * ec2
1312 * ansible
1304* support for galaxy roles 1313* support for galaxy roles
1305* in-line code-review comments from Zuul 1314* in-line code-review comments from Zuul
1306* shared job doc generation 1315* native container/kubernetes job execution
1307* native container/kubernetes execution?
1308 1316
1309Important Links 1317Important Links
1310=============== 1318===============
1311 1319
1312* https://git.openstack.org/cgit/openstack-infra/zuul/log/?h=feature/zuulv3 1320* https://zuul-ci.org/
1313* https://docs.openstack.org/infra/zuul/feature/zuulv3/ 1321* https://git.openstack.org/cgit/openstack-infra/zuul
1322* https://docs.openstack.org/infra/zuul
1314* https://docs.openstack.org/infra/manual/zuulv3.html 1323* https://docs.openstack.org/infra/manual/zuulv3.html
1324* https://docs.openstack.org/infra/zuul-jobs/
1315* https://docs.openstack.org/infra/openstack-zuul-jobs/ 1325* https://docs.openstack.org/infra/openstack-zuul-jobs/
1316* https://storyboard.openstack.org/#!/project/679 1326* https://storyboard.openstack.org/#!/project/679
1317* https://storyboard.openstack.org/#!/board/41 1327* https://storyboard.openstack.org/#!/board/41