summaryrefslogtreecommitdiff
path: root/src
diff options
context:
space:
mode:
Diffstat (limited to 'src')
-rw-r--r--src/posts/whats-coming-zuulv3.hbs296
1 files changed, 296 insertions, 0 deletions
diff --git a/src/posts/whats-coming-zuulv3.hbs b/src/posts/whats-coming-zuulv3.hbs
new file mode 100644
index 0000000..b9b696e
--- /dev/null
+++ b/src/posts/whats-coming-zuulv3.hbs
@@ -0,0 +1,296 @@
1<!doctype html>
2<html>
3 <head>
4 <title>Zuul v3 - What's Coming</title>
5 <meta name="description" content="What to expect with the Zuul v3 Rollout" />
6 </head>
7 <body>
8
9 <blockquote>
10 tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the
11 first Zuul v3 job, so it's time to start getting everybody ready for what's
12 coming</blockquote>
13
14 <p>
15 <em class="text-danger">Don't Panic!</em>
16 Awesome changes are coming, but you are
17 <em>NOT</em> on the hook for rewriting all of your project's gate jobs or
18 anything crazy like that. Now grab a seat by the fire, pour yourself a
19 drink while I spin a yarn about days gone by and days yet to come.
20 </p>
21
22 <h3> First, some background </h3>
23
24 <p>The OpenStack Infra team has been hard at work for quite a while on a
25 new version of Zuul (where by 'quite some time' I mean that Jim Blair and
26 I had our first Zuul v3 design whiteboarding session in 2014). As you might
27 be able to guess given the amount of time, there are some big things
28 coming that will have a real and visible impact on the OpenStack community
29 and beyond. Since we have a running Zuul v3 now<sup>
30 <a href="#fn1">[1]</a></sup>, it seemed like the time
31 to start getting folks up to speed on what to expect.</p>
32
33 <p>There is other deep-dive information on architecture and rationale if
34 you're interested<sup><a href="#fn2">[2]</a></sup>, but for now we'll
35 focus on what's relevant for end users. We're also going to start sending
36 out a bi-weekly "Status of Zuul v3" email to the
37 <a href='http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev'>
38 openstack-dev@lists.openstack.org
39 </a> mailing list ... so stay tuned!
40 </p>
41
42 <p><em>Important Note</em> This post includes some code snippets - but v3
43 is still a work in progress. We know of at least one breaking change that
44 is coming to the config format, so please treat this not as a tutorial, but
45 as a conceptual overview. Syntax is subject to change.</p>
46
47 <h3> The Big Ticket Items </h3>
48
49 <p>While there are a bunch of changes behind the scenes, there are a
50 reasonably tractable number of user-facing differences.
51
52 <ul>
53 <li> Self-testing In-Repo Job Config </li>
54 <li> Ansible Job Content </li>
55 <li> First-class Multi-node Jobs </li>
56 <li> Improved Job Reuse </li>
57 <li> Support for non-OpenStack Code and Node Systems </li>
58 <li> and Much, Much More </li>
59 </ul>
60
61 <h3> Self-testing In-Repo Job Config </h3>
62
63 <p>This is probably the biggest deal. There are a lot of OpenStack Devs
64 (around 2k in Ocata) and a lot of repositories (1689) There a lot fewer
65 folks on the project-config-core team who are the ones who review all of
66 the job config changes (please everyone thank Andreas Jaeger next time
67 you see him). That's not awesome.</p>
68
69 <p>Self-testing in-repo job config is awesome.</p>
70
71 <p>Many systems out there these days have an in-repo job config system.
72 Travis CI has had it since day one, and Jenkins has recently added support
73 for a Jenkinsfile inside of git repos. With Zuul v3, we'll have it too.</p>
74
75 <p>Once we roll out v3 to everyone, as a supplement to jobs defined in our
76 central config repositories, each project will be able to add a zuul.yaml
77 file to their own repo:</p>
78
79 <pre><code>
80- job:
81 name: my_awesome_job
82 nodes:
83 - name: controller
84 label: centos-7
85
86- project:
87 name: openstack/awesome_project
88 check:
89 jobs:
90 - my_awesome_job
91 </code></pre>
92
93 <p>It's a small file, but there is a lot going on, so let's unpack it.</p>
94
95 <p>First we define a job to run. It's named <em>my_awesome_job</em> and it
96 needs one node. That node will be named <em>controller</em> and will be
97 based on the <em>centos-7</em> base node in nodepool.</p>
98
99 <p>In the next section, we say that we want to run that job in the
100 <em>check</em> pipeline, which in OpenStack is defined as the jobs
101 that run when patchsets are proposed.</p>
102
103 <p>And it's also self-testing!</p>
104
105 <p>Everyone knows the fun game of writing a patch to the test jobs,
106 getting it approved, then hoping it works once it starts running. With
107 Zuul v3 in-repo jobs, if there is a change to job definitions in a
108 proposed patch, that patch will be tested with those changes applied. And
109 since it's Zuul, Depends-On footers are honored as well - so iteration
110 on getting a test job right becomes just like iterating on any other
111 patch or sequence of patches.</p>
112
113 <h3> Ansible Job Content </h3>
114
115 <p>The job <em>my_awesome_job</em> isn't very useful if it doesn't define
116 any content. That's done in the repo as well, in
117 playbooks/my_awesome_job.yaml:</p>
118
119 <pre><code>
120- hosts: controller
121 tasks:
122 - name: Run make tests
123 shell: make distcheck
124 </code></pre>
125
126 <p>As previously mentioned, the job content is now defined in Ansible
127 rather than using our Jenkins Job Builder tool. This playbook is going
128 to run a tasks on a host called <em>controller</em> which you may
129 remember we requested in the job definition. On that host, it will run
130 <em>make distcheck</em>. Pretty much anything you can do in Ansible, you
131 can do in a Zuul job now, and the playbooks should also be re-usable
132 outside of a testing context.</p>
133
134 <h3> First Class Multi-Node Jobs </h3>
135
136 <p>The previous example was for running a job on a node. What if you
137 want to do multi-node?</p>
138
139 <pre><code>
140- job:
141 name: my_awesome_job
142 nodes:
143 - name: controller
144 label: ubuntu-xenial
145 - name: compute
146 label: centos-7
147
148- project:
149 name: openstack/awesome_project
150 check:
151 jobs:
152 - my_awesome_job
153 </code></pre>
154
155 <p>As you may have surmised, nodes is a list, so you can have more than
156 one. Then, since Ansible is naturally mutli-node aware, you use that to
157 write the multi-node content:</p>
158
159 <pre><code>
160- hosts: controller
161 tasks:
162 - name: Install Keystone
163 shell: pip install {{ zuul.git_root }}/openstack/keystone
164- hosts: compute
165 tasks:
166 - name: Install Nova
167 shell: pip install {{ zuul.git_root }}/openstack/nova
168- hosts: *
169 tasks:
170 - name: Install CloudKitty
171 shell: pip install {{ zuul.git_root }}/openstack/cloudkitty
172 </code></pre>
173
174 <p>That will install Keystone on <em>controller</em>, Nova on
175 <em>compute</em> and CloudKitty on both.</p>
176
177 <h3> Improved Job Reuse </h3>
178
179 <p>In our current system, because of some details about how Jenkins works
180 and the fact that our CI system used to be based on Jenkins, we have a ton
181 of templated jobs that lead both to magically long job names and a bunch
182 of cargo culting of job definitions.</p>
183
184 <p>In the new system, a lot of the duplication goes away. So instead of
185 having gate-nova-python27 and gate-swift-python27 and
186 gate-keystone-python27, there will just be a job called "python27" and each
187 of the projects can use it. Similarly, for more complex job content like
188 devstack-gate, since Ansible is a fully-fledged system on its own that
189 was designed for modularity and re-use, we can compose things into roles
190 that take parameters and can be reused without copy/paste.</p>
191
192 <p class='text-muted'><small>
193 ssssh! In fact, the python27 job will almost certainly be a job that
194 uses an extremely small playbook that itself uses a role called tox. But
195 also, the tox role, the python27 playbook and the python27 job definition
196 will all be things we define centrally in a standard library of pieces,
197 so as a user of the system you should be able to just choose to run
198 "python27" and not worry about it - unless you want to dig in and learn
199 more.</small></p>
200
201 <h3> Support for non-OpenStack Code and Node Systems </h3>
202
203 <p>Zuul was originally written to support the OpenStack project, but since
204 then we've grown more people who have interest in running Zuul. Since we
205 wrote it the first time to solve our problems of extremely massive scale,
206 we didn't put a ton of effort into making it easily consumable elsewhere.
207 That hasn't stopped people, and there are tons of Zuul installations out
208 there ... but that doesn't mean life is easy for those folks. With Zuul v3
209 we've also been explicitly focused on making it much more easily
210 reconsumable.</p>
211
212 <p>Part of supporting friends in other communities means embracing support
213 of tools that OpenStack does not use. The fine folks at Gooddata wrote a
214 set of patches adding support for Github which they have been using for a
215 while. We'll be landing those, which should allow us to add jobs to the
216 system that check things like "will this pull request to pip break
217 devstack". There is also work from the CentOS community via a tool called
218 linch-pin that we're looking at incorporating into Nodepool that should
219 allow creating build nodes on any system Ansible knows how to talk to.
220 Those features are intended to follow quickly on after we get OpenStack
221 migrated.</p>
222
223 <h3> What's Next? </h3>
224
225 <ol>
226 <li>Zuul on Zuul</li>
227 <li>Infra Repos</li>
228 <li>Job conversion Script</li>
229 <li>OpenStack Migration</li>
230 </ol>
231
232 <p>We currently have a Zuul v3 running against changes to the Zuul repo.
233 We're using to iterate on job content and other features. There is a
234 change coming to the job definition syntax to allow job dependencies to
235 be a graph instead of just a tree which will be fairly invasive, so we're
236 keeping the affected surface area small until that's ready.</p>
237
238 <p>Once we're happy with how things are running, we'll move the rest of
239 the Infra repos over, probably in chunks. Although Infra test jobs are
240 typically a bit different than the jobs in the rest of OpenStack, we do
241 have enough representative examples that we should be able to work out
242 the kinks before we throw things at other folks. (shade and nodepool both
243 do integration testing on devstack-gate jobs, for instance)</p>
244
245 <p>While we work on Infra migration, we'll be developing a conversion
246 script to convert the existing jobs. A good portion of that will be fully
247 automatable. For instance, mapping everyone's gate-{project}-python27 to
248 a reference to the python27 job is easy for a computer to do. However,
249 there's still a ton of snowflake jobs that we'll likely wind up just
250 converting the content of as is and then iterating on refactoring to be
251 more efficient or improved over time.</p>
252
253 <p>Then the Big Day will arrive. When the conversion script is as good as
254 we can get it and we're satisfied with stability of the job language,
255 security and scalability, there will be a Big Bang cutover of all of the
256 rest of OpenStack. If all goes well, most developers should mostly just
257 notice that a bunch of job names got shorter and that it's a user named
258 Zuul commenting on patches. Folks who have patches to project-config
259 in-flight at that time will need to rework patches, but the conversion
260 script should hopefully make that a minimal burden.</p>
261
262 <h3> and Much, Much More </h3>
263
264 <p>There are far too many new and exciting things in Zuul v3 to cover in
265 a single post, and many of the topics (such as Ansible Jobs, or Job
266 Inheritance and Reuse) are deep topics we can dive in to over time. The
267 long and short of it is that Zuul v3 is coming soon to an OpenStack Infra
268 near you, so expect more and more communication about what that means
269 over the next few months.</p>
270
271 <h4> Notes </h4>
272 <ol>
273 <li id="fn1">OpenStack is not running Zuul v3 in production at the
274 moment. We have an instance running and only responding to events from
275 the Zuul v3 repo. As of the time of this writing, OpenStack is still
276 running 2.5 in production. Believe me - when we hit production, you'll
277 know it.</li>
278 <li id="fn2"> Links to deeper information:
279 <ul>
280 <li><a href="http://amo-probos.org/post/18">
281 "There is no Jenkins, only Zuul" post about Zuul 2.5</a></li>
282 <li><a href="https://www.youtube.com/watch?v=R4EmE1QEvNU">
283 Jim Blair's Zuul v3 Talk from OpenStack Summit Barcelona</a></li>
284 <li>
285 <a href="https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html">
286 Zuul v3 Spec</a></li>
287 <li>
288 <a href="http://lists.openstack.org/pipermail/openstack-infra/2017-February/005131.html">
289 Pre-PTG Zuul v3 Conceptual Deep Dive
290 </a></li>
291 </ul>
292 </li>
293 </ol>
294
295</body>
296</html>
generated by cgit v1.2.3 (git 2.25.1) at 2025-10-28 05:10:07 +0000