diff options
author | Monty Taylor <mordred@inaugust.com> | 2017-02-28 13:56:59 -0600 |
---|---|---|
committer | Monty Taylor <mordred@inaugust.com> | 2017-02-28 16:30:23 -0600 |
commit | 05df6770dbf8a881e26509ab0df8f10be9dd4281 (patch) | |
tree | 37bc7ab41c76a97ec2ddfd2525db88d2fd6fc4e3 /src/posts | |
parent | 9d3b791daef95dd51be2a7eaa7d2d0ca8396967e (diff) |
Add blog post about zuul v3
Diffstat (limited to 'src/posts')
-rw-r--r-- | src/posts/whats-coming-zuulv3.hbs | 296 |
1 files changed, 296 insertions, 0 deletions
diff --git a/src/posts/whats-coming-zuulv3.hbs b/src/posts/whats-coming-zuulv3.hbs new file mode 100644 index 0000000..b9b696e --- /dev/null +++ b/src/posts/whats-coming-zuulv3.hbs | |||
@@ -0,0 +1,296 @@ | |||
1 | <!doctype html> | ||
2 | <html> | ||
3 | <head> | ||
4 | <title>Zuul v3 - What's Coming</title> | ||
5 | <meta name="description" content="What to expect with the Zuul v3 Rollout" /> | ||
6 | </head> | ||
7 | <body> | ||
8 | |||
9 | <blockquote> | ||
10 | tl;dr - At last week's OpenStack PTG, the OpenStack Infra team ran the | ||
11 | first Zuul v3 job, so it's time to start getting everybody ready for what's | ||
12 | coming</blockquote> | ||
13 | |||
14 | <p> | ||
15 | <em class="text-danger">Don't Panic!</em> | ||
16 | Awesome changes are coming, but you are | ||
17 | <em>NOT</em> on the hook for rewriting all of your project's gate jobs or | ||
18 | anything crazy like that. Now grab a seat by the fire, pour yourself a | ||
19 | drink while I spin a yarn about days gone by and days yet to come. | ||
20 | </p> | ||
21 | |||
22 | <h3> First, some background </h3> | ||
23 | |||
24 | <p>The OpenStack Infra team has been hard at work for quite a while on a | ||
25 | new version of Zuul (where by 'quite some time' I mean that Jim Blair and | ||
26 | I had our first Zuul v3 design whiteboarding session in 2014). As you might | ||
27 | be able to guess given the amount of time, there are some big things | ||
28 | coming that will have a real and visible impact on the OpenStack community | ||
29 | and beyond. Since we have a running Zuul v3 now<sup> | ||
30 | <a href="#fn1">[1]</a></sup>, it seemed like the time | ||
31 | to start getting folks up to speed on what to expect.</p> | ||
32 | |||
33 | <p>There is other deep-dive information on architecture and rationale if | ||
34 | you're interested<sup><a href="#fn2">[2]</a></sup>, but for now we'll | ||
35 | focus on what's relevant for end users. We're also going to start sending | ||
36 | out a bi-weekly "Status of Zuul v3" email to the | ||
37 | <a href='http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev'> | ||
38 | openstack-dev@lists.openstack.org | ||
39 | </a> mailing list ... so stay tuned! | ||
40 | </p> | ||
41 | |||
42 | <p><em>Important Note</em> This post includes some code snippets - but v3 | ||
43 | is still a work in progress. We know of at least one breaking change that | ||
44 | is coming to the config format, so please treat this not as a tutorial, but | ||
45 | as a conceptual overview. Syntax is subject to change.</p> | ||
46 | |||
47 | <h3> The Big Ticket Items </h3> | ||
48 | |||
49 | <p>While there are a bunch of changes behind the scenes, there are a | ||
50 | reasonably tractable number of user-facing differences. | ||
51 | |||
52 | <ul> | ||
53 | <li> Self-testing In-Repo Job Config </li> | ||
54 | <li> Ansible Job Content </li> | ||
55 | <li> First-class Multi-node Jobs </li> | ||
56 | <li> Improved Job Reuse </li> | ||
57 | <li> Support for non-OpenStack Code and Node Systems </li> | ||
58 | <li> and Much, Much More </li> | ||
59 | </ul> | ||
60 | |||
61 | <h3> Self-testing In-Repo Job Config </h3> | ||
62 | |||
63 | <p>This is probably the biggest deal. There are a lot of OpenStack Devs | ||
64 | (around 2k in Ocata) and a lot of repositories (1689) There a lot fewer | ||
65 | folks on the project-config-core team who are the ones who review all of | ||
66 | the job config changes (please everyone thank Andreas Jaeger next time | ||
67 | you see him). That's not awesome.</p> | ||
68 | |||
69 | <p>Self-testing in-repo job config is awesome.</p> | ||
70 | |||
71 | <p>Many systems out there these days have an in-repo job config system. | ||
72 | Travis CI has had it since day one, and Jenkins has recently added support | ||
73 | for a Jenkinsfile inside of git repos. With Zuul v3, we'll have it too.</p> | ||
74 | |||
75 | <p>Once we roll out v3 to everyone, as a supplement to jobs defined in our | ||
76 | central config repositories, each project will be able to add a zuul.yaml | ||
77 | file to their own repo:</p> | ||
78 | |||
79 | <pre><code> | ||
80 | - job: | ||
81 | name: my_awesome_job | ||
82 | nodes: | ||
83 | - name: controller | ||
84 | label: centos-7 | ||
85 | |||
86 | - project: | ||
87 | name: openstack/awesome_project | ||
88 | check: | ||
89 | jobs: | ||
90 | - my_awesome_job | ||
91 | </code></pre> | ||
92 | |||
93 | <p>It's a small file, but there is a lot going on, so let's unpack it.</p> | ||
94 | |||
95 | <p>First we define a job to run. It's named <em>my_awesome_job</em> and it | ||
96 | needs one node. That node will be named <em>controller</em> and will be | ||
97 | based on the <em>centos-7</em> base node in nodepool.</p> | ||
98 | |||
99 | <p>In the next section, we say that we want to run that job in the | ||
100 | <em>check</em> pipeline, which in OpenStack is defined as the jobs | ||
101 | that run when patchsets are proposed.</p> | ||
102 | |||
103 | <p>And it's also self-testing!</p> | ||
104 | |||
105 | <p>Everyone knows the fun game of writing a patch to the test jobs, | ||
106 | getting it approved, then hoping it works once it starts running. With | ||
107 | Zuul v3 in-repo jobs, if there is a change to job definitions in a | ||
108 | proposed patch, that patch will be tested with those changes applied. And | ||
109 | since it's Zuul, Depends-On footers are honored as well - so iteration | ||
110 | on getting a test job right becomes just like iterating on any other | ||
111 | patch or sequence of patches.</p> | ||
112 | |||
113 | <h3> Ansible Job Content </h3> | ||
114 | |||
115 | <p>The job <em>my_awesome_job</em> isn't very useful if it doesn't define | ||
116 | any content. That's done in the repo as well, in | ||
117 | playbooks/my_awesome_job.yaml:</p> | ||
118 | |||
119 | <pre><code> | ||
120 | - hosts: controller | ||
121 | tasks: | ||
122 | - name: Run make tests | ||
123 | shell: make distcheck | ||
124 | </code></pre> | ||
125 | |||
126 | <p>As previously mentioned, the job content is now defined in Ansible | ||
127 | rather than using our Jenkins Job Builder tool. This playbook is going | ||
128 | to run a tasks on a host called <em>controller</em> which you may | ||
129 | remember we requested in the job definition. On that host, it will run | ||
130 | <em>make distcheck</em>. Pretty much anything you can do in Ansible, you | ||
131 | can do in a Zuul job now, and the playbooks should also be re-usable | ||
132 | outside of a testing context.</p> | ||
133 | |||
134 | <h3> First Class Multi-Node Jobs </h3> | ||
135 | |||
136 | <p>The previous example was for running a job on a node. What if you | ||
137 | want to do multi-node?</p> | ||
138 | |||
139 | <pre><code> | ||
140 | - job: | ||
141 | name: my_awesome_job | ||
142 | nodes: | ||
143 | - name: controller | ||
144 | label: ubuntu-xenial | ||
145 | - name: compute | ||
146 | label: centos-7 | ||
147 | |||
148 | - project: | ||
149 | name: openstack/awesome_project | ||
150 | check: | ||
151 | jobs: | ||
152 | - my_awesome_job | ||
153 | </code></pre> | ||
154 | |||
155 | <p>As you may have surmised, nodes is a list, so you can have more than | ||
156 | one. Then, since Ansible is naturally mutli-node aware, you use that to | ||
157 | write the multi-node content:</p> | ||
158 | |||
159 | <pre><code> | ||
160 | - hosts: controller | ||
161 | tasks: | ||
162 | - name: Install Keystone | ||
163 | shell: pip install {{ zuul.git_root }}/openstack/keystone | ||
164 | - hosts: compute | ||
165 | tasks: | ||
166 | - name: Install Nova | ||
167 | shell: pip install {{ zuul.git_root }}/openstack/nova | ||
168 | - hosts: * | ||
169 | tasks: | ||
170 | - name: Install CloudKitty | ||
171 | shell: pip install {{ zuul.git_root }}/openstack/cloudkitty | ||
172 | </code></pre> | ||
173 | |||
174 | <p>That will install Keystone on <em>controller</em>, Nova on | ||
175 | <em>compute</em> and CloudKitty on both.</p> | ||
176 | |||
177 | <h3> Improved Job Reuse </h3> | ||
178 | |||
179 | <p>In our current system, because of some details about how Jenkins works | ||
180 | and the fact that our CI system used to be based on Jenkins, we have a ton | ||
181 | of templated jobs that lead both to magically long job names and a bunch | ||
182 | of cargo culting of job definitions.</p> | ||
183 | |||
184 | <p>In the new system, a lot of the duplication goes away. So instead of | ||
185 | having gate-nova-python27 and gate-swift-python27 and | ||
186 | gate-keystone-python27, there will just be a job called "python27" and each | ||
187 | of the projects can use it. Similarly, for more complex job content like | ||
188 | devstack-gate, since Ansible is a fully-fledged system on its own that | ||
189 | was designed for modularity and re-use, we can compose things into roles | ||
190 | that take parameters and can be reused without copy/paste.</p> | ||
191 | |||
192 | <p class='text-muted'><small> | ||
193 | ssssh! In fact, the python27 job will almost certainly be a job that | ||
194 | uses an extremely small playbook that itself uses a role called tox. But | ||
195 | also, the tox role, the python27 playbook and the python27 job definition | ||
196 | will all be things we define centrally in a standard library of pieces, | ||
197 | so as a user of the system you should be able to just choose to run | ||
198 | "python27" and not worry about it - unless you want to dig in and learn | ||
199 | more.</small></p> | ||
200 | |||
201 | <h3> Support for non-OpenStack Code and Node Systems </h3> | ||
202 | |||
203 | <p>Zuul was originally written to support the OpenStack project, but since | ||
204 | then we've grown more people who have interest in running Zuul. Since we | ||
205 | wrote it the first time to solve our problems of extremely massive scale, | ||
206 | we didn't put a ton of effort into making it easily consumable elsewhere. | ||
207 | That hasn't stopped people, and there are tons of Zuul installations out | ||
208 | there ... but that doesn't mean life is easy for those folks. With Zuul v3 | ||
209 | we've also been explicitly focused on making it much more easily | ||
210 | reconsumable.</p> | ||
211 | |||
212 | <p>Part of supporting friends in other communities means embracing support | ||
213 | of tools that OpenStack does not use. The fine folks at Gooddata wrote a | ||
214 | set of patches adding support for Github which they have been using for a | ||
215 | while. We'll be landing those, which should allow us to add jobs to the | ||
216 | system that check things like "will this pull request to pip break | ||
217 | devstack". There is also work from the CentOS community via a tool called | ||
218 | linch-pin that we're looking at incorporating into Nodepool that should | ||
219 | allow creating build nodes on any system Ansible knows how to talk to. | ||
220 | Those features are intended to follow quickly on after we get OpenStack | ||
221 | migrated.</p> | ||
222 | |||
223 | <h3> What's Next? </h3> | ||
224 | |||
225 | <ol> | ||
226 | <li>Zuul on Zuul</li> | ||
227 | <li>Infra Repos</li> | ||
228 | <li>Job conversion Script</li> | ||
229 | <li>OpenStack Migration</li> | ||
230 | </ol> | ||
231 | |||
232 | <p>We currently have a Zuul v3 running against changes to the Zuul repo. | ||
233 | We're using to iterate on job content and other features. There is a | ||
234 | change coming to the job definition syntax to allow job dependencies to | ||
235 | be a graph instead of just a tree which will be fairly invasive, so we're | ||
236 | keeping the affected surface area small until that's ready.</p> | ||
237 | |||
238 | <p>Once we're happy with how things are running, we'll move the rest of | ||
239 | the Infra repos over, probably in chunks. Although Infra test jobs are | ||
240 | typically a bit different than the jobs in the rest of OpenStack, we do | ||
241 | have enough representative examples that we should be able to work out | ||
242 | the kinks before we throw things at other folks. (shade and nodepool both | ||
243 | do integration testing on devstack-gate jobs, for instance)</p> | ||
244 | |||
245 | <p>While we work on Infra migration, we'll be developing a conversion | ||
246 | script to convert the existing jobs. A good portion of that will be fully | ||
247 | automatable. For instance, mapping everyone's gate-{project}-python27 to | ||
248 | a reference to the python27 job is easy for a computer to do. However, | ||
249 | there's still a ton of snowflake jobs that we'll likely wind up just | ||
250 | converting the content of as is and then iterating on refactoring to be | ||
251 | more efficient or improved over time.</p> | ||
252 | |||
253 | <p>Then the Big Day will arrive. When the conversion script is as good as | ||
254 | we can get it and we're satisfied with stability of the job language, | ||
255 | security and scalability, there will be a Big Bang cutover of all of the | ||
256 | rest of OpenStack. If all goes well, most developers should mostly just | ||
257 | notice that a bunch of job names got shorter and that it's a user named | ||
258 | Zuul commenting on patches. Folks who have patches to project-config | ||
259 | in-flight at that time will need to rework patches, but the conversion | ||
260 | script should hopefully make that a minimal burden.</p> | ||
261 | |||
262 | <h3> and Much, Much More </h3> | ||
263 | |||
264 | <p>There are far too many new and exciting things in Zuul v3 to cover in | ||
265 | a single post, and many of the topics (such as Ansible Jobs, or Job | ||
266 | Inheritance and Reuse) are deep topics we can dive in to over time. The | ||
267 | long and short of it is that Zuul v3 is coming soon to an OpenStack Infra | ||
268 | near you, so expect more and more communication about what that means | ||
269 | over the next few months.</p> | ||
270 | |||
271 | <h4> Notes </h4> | ||
272 | <ol> | ||
273 | <li id="fn1">OpenStack is not running Zuul v3 in production at the | ||
274 | moment. We have an instance running and only responding to events from | ||
275 | the Zuul v3 repo. As of the time of this writing, OpenStack is still | ||
276 | running 2.5 in production. Believe me - when we hit production, you'll | ||
277 | know it.</li> | ||
278 | <li id="fn2"> Links to deeper information: | ||
279 | <ul> | ||
280 | <li><a href="http://amo-probos.org/post/18"> | ||
281 | "There is no Jenkins, only Zuul" post about Zuul 2.5</a></li> | ||
282 | <li><a href="https://www.youtube.com/watch?v=R4EmE1QEvNU"> | ||
283 | Jim Blair's Zuul v3 Talk from OpenStack Summit Barcelona</a></li> | ||
284 | <li> | ||
285 | <a href="https://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html"> | ||
286 | Zuul v3 Spec</a></li> | ||
287 | <li> | ||
288 | <a href="http://lists.openstack.org/pipermail/openstack-infra/2017-February/005131.html"> | ||
289 | Pre-PTG Zuul v3 Conceptual Deep Dive | ||
290 | </a></li> | ||
291 | </ul> | ||
292 | </li> | ||
293 | </ol> | ||
294 | |||
295 | </body> | ||
296 | </html> | ||