summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorMonty Taylor <mordred@inaugust.com>2016-10-06 07:13:44 +0200
committerMonty Taylor <mordred@inaugust.com>2016-10-06 07:47:33 +0200
commit46d929fad9cd6363b6ce8d3f440183fad96ae95d (patch)
treebc73abab61c9570af19a9cadf801aa02c47c4c55
parente2f00edda0c8a768879682da23f8a62c1f00bf36 (diff)
Add talk for LinuxConf Europe
Diffstat
-rw-r--r--gulpfile.js1
-rw-r--r--src/talks/test-it-like-you-deploy-it.hbs346
2 files changed, 347 insertions, 0 deletions
diff --git a/gulpfile.js b/gulpfile.js
index d6283f0..c4f5d42 100644
--- a/gulpfile.js
+++ b/gulpfile.js
@@ -400,6 +400,7 @@
400 gulp.watch(paths.html, ['package:html']); 400 gulp.watch(paths.html, ['package:html']);
401 gulp.watch(paths.images, ['package:images']); 401 gulp.watch(paths.images, ['package:images']);
402 gulp.watch(paths.talks, ['package:talks']); 402 gulp.watch(paths.talks, ['package:talks']);
403 gulp.watch(dir.src + '/talks/*.hbs', ['package:talks']);
403 gulp.watch(paths.css, ['package:css']); 404 gulp.watch(paths.css, ['package:css']);
404 gulp.watch(paths.js, ['package:js']); 405 gulp.watch(paths.js, ['package:js']);
405 gulp.watch(paths.posts, ['package:posts']); 406 gulp.watch(paths.posts, ['package:posts']);
diff --git a/src/talks/test-it-like-you-deploy-it.hbs b/src/talks/test-it-like-you-deploy-it.hbs
new file mode 100644
index 0000000..808cad8
--- /dev/null
+++ b/src/talks/test-it-like-you-deploy-it.hbs
@@ -0,0 +1,346 @@
1<!doctype html>
2<html lang="en">
3
4 <head>
5 <meta charset="utf-8">
6
7 <title>Test it like you Deploy It</title>
8
9 </head>
10 <body>
11
12 <section id="who-am-i-redhat" class="slide level2">
13 <h1>Who am I?</h1>
14 <img style="float:right; margin:24pt" src="/images/Logo_RH_CMYK_Default.jpg" />
15 <p> Office of Technology </p>
16 <p> Zuul </p>
17 <p> Ansible </p>
18 </section>
19
20 <section id="who-am-i-openstack" class="slide level2">
21 <h1>Who am I?</h1>
22 <img style="float:right; margin-right:24pt; width:300px; height: auto" src="/images/openstack-cloud-software-vertical-large.png" />
23 <p>Technical Committee</p>
24 <p>Foundation Board of Directors</p>
25 <p>Developer Infrastructure Core Team</p>
26 </section>
27
28 <section id="what-are-we-going-to-talk-about" class="slide level2">
29 <h1>What are we going to talk about?</h1>
30 <ul>
31 <li>Zuul</li>
32 <li>Ansible</li>
33 </ul>
34 </section>
35
36 <section id="what-is-zuul" class="slide level2">
37 <h1>What is Zuul?</h1>
38 <ul>
39 <li>Multi-cloud, scalable, elastic CI/CD engine</li>
40 <li>Validation of speculative future states</li>
41 <li>Single-use VM build nodes - safely run tests that need root</li>
42 <li>Multi-node builds</li>
43 <li>Multi-repo projects</li>
44 <li>Native support for gating configuration</li>
45 </ul>
46 </section>
47
48 <section id="terminoloy" class="slide level2">
49 <h1>Terminology</h1>
50 <ul>
51 <li>Periodic: jobs run in response to a timer</li>
52 <li>Post: jobs run after a change</li>
53 <li>Check: job run when someone proposes a change</li>
54 <li>Gate: jobs run between change approval and landing</li>
55 </ul>
56 </section>
57
58 <section id="why" class="slide level2">
59 <h1>Why?</h1>
60 <h2> The original OpenStack use case</h2>
61 <ul>
62 <li>Fully automated gated commits</li>
63 <li>Full end-to-end integration tests from scratch for every commit</li>
64 <li>Massive scale</li>
65 </ul>
66 </section>
67
68 <section id="openstack-scale" class="slide level2">
69 <h1> OpenStack Scale by the numbers</h1>
70 <ul>
71 <li>2 KJPH (kilo-jobs / hour)</li>
72 <li>2376 arbitrary developers</li>
73 <li>1474 git repositories</li>
74 <li>11727 Jobs</li>
75 <li>Merge 10k Changes / Month</li>
76 </ul>
77 <p class='fragment'>ansible has <em>received</em> 13171 PRs (changes),
78 has merged 8190 of them and has 37788 commits in its entire lifetime</p>
79 </section>
80
81 <section id="speculative-execution" class="slide level2">
82 <h1>Multi Repository Speculative Execution</h1>
83 <ul>
84 <li>Zuul constructs speculative states as-if a change were merged</li>
85 <li>Tests future states without landing those changes first</li>
86 <li>The as-if spans multiple repos</li>
87 <li>In the Gate pipeline, speculative changes are put into a
88 virtual serial queue, then tested in parallel as-if each change
89 combination in front of them had landed</li>
90 </ul>
91 </section>
92
93 <section id="animation" class="slide level2">
94 <h1>Zuul Animation</h1>
95 <a href='http://docs.openstack.org/infra/publications/zuul/#(18)'>http://docs.openstack.org/infra/publications/zuul/#(18)</a>
96 </section>
97
98 <section id="depends-on" class="slide level2">
99 <h1>Multi-Repo Dependencies</h1>
100 <pre><code>
101commit 30039f04109efa2263aba6eb302a29bd8d5e8f53
102Author: Monty Taylor
103Date: Mon Sep 26 14:14:15 2016 -0500
104
105 Add simple field for disabled flavors
106
107 When we were equalizing out the old silly names from the new pretty
108 ones, we missed disabled.
109
110 Change-Id: I4cbf5f7c27f640c566460c18951ab9030aae84e4
111 Depends-On: I523e0ab6e376f5ff6205b1cc1748aa6d546919cb
112 </code></pre>
113 </section>
114
115 <section id="depends-on" class="slide level2">
116 <h1>Multi-Repo Dependencies - an example</h1>
117 <ul>
118 <li>shade library runs functional tests against openstack</li>
119 <li>neutron change breaks shade tests (more later)</li>
120 <li>neutron fix is proposed</li>
121 <li>shade change Depends-On proposed neutron change</li>
122 <li>shade tests are run as-if neutron change has landed</li>
123 <li>shade change cannot land until neutron change lands</li>
124 </ul>
125 </section>
126
127 <section id="depends-on" class="slide level2">
128 <h1>Isn't this supposed to prevent neutron from breaking shade?</h1>
129 <ul>
130 <li>shade and neutron do not share a gating relationship</li>
131 <li>shade has, by choice, a test that tests against master of OpenStack</li>
132 <li>Such a test is 'risky' to shade devs</li>
133 <li>shade devs desire the risk - can work around breaks, or submit bugs</li>
134 </ul>
135 </section>
136
137 <section id="not-specific" class="slide level2">
138 <h1>Not Specific to OpenStack</h1>
139 <ul>
140 <li>"Gate" and "Check" are merely configurations</li>
141 <li>50+ OpenStack Vendors use Zuul for "3rd Party CI" of drivers</li>
142 <li>Wikimedia uses Zuul</li>
143 </ul>
144 </section>
145
146 <section id="status-pages" class="slide level2">
147 <h1>Status Pages</h1>
148 <p><a href="https://integration.wikimedia.org/zuul/">
149 https://integration.wikimedia.org/zuul/</a></p>
150 <p><a href="http://status.openstack.org/zuul/">
151 http://status.openstack.org/zuul/</a></p>
152 </section>
153
154 <section id="pluggable" class="slide level2">
155 <h1>Pluggable</h1>
156 <ul>
157 <li>Triggers</li>
158 <li>Reporters</li>
159 <li>Node Providers</li>
160 <li>Execution content</li>
161 </ul>
162 </section>
163
164 <section id="zuul-v2" class="slide level2">
165 <h1>Zuul v2</h1>
166 <ul>
167 <li>In production for OpenStack for 4 years</li>
168 <li>What most people run</li>
169 <li>Triggers: Gerrit, Periodic</li>
170 <li>Reporters: Gerrit, Email, MySQL</li>
171 <li>Node Providers: Elastic OpenStack Nodepool, Static servers</li>
172 <li>Jobs executed by Jenkins</li>
173 </ul>
174 </section>
175
176 <section id="v2-architecture" class="slide level2">
177 <p><img src="/images/infra_architecture.jpg" alt="image" /></p>
178 </section>
179
180 <section id="zuul-v2.5" class="slide level2">
181 <h1>Zuul v2.5</h1>
182 <ul>
183 <li>In use only by OpenStack (on purpose)</li>
184 <li>Replaced Jenkins with Ansible</li>
185 <li>Jobs still written using JJB - playbooks generated on the fly</li>
186 </ul>
187 <p><a href='http://logs.openstack.org/09/352209/7/check/gate-networking-ovn-python35/0fae86c/_zuul_ansible/'>
188 http://logs.openstack.org/09/352209/7/check/gate-networking-ovn-python35/0fae86c/_zuul_ansible/
189 </a></p>
190 </section>
191
192 <section id="replace-jenkins" class="slide level2">
193 <h1>Why Replace Jenkins?</h1>
194 </section>
195
196 <section id="no-lack-of-trying" class="slide level2">
197 <h1>Not for lack of trying</h1>
198 <ul>
199 <li>OpenStack started on Jenkins (actually, on Hudson, remember that?)</li>
200 <li>We funded the Jenkins JClouds Plugin</li>
201 <li>We did deep dev in the Gerrit Trigger Plugin</li>
202 <li>Maintain the SCP artifact plugin (added console log support)</li>
203 <li>Added 0mq notification plugin</li>
204 <li>Added Gearman Worker plugin - allowed us to grow to 8 Masters/1000 slaves</li>
205 <li>We wrote Jenkins Job Builder</li>
206 </ul>
207 </section>
208
209 <section id="problems" class="slide level2">
210 <h1>Before I answer that ...</h1>
211 <p>The world is a better place with Jenkins existing</p>
212 </section>
213
214 <section id="problems" class="slide level2">
215 <h1>Jenkins Problems</h1>
216 <p>Security</p>
217 <ul>
218 <li>don't run WebUI on the internet</li>
219 <li> ssh slave plugin - it's possible for a slave to run arbitrary
220 code on the master</li>
221 </ul>
222 <p>Stability</p>
223 <ul><li>almost every Jenkins upgrade has broken us</li></ul>
224 <p>Scalability</p>
225 <ul>
226 <li>Jenkins has global mutexes, especially in plugins</li>
227 <li>Extra large cloud server could handle ~100 concurrent jobs</li>
228 <li>We ran 8 Jenkins Masters with slaves sharded across them</li>
229 </ul>
230 <p>Overkill</p>
231 <ul><li>we only used it as a remote shell execution engine</li></ul>
232 </section>
233
234 <section id="better-engine" class="slide level2">
235 <h1>We know a better engine for remote execution</h1>
236 </section>
237
238 <section id="zuul-v3-architecture" class="slide level2">
239 <p><img src="/images/infra-zuul-architecture.png" alt="image" /></p>
240 </section>
241
242 <section id="zuul-v3" class="slide level2">
243 <h1>Zuul V3</h1>
244 <h2>Test things like you deploy them</h2>
245 <ul>
246 <li>Intended for broad use</li>
247 <li>Jobs written in and executed with ansible</li>
248 <li>Fully support Bare Metal, VMs and Containers</li>
249 <li>Add GitHub PR Support</li>
250 <li>Add AWS and GCE Node Provider Support</li>
251 <li>Multi-node build clusters as first class resource</li>
252 <li>Docker and Kubernetes Build Resources</li>
253 <li>Multi-Tenant</li>
254 <li>In-repo config</li>
255 <li>Self-testing tests</li>
256 </ul>
257 </section>
258
259 <section id="ansible-execution" class="slide level2">
260 <h1> Ansible execution </h1>
261 <h2>Merge the mergers and the launchers</h2>
262 <ul>
263 <li>Constructs relevant repo states and inventory</li>
264 <li>Push repos to test nodes</li>
265 <li>Execute playbook with inventory</li>
266 <li>Playbook can be defined centrally or in a repo</li>
267 </ul>
268 </section>
269
270 <section id="in-repo-config" class="slide level2">
271 <h1>In repo config</h1>
272 <ul>
273 <li>Minimal config at start - other than "where are my repos"</li>
274 <li>'trusted' config repos</li>
275 <li>config per-repo </li>
276 <li>At start, zuul asks the fleet of merger-launchers to calculate config</li>
277 </ul>
278 </section>
279
280 <section id="test-tests" class="slide level2">
281 <h1> Test tests before you land them </h1>
282 <ul>
283 <li>Config is in git. Zuul does multo-repo git depends.</li>
284 <li>v3 adds with in-tree test definitions</li>
285 <li>Speculative state includes speculative state of test jobs</li>
286 </ul>
287 </section>
288
289 <section id="multi-node-support" class="slide level2">
290 <h1>First class multi-node support</h1>
291 <pre><code>
292- job:
293 name: ursula
294 parent: base
295 nodes:
296 - name: controller
297 image: ubuntu-trusty
298 - name: db
299 image: centos-7
300 </code></pre>
301 </section>
302
303 <section id="active-node-requests" class="slide level2">
304 <h1>Active node requests</h1>
305 <ul>
306 <li>zuul requests named resources</li>
307 <li>resources are created/checked out of pool</li>
308 <li>at finish, checked back in</li>
309 <li>Single-use environments delete on check in</li>
310 <li>Multi-use environments become available for subsequent jobs</li>
311 <li>Request non-node resources, such as kubernetes cluster</li>
312 </ul>
313 </section>
314
315 <section id="status" class="slide level2">
316 <h1>Status</h1>
317 <p>Focus</p>
318 <ul>
319 <li>OpenStack first, and the hard problems that brings</li>
320 <li>Extra-hard is handled. So is simple - but zuul is complex to run
321 if you only have the simple use cases</li>
322 </ul>
323 <p>Next up</p>
324 <ul>
325 <li>Get it ready for Ansible project</li>
326 <li>Making it truly suitable for not-OpenStack Infra to run</li>
327 <li>Making the easy tasks simple</li>
328 </ul>
329 </section>
330
331 <section id="more-info" class="slide level2">
332 <h1>For More Information</h1>
333
334 <ul>
335 <li><a href='href=http://docs.openstack.org/infra/zuul/'>
336 http://docs.openstack.org/infra/zuul/</a></li>
337 <li><a href='http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html'>
338 http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html</a></li>
339 <li>freenode:#zuul</li>
340 <li><a href='http://docs.openstack.org/infra/publications/zuul/#'>
341 http://docs.openstack.org/infra/publications/zuul/#(1)</a></li>
342 </ul>
343 </section>
344
345 </body>
346</html>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-21 02:50:39 +0000