summaryrefslogtreecommitdiff
path: root/src
diff options
context:
space:
mode:
Diffstat (limited to 'src')
-rw-r--r--src/talks/real-slim-shade.hbs669
1 files changed, 669 insertions, 0 deletions
diff --git a/src/talks/real-slim-shade.hbs b/src/talks/real-slim-shade.hbs
new file mode 100644
index 0000000..6955df3
--- /dev/null
+++ b/src/talks/real-slim-shade.hbs
@@ -0,0 +1,669 @@
1<!doctype html>
2<html lang="en">
3
4 <head>
5 <meta charset="utf-8">
6
7 <title>The Real Slim Shade</title>
8 </head>
9 <body>
10
11 <section id="who-am-i-ibm">
12 <h1>Who am I?</h1>
13 <img style="float:right; margin-right:24pt" src="/images/ibm-logo.png"/>
14 <p> Distinguished Engineer </p>
15 <p> IBM Cloud </p>
16 </section>
17
18 <section id="who-am-i-openstack">
19 <h1>Who am I?</h1>
20 <img style="float:right; margin-right:24pt" src="/images/openstack-cloud-software-vertical-large.png" />
21 <p>Technical Committee</p>
22 <p>Foundation Board of Directors</p>
23 <p>Developer Infrastructure Core Team</p>
24 </section>
25
26 <section id="what-are-we-going-to-talk-about" class="slide level2">
27 <h1>What are we going to talk about?</h1>
28 <ul>
29 <li>What</li>
30 <li>Why</li>
31 <li>Configuration</li>
32 <li>Basics</li>
33 <li>Advanced things
34 <ul>
35 <li>Caching</li>
36 <li>Task Management</li>
37 </ul>
38 </li>
39 </ul>
40 </section>
41
42 <section class="slide level2">
43 <h1>shade: a Python library to wrap business logic around
44 OpenStack resources and operations</h1>
45 </section>
46
47 <section class="slide level2">
48 <h1>Design Principles</h1>
49 <ul>
50 <li class='fragment'>Expose a single API that works on all clouds</li>
51 <li class='fragment'>Hide all vendor or deployer differences</li>
52 <li class='fragment'>Support multi-cloud (<em>write once, run anywhere</em>)</li>
53 <li class='fragment'>Simple to use (<em>sane defaults</em>)</li>
54 <li class='fragment'>No plugins *</li>
55 <li class='fragment'>Efficient at scale</li>
56 <li class='fragment'>API always backwards compatible</li>
57 </ul>
58 </section>
59
60 <section class="slide level2">
61 <h1>Current Status</h1>
62 <h3>https://git.openstack.org/cgit/openstack-infra/shade</h3>
63 <h3>https://pypi.python.org/pypi/shade</h3>
64 <ul>
65 <li class='fragment'>Version 1.0 Released!</li>
66 <li class='fragment'>Used in Ansible 2.0</li>
67 <li class='fragment'>Partially used in Infra Nodepool</li>
68 </ul>
69 </section>
70
71 <section class="slide level2">
72 <h1>Why?</h1>
73 </section>
74
75 <section class="slide level2">
76 <blockquote>
77 Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.</blockquote>
78 </section>
79
80 <section class="slide level2" data-transition='zoom'>
81 <h1>OpenStack Leaks Abstractions</h1>
82 </section>
83
84 <section class="slide level2" data-transition='zoom'>
85 <h1>OpenStack Breaks APIs</h1>
86 </section>
87
88 <section class="slide level2" data-transition='zoom'>
89 <h1>Basic concepts are needlessly complex</h1>
90 </section>
91
92 <section class="slide level2" data-transition='zoom'>
93 <h1>libcloud doens't really work</h1>
94 </section>
95
96 <section class="slide level2" data-transition='zoom'>
97 <h1>Client libraries are really for server-server communication</h1>
98 </section>
99
100 <section class="slide level2">
101 <h1>Infra solved these problems</h1>
102 <p>Infra runs across five clouds at massive scale</p>
103 <p>Why not share what we've learned with other people?</p>
104 </section>
105
106 <section class="slide level2">
107 <h1>simplicity</h1>
108 <p>This is what using a cloud should look like</p>
109 <pre><code>
110cloud = openstack_cloud('vexxhost')
111image = cloud.create_image(
112 'image-name', filename='image-filename.qcow2', wait=True)
113flavor = cloud.get_flavor_by_ram(512)
114cloud.create_server(
115 'my-server', image=image['id'], flavor=flavor['id'],
116 wait=True, auto_ip=True)
117 </code></pre>
118 </section>
119
120 <section class="slide level2" data-transition='zoom'>
121 <h1>existence of shade is a bug</h1>
122 </section>
123
124 <section class="slide level2">
125 <blockquote>
126 Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.</blockquote>
127 </section>
128
129 <section class="slide level2" data-transition='zoom'>
130 <h1>Let's drive towards profitable homogenization</h1>
131 </section>
132
133 <section class="slide level2">
134 <h1>Using shade</h1>
135 <h2>Step One: Configuration</h2>
136 </section>
137
138 <section class="slide level2">
139 <h1>os-client-config</h1>
140 <h3>http://git.openstack.org/cgit/openstack/os-client-config</h3>
141 <ul>
142 <li>A library to handle config information for openstack clients</li>
143 <li>Tracks differences in vendors that can't be discovered</li>
144 <li>In use in python-openstackclient and ansible</li>
145 <li>Patches up for neutronclient - on the way for nova and glance</li>
146 <li>Reads clouds.yaml config file, environment vars and argparse</li>
147 </ul>
148 </section>
149
150 <section class="slide level2">
151 <h1>os-client-config</h1>
152 <ul>
153 <li>OS_ Environment Variables</li>
154 <li>~/.config/openstack/clouds.yaml</li>
155 </ul>
156 <pre><code>
157clouds:
158 dreamhost:
159 profile: dreamhost
160 auth:
161 username: montay6
162 project_name: dhc2111978
163 password: XXXXXXXXXXXXX
164 montytaylor-sjc:
165 region_name: RegionOne
166 auth:
167 username: openstackjenkins
168 password: XXXXXXXXXXXXXX
169 project_name: openstackjenkins
170 auth_url: https://montytaylor-sjc.openstack.blueboxgrid.com:5001/v2.0
171 ustack:
172 profile: unitedstack
173 verify: False
174 auth_type: v3password
175 auth:
176 username: mordred
177 password: XXXXXXXXXXXXX
178 project_name: mordred_project
179 user_domain_id: 7fda84e224134e118158afd7d69936fd
180 project_domain_id: 7fda84e224134e118158afd7d69936fd
181 </code></pre>
182 </section>
183
184 <section class="slide level2">
185 <h1>auth_type</h1>
186 <p>keystone has pluggable authentication</p>
187
188 <ul>
189 <li>Defaults to 'password' which autodetects based on parameters</li>
190 <li>Other options: admin_token, v3oidcpassword, v3oidcauthcode</li>
191 <li>This is where I said "*" on "no plugins" earlier</li>
192 </ul>
193 </section>
194
195 <section class="slide level2">
196 <h1>python-openstackclient</h1>
197 <pre>
198openstack --os-cloud=rax --os-region=DFW servers list
199 </pre>
200 </section>
201
202 <section class="slide level2">
203 <h1>How I do it</h1>
204 <pre><code>
205function use {
206 declare -a CloudRegion=(${1//:/ })
207 export OS_CLOUD=${CloudRegion[0]}
208 export OS_REGION_NAME=${CloudRegion[1]}
209}
210PS1='${debian_chroot:+($debian_chroot)}${OS_CLOUD:+${OS_CLOUD}:}${OS_REGION_NAME:+${OS_REGION_NAME}:}\u@\h:\w\$ '
211 </code></pre>
212
213 <pre>
214mordred@camelot:~$ use rax:DFW
215rax:DFW:mordred@camelot:~$ openstack servers list
216 </pre>
217 </section>
218
219 <section class="slide level2">
220 <h1>shade inventory</h1>
221 <ul>
222 <li>Code behind ansible OpenStack dynamic inventory plugin</li>
223 <li>All resources in all of your clouds</li>
224 </ul>
225 <pre>
226mordred@camelot:~$ shade-inventory --list --yaml
227 </pre>
228 <pre><code>
229- HUMAN_ID: true
230 NAME_ATTR: name
231 OS-EXT-AZ:availability_zone: az2
232 OS-EXT-STS:power_state: 1
233 OS-EXT-STS:task_state: null
234 OS-EXT-STS:vm_state: active
235 accessIPv4: 15.126.131.74
236 accessIPv6: ''
237 addresses:
238 mordred@inaugust.com-network:
239 - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:f3:20:9b
240 OS-EXT-IPS:type: fixed
241 addr: 10.0.0.106
242 version: 4
243 - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:f3:20:9b
244 OS-EXT-IPS:type: floating
245 addr: 15.126.131.74
246 version: 4
247 az: az2
248 cloud: hp
249 config_drive: 'True'
250 created: '2015-04-25T15:15:22Z'
251 flavor:
252 id: '100'
253 name: standard.xsmall
254 hostId: 2a84ca599420d68466b782308f7892b81b4eda74debd037fe76c3ded
255 human_id: launch_node
256 id: f39df771-dd82-49b3-b325-b0b1a6842259
257 image:
258 id: c47a348c-fd37-437d-b7c4-e17ff6c86211
259 interface_ip: 15.126.131.74
260 key_name: null
261 metadata: {}
262 name: launch_node
263 networks:
264 mordred@inaugust.com-network:
265 - 10.0.0.106
266 - 15.126.131.74
267 private_v4: 10.0.0.106
268 progress: 0
269 public_v4: 15.126.131.74
270 public_v6: ''
271 region: region-b.geo-1
272 security_groups:
273 - name: default
274 status: ACTIVE
275 tenant_id: '51595564575618'
276 updated: '2015-06-06T17:26:09Z'
277 user_id: '71312979441162'
278 volumes: []
279- HUMAN_ID: true
280 NAME_ATTR: name
281 OS-DCF:diskConfig: MANUAL
282 OS-EXT-AZ:availability_zone: nova
283 OS-EXT-STS:power_state: 1
284 OS-EXT-STS:task_state: null
285 OS-EXT-STS:vm_state: active
286 OS-SRV-USG:launched_at: '2015-08-01T19:52:02.000000'
287 OS-SRV-USG:terminated_at: null
288 accessIPv4: 162.253.54.192
289 accessIPv6: ''
290 addresses:
291 public:
292 - OS-EXT-IPS-MAC:mac_addr: fa:16:3e:9f:46:3e
293 OS-EXT-IPS:type: fixed
294 addr: 162.253.54.192
295 version: 4
296 az: nova
297 cloud: vexxhost
298 config_drive: 'True'
299 created: '2015-08-01T19:52:16Z'
300 flavor:
301 id: bbcb7eb5-5c8d-498f-9d7e-307c575d3566
302 name: v1-standard-1
303 hostId: d998a94193237f8d12111cfdc856b83559e3a5b2e7716326ff46ad65
304 human_id: mordred-irc
305 id: 811c5197-dba7-4d3a-a3f6-68ca5328b9a7
306 image:
307 id: 69c99b45-cd53-49de-afdc-f24789eb8f83
308 name: Ubuntu 14.04.2 LTS
309 interface_ip: 162.253.54.192
310 key_name: mordred
311 metadata: {}
312 name: mordred-irc
313 networks:
314 public:
315 - 162.253.54.192
316 os-extended-volumes:volumes_attached: []
317 private_v4: ''
318 progress: 0
319 public_v4: 162.253.54.192
320 public_v6: ''
321 region: ca-ymq-1
322 security_groups:
323 - name: default
324 status: ACTIVE
325 tenant_id: db92b20496ae4fbda850a689ea9d563f
326 updated: '2015-08-01T19:52:02Z'
327 user_id: e9b21dc437d149858faee0898fb08e92
328 volumes: []
329 </code></pre>
330 </section>
331
332 <section class="slide level2">
333 <h1>Cloud-Region</h1>
334 <p>Each OpenStackCloud object represents one region of one cloud</p>
335 <ul>
336 <li>'envvars', $OS_REGION_NAME</li>
337 <li>cloud_config.name, cloud_config.region</li>
338 </ul>
339 </section>
340
341 <section class="slide level2">
342 <h1>Simplest Cloud Construction</h1>
343 <pre><code>
344import shade
345cloud = shade.openstack_cloud()
346 </code></pre>
347 </section>
348
349 <section class="slide level2">
350 <h1>Simple Cloud Construction</h1>
351 <pre><code>
352import shade
353cloud = shade.openstack_cloud(cloud='ustack', region_name='bj1')
354 </code></pre>
355 </section>
356
357 <section class="slide level2">
358 <h1>Complex Cloud Construction</h1>
359 <pre><code>
360import os_client_config
361import shade
362
363config = os_client_config.OpenStackConfig()
364cloud_config = config.get_one_cloud(
365 cloud='ustack', region_name='bj1',
366 argparse=my_args, **other_arguments)
367cloud = shade.OpenStackCloud(cloud_config=cloud_config)
368 </code></pre>
369 </section>
370
371 <section class="slide level2">
372 <h1>logging</h1>
373 <ul>
374 <li>Python logging</li>
375 <li>shade.simple_logging(debug=True) helper function</li>
376 </ul>
377 </section>
378
379 <section class="slide level2">
380 <h1>A note on Exceptions</h1>
381 <ul>
382 <li>All python*client exceptions are hidden and re-raised</li>
383 <li>Hiding exceptions is evil and I'm a bad person</li>
384 <li>Exceptions are part of the interface</li>
385 <li>Can't hide vendor choice if shade user catches underlying exception</li>
386 <li>Plan to get rid of python*client in the future</li>
387 </ul>
388 </section>
389
390 <section class="slide level2">
391 <h1>Putting it all together</h1>
392 <pre><code>
393import shade
394
395shade.simple_logging(debug=True)
396cloud = shade.openstack_cloud(cloud='vexxhost')
397image = cloud.create_image(
398 'ubuntu-trusty', filename='ubuntu-trusty.qcow2', wait=True)
399flavor = cloud.get_flavor_by_ram(512)
400cloud.create_server(
401 'my-server', image=image['id'], flavor=flavor['id'],
402 wait=True, auto_ip=True)
403 </code></pre>
404 </section>
405
406 <section class="slide level2">
407 <pre>
408Manager vexxhost running task GlanceImageList
409Manager vexxhost ran task GlanceImageList in 1.52185106277s
410Manager vexxhost running task ImageCreate
411Manager vexxhost ran task ImageCreate in 0.535313844681s
412Manager vexxhost running task ImageUpload
413Manager vexxhost ran task ImageUpload in 3.40991711617s
414Manager vexxhost running task GlanceImageList
415Manager vexxhost ran task GlanceImageList in 0.513506174088s
416Manager vexxhost running task FlavorList
417Manager vexxhost ran task FlavorList in 1.30724596977s
418Manager vexxhost running task ServerCreate
419Manager vexxhost ran task ServerCreate in 0.609236955643s
420Manager vexxhost running task ServerGet
421Manager vexxhost ran task ServerGet in 0.275503873825s
422Manager vexxhost running task ServerList
423Manager vexxhost ran task ServerList in 0.338351964951s
424Waiting 5 seconds <span style='color: red' class='fragment'>Poll loop</span>
425Manager vexxhost running task ServerList
426Manager vexxhost ran task ServerList in 0.318593025208s
427Manager vexxhost running task NetworkList
428Manager vexxhost ran task NetworkList in 1.11756396294s
429 </pre>
430 </section>
431
432 <section class="slide level2">
433 <h1>Problem: Image API version</h1>
434 <ul>
435 <li class='fragment'>
436 v1 PUT: HP, Catalyst, Datacentred, Internap
437 </li>
438 <li class='fragment'>
439 v2 PUT: Auro, Blue Box, City Cloud, Dreamhost, Elastx, Enter Cloud Suite, HP, OVH, RunAbove, Vexxhost, Ultimum, UnitedStack
440 </li>
441 <li class='fragment'>
442 v2 Tasks: Switch Engines, Rackspace
443 </li>
444 </ul>
445 </section>
446
447 <section class="slide level2">
448 <pre>
449Manager vexxhost running task GlanceImageList
450Manager vexxhost ran task GlanceImageList in 1.52185106277s
451Manager vexxhost running task ImageCreate
452Manager vexxhost ran task ImageCreate in 0.535313844681s
453Manager vexxhost running task ImageUpload
454Manager vexxhost ran task ImageUpload in 3.40991711617s
455Manager vexxhost running task GlanceImageList
456Manager vexxhost ran task GlanceImageList in 0.513506174088s
457 </pre>
458 </section>
459
460 <section class="slide level2">
461 <h1>Image Tasks</h1>
462 <pre><code>
463cloud = shade.openstack_cloud(cloud='rax')
464cloud.create_image('ubuntu-trusty', filename='ubuntu-trusty.vhd', wait=True)
465 </code></pre>
466 </section>
467
468 <section class="slide level2">
469 <pre>
470Manager rax running task GlanceImageList
471Manager rax ran task GlanceImageList in 2.0544719696s
472Manager rax running task ObjectCapabilities
473Manager rax ran task ObjectCapabilities in 0.616773843765s
474Manager rax running task ContainerGet
475Manager rax ran task ContainerGet in 0.978188037872s
476Manager rax running task ObjectMetadata
477Manager rax ran task ObjectMetadata in 0.427683115005s
478swift stale check, no object: images/ubuntu-trusty
479swift uploading ubuntu-trusty.vhd to images/ubuntu-trusty <span style='color: red' class='fragment'>Uses Swift Service</span>
480Manager rax running task ObjectCreate
481Manager rax ran task ObjectCreate in 0.00043797492981s
482Manager rax running task GlanceImageList
483Manager rax ran task GlanceImageList in 0.844120025635s
484Manager rax running task ImageTaskCreate
485Manager rax ran task ImageTaskCreate in 1.22424006462s
486Manager rax running task ImageTaskGet
487Manager rax ran task ImageTaskGet in 0.235086917877s
488Waiting 2 seconds
489Manager rax running task ImageTaskGet
490Manager rax ran task ImageTaskGet in 0.215805053711s
491 </pre>
492 </section>
493
494 <section class="slide level2">
495 <h1>Problem: Networking choices</h1>
496 <ul>
497 <li class='fragment'>Cloud has externally routable IP from neutron (RunAbove, OVH)</li>
498 <li class='fragment'>Cloud has externally routable IP neutron AND supports optional private tenant networks (vexxhost)</li>
499 <li class='fragment'>Cloud has private tenant network provided by neutron and requires floating IP (HP, Dreamhost)</li>
500 <li class='fragment'>Cloud has private tenant network provided by nova-network and requires floating-ip for external routing (auro)</li>
501 <li class='fragment'>Cloud has externally routable IP from neutron but no neutron APIs (Rackspace)</li>
502 </ul>
503 </section>
504
505 <section class="slide level2">
506 <h1>The Floating IP Case</h1>
507 <pre><code>
508import shade
509
510shade.simple_logging(debug=True)
511cloud = shade.openstack_cloud(cloud='hp')
512flavor = cloud.get_flavor_by_ram(512)
513image = cloud.get_image('ubuntu-trusty')
514cloud.create_server(
515 'my-server', image=image['id'], flavor=flavor['id'],
516 wait=True, auto_ip=True)
517 </code></pre>
518 </section>
519
520 <section class="slide level2">
521 <h1>The Floating IP Case</h1>
522 <pre>
523Manager hp running task ServerCreate
524Manager hp ran task ServerCreate in 1.52772402763s
525Manager hp running task ServerGet
526Manager hp ran task ServerGet in 0.483407020569s
527Manager hp running task ServerList
528Manager hp ran task ServerList in 0.809002876282s
529Waiting 5 seconds
530Manager hp running task ServerList
531Manager hp ran task ServerList in 1.091807127s
532Manager hp running task NetworkList
533Manager hp ran task NetworkList in 21.8774280548s
534Manager hp running task NeutronFloatingIPList
535Manager hp ran task NeutronFloatingIPList in 0.213263988495s
536Manager hp running task PortList
537Manager hp ran task PortList in 0.342208862305s
538Manager hp running task NeutronFloatingIPUpdate
539Manager hp ran task NeutronFloatingIPUpdate in 0.356763124466s
540Manager hp running task ServerGet
541Manager hp ran task ServerGet in 0.433247089386s
542Waiting 2 seconds
543Manager hp running task ServerGet
544Manager hp ran task ServerGet in 0.56911110878s
545Manager hp running task ServerGet
546Manager hp ran task ServerGet in 0.302348852158s
547 </pre>
548 </section>
549
550 <section class="slide level2">
551 <h1>Don't Reuse Floating IP Case</h1>
552 <pre><code>
553import shade
554
555shade.simple_logging(debug=True)
556cloud = shade.openstack_cloud(cloud='bluebox')
557flavor = cloud.get_flavor_by_ram(512)
558image = cloud.get_image('ubuntu-trusty')
559cloud.create_server(
560 'my-server', image=image['id'], flavor=flavor['id'],
561 wait=True, auto_ip=True, reuse=False)
562 </code></pre>
563 </section>
564
565 <section class="slide level2">
566 <pre>
567Manager bluebox running task NetworkList
568Manager bluebox ran task NetworkList in 0.432415008545s
569Manager bluebox running task NeutronFloatingIPList
570Manager bluebox ran task NeutronFloatingIPList in 0.54718708992s
571Manager bluebox running task NetworkList
572Manager bluebox ran task NetworkList in 0.495000839233s
573Manager bluebox running task NeutronFloatingIPCreate
574Manager bluebox ran task NeutronFloatingIPCreate in 0.819630146027s
575Manager bluebox running task PortList
576Manager bluebox ran task PortList in 0.736854076385s
577Manager bluebox running task NeutronFloatingIPUpdate
578Manager bluebox ran task NeutronFloatingIPUpdate in 0.588222026825s
579Manager bluebox running task ServerGet
580Manager bluebox ran task ServerGet in 0.646552085876s
581Waiting 2 seconds
582Manager bluebox running task ServerGet
583Manager bluebox ran task ServerGet in 0.726951122284s
584Manager bluebox running task ServerGet
585Manager bluebox ran task ServerGet in 0.668478965759s
586 </pre>
587 </section>
588
589 <section class="slide level2">
590 <h1>Advanced Topics</h1>
591 </section>
592
593 <section class="slide level2">
594 <h1>Task Manager</h1>
595 <pre>
596Manager bluebox running task ServerGet
597Manager bluebox ran task ServerGet in 0.726951122284s
598 </pre>
599 <ul>
600 <li>Every API operation is a Task() that is run by a TaskManager()</li>
601 <li>Default shade TaskManager immediately executes the call</li>
602 <li>nodepool passes in a threaded TaskManager to manage API throttling</li>
603 <pre><code>
604cloud = shade.OpenStackCloud(
605 cloud_config=cloud_config,
606 manager=MyTaskManager())
607 </code></pre>
608 </ul>
609 </section>
610
611 <section class="slide level2">
612 <h1>Caching</h1>
613 <ul>
614 <li>API Operations can be expensive</li>
615 <li>shade uses dogpile.cache - default to NullCache</li>
616 </li>
617 </section>
618
619 <section class="slide level2">
620 <h1>Cache Config</h1>
621 <h2>clouds.yaml</h2>
622 <pre><code>
623cache:
624 class: dogpile.cache.dbm
625 expiration_time: 3600
626 arguments:
627 filename: /home/mordred/.cache/openstack/shade.dbm
628 expiration:
629 server: 5
630 </code></pre>
631 </section>
632
633 <section class="slide level2">
634 <h1>ansible 2.0</h1>
635 <p>Brand new modules, based on shade</p>
636 <p>Coming in 2.0 release</p>
637 <pre><code>
638- os_image:
639 cloud: vexxhost
640 name: ubuntu-trusty
641 file: ubuntu-trusty.qcow2
642 wait: true
643- os_server:
644 cloud: vexxhost
645 name: my-server
646 flavor_ram: 1024
647 image: ubuntu-trusty
648 wait: true
649 </code></pre>
650 </section>
651
652 <section class="slide level2">
653 <h1>ansible</h1>
654 <p>multi-cloud support</p>
655 <pre><code>
656- os_keypair:
657 cloud: "{{ item }"
658 name: mordred
659 public_key_file: ~/.ssh/id_rsa.pub
660 with-items:
661 - vexxhost
662 - rackspace
663 - hp
664 - ovh
665 </code></pre>
666 </section>
667
668 </body>
669</html>
generated by cgit v1.2.3 (git 2.25.1) at 2024-09-22 03:37:43 +0000