- Notifications
You must be signed in to change notification settings - Fork 54
/
Copy pathGenerators_and_Comprehensions.html
686 lines (648 loc) · 68.7 KB
/
Generators_and_Comprehensions.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
<!DOCTYPE html>
<htmlclass="writer-html5" lang="en" >
<head>
<metacharset="utf-8" /><metacontent="Topic: generators comprehensions and efficiency, Difficulty: Easy, Category: Section" name="description" />
<metacontent="generator, range, list comprehension, generator comprehension, nested comprehensions, inline for-loop, filtered, iterator" name="keywords" />
<metaname="viewport" content="width=device-width, initial-scale=1.0" />
<title>Generators & Comprehension Expressions — Python Like You Mean It</title>
<linkrel="stylesheet" href="../_static/pygments.css" type="text/css" />
<linkrel="stylesheet" href="../_static/css/theme.css" type="text/css" />
<linkrel="stylesheet" href="../_static/my_theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<scriptdata-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<scriptsrc="../_static/jquery.js"></script>
<scriptsrc="../_static/underscore.js"></script>
<scriptsrc="../_static/doctools.js"></script>
<scriptasync="async" src="https://www.googletagmanager.com/gtag/js?id=UA-115029372-1"></script>
<scriptsrc="../_static/gtag.js"></script>
<scriptcrossorigin="anonymous" integrity="sha256-Ae2Vz/4ePdIu6ZyI/5ZGsYnb+m0JlOmKPjt6XZ9JJkA=" src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.4/require.min.js"></script>
<script>window.MathJax={"tex": {"inlineMath": [["$","$"],["\\(","\\)"]],"processEscapes": true},"options": {"ignoreHtmlClass": "tex2jax_ignore|mathjax_ignore|document","processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<scriptdefer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<scriptsrc="../_static/js/theme.js"></script>
<linkrel="index" title="Index" href="../genindex.html" />
<linkrel="search" title="Search" href="../search.html" />
<linkrel="next" title="Python’s “Itertools”" href="Itertools.html" />
<linkrel="prev" title="Iterables" href="Iterables.html" />
</head>
<bodyclass="wy-body-for-nav">
<divclass="wy-grid-for-nav">
<navdata-toggle="wy-nav-shift" class="wy-nav-side">
<divclass="wy-side-scroll">
<divclass="wy-side-nav-search" >
<ahref="../index.html" class="icon icon-home"> Python Like You Mean It
</a>
<divclass="version">
1.4
</div>
<divrole="search">
<formid="rtd-search-form" class="wy-form" action="../search.html" method="get">
<inputtype="text" name="q" placeholder="Search docs" />
<inputtype="hidden" name="check_keywords" value="yes" />
<inputtype="hidden" name="area" value="default" />
</form>
</div>
</div><divclass="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<pclass="caption" role="heading"><spanclass="caption-text">Table of Contents:</span></p>
<ulclass="current">
<liclass="toctree-l1"><aclass="reference internal" href="../intro.html">Python Like You Mean It</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_1.html">Module 1: Getting Started with Python</a></li>
<liclass="toctree-l1 current"><aclass="reference internal" href="../module_2.html">Module 2: The Essentials of Python</a><ulclass="current">
<liclass="toctree-l2"><aclass="reference internal" href="Basic_Objects.html">Basic Object Types</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="SequenceTypes.html">Sequence Types</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="Variables_and_Assignment.html">Variables & Assignment</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="Introduction.html">Introducing Control Flow</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="ConditionalStatements.html">Conditional Statements</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="ForLoops.html">For-Loops and While-Loops</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="Iterables.html">Iterables</a></li>
<liclass="toctree-l2 current"><aclass="current reference internal" href="#">Generators & Comprehension Expressions</a><ul>
<liclass="toctree-l3"><aclass="reference internal" href="#Introducing-Generators">Introducing Generators</a><ul>
<liclass="toctree-l4"><aclass="reference internal" href="#The-range-generator">The <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> generator</a></li>
</ul>
</li>
<liclass="toctree-l3"><aclass="reference internal" href="#Creating-your-own-generator:-generator-comprehensions">Creating your own generator: generator comprehensions</a><ul>
<liclass="toctree-l4"><aclass="reference internal" href="#Storing-generators">Storing generators</a></li>
<liclass="toctree-l4"><aclass="reference internal" href="#Consuming-generators">Consuming generators</a></li>
<liclass="toctree-l4"><aclass="reference internal" href="#Chaining-comprehensions">Chaining comprehensions</a></li>
<liclass="toctree-l4"><aclass="reference internal" href="#Using-generator-comprehensions-on-the-fly">Using generator comprehensions on the fly</a></li>
</ul>
</li>
<liclass="toctree-l3"><aclass="reference internal" href="#Iterating-over-generators-using-next">Iterating over generators using <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code></a><ul>
<liclass="toctree-l4"><aclass="reference internal" href="#Iterables-vs. Iterators">Iterables vs. Iterators</a></li>
</ul>
</li>
<liclass="toctree-l3"><aclass="reference internal" href="#List-&-Tuple-Comprehensions">List & Tuple Comprehensions</a><ul>
<liclass="toctree-l4"><aclass="reference internal" href="#Nesting-comprehensions">Nesting comprehensions</a></li>
</ul>
</li>
<liclass="toctree-l3"><aclass="reference internal" href="#Links-to-Official-Documentation">Links to Official Documentation</a></li>
<liclass="toctree-l3"><aclass="reference internal" href="#Reading-Comprehension-Exercise-Solutions:">Reading Comprehension Exercise Solutions:</a></li>
</ul>
</li>
<liclass="toctree-l2"><aclass="reference internal" href="Itertools.html">Python’s “Itertools”</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="Functions.html">Basics of Functions</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="Scope.html">Scope</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="DataStructures.html">Data Structures (Part I): Introduction</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="DataStructures_II_Dictionaries.html">Data Structures (Part II): Dictionaries</a></li>
<liclass="toctree-l2"><aclass="reference internal" href="DataStructures_III_Sets_and_More.html">Data Structures (Part III): Sets & the Collections Module</a></li>
</ul>
</li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_2_problems.html">Module 2: Problems</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_3.html">Module 3: The Essentials of NumPy</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_3_problems.html">Module 3: Problems</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_4.html">Module 4: Object Oriented Programming</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../module_5.html">Module 5: Odds and Ends</a></li>
<liclass="toctree-l1"><aclass="reference internal" href="../changes.html">Changelog</a></li>
</ul>
</div>
</div>
</nav>
<sectiondata-toggle="wy-nav-shift" class="wy-nav-content-wrap"><navclass="wy-nav-top" aria-label="Mobile navigation menu" >
<idata-toggle="wy-nav-top" class="fa fa-bars"></i>
<ahref="../index.html">Python Like You Mean It</a>
</nav>
<divclass="wy-nav-content">
<divclass="rst-content">
<divrole="navigation" aria-label="Page navigation">
<ulclass="wy-breadcrumbs">
<li><ahref="../index.html" class="icon icon-home"></a> »</li>
<li><ahref="../module_2.html">Module 2: The Essentials of Python</a> »</li>
<li>Generators & Comprehension Expressions</li>
<liclass="wy-breadcrumbs-aside">
<ahref="../_sources/Module2_EssentialsOfPython/Generators_and_Comprehensions.md.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<divrole="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<divitemprop="articleBody">
<style>
/* CSS overrides for sphinx_rtd_theme */
/* 24px margin */
.nbinput.nblast.container,
.nboutput.nblast.container {
margin-bottom:19px; /* padding has already 5px */
}
/* ... except between code cells! */
.nblast.container+ .nbinput.container {
margin-top:-19px;
}
.admonition>p:before {
margin-right:4px; /* make room for the exclamation icon */
}
/* Fix math alignment, see https://github.com/rtfd/sphinx_rtd_theme/pull/686 */
.math {
text-align: unset;
}
</style>
<divclass="section" id="Generators-&-Comprehension-Expressions">
<h1>Generators & Comprehension Expressions<aclass="headerlink" href="#Generators-&-Comprehension-Expressions" title="Permalink to this headline"></a></h1>
<divclass="admonition warning">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Note</strong>:</p>
<p>There are reading-comprehension exercises included throughout the text. These are meant to help you put your reading to practice. Solutions for the exercises are included at the bottom of this page.</p>
</div>
<divclass="section" id="Introducing-Generators">
<h2>Introducing Generators<aclass="headerlink" href="#Introducing-Generators" title="Permalink to this headline"></a></h2>
<p>Now we introduce an important type of object called a <strong>generator</strong>, which allows us to generate arbitrarily-many items in a series, without having to store them all in memory at once.</p>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Definition</strong>:</p>
<p>A <strong>generator</strong> is a special kind of iterator, which stores the instructions for how to <em>generate</em> each of its members, in order, along with its current state of iterations. It generates each member, one at a time, only as it is requested via iteration.</p>
</div>
<p>Recall that a list readily stores all of its members; you can access any of its contents via indexing. A generator, on the other hand, <em>does not store any items</em>. Instead, it stores the instructions for generating each of its members, and stores its iteration state; this means that the generator will know if it has generated its second member, and will thus generate its third member the next time it is iterated on.</p>
<p>The whole point of this is that you can use a generator to produce a long sequence of items, without having to store them all in memory.</p>
<divclass="section" id="The-range-generator">
<h3>The <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> generator<aclass="headerlink" href="#The-range-generator" title="Permalink to this headline"></a></h3>
<p>An extremely popular built-in generator is <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code>, which, given the values:</p>
<ulclass="simple">
<li><p>‘start’ (inclusive, default=0)</p></li>
<li><p>‘stop’ (exclusive)</p></li>
<li><p>‘step’ (default=1)</p></li>
</ul>
<p>will generate the corresponding sequence of integers (from start to stop, using the step size) upon iteration. Consider the following example usages of <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code>:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># start: 2 (included)</span>
<spanclass="c1"># stop: 7 (excluded)</span>
<spanclass="c1"># step: 1 (default)</span>
<spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">2</span><spanclass="p">,</span><spanclass="mi">7</span><spanclass="p">):</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">)</span>
<spanclass="c1"># prints: 2.. 3.. 4.. 5.. 6</span>
</pre></div>
</div>
<hrclass="docutils" />
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># start: 1 (included)</span>
<spanclass="c1"># stop: 10 (excluded)</span>
<spanclass="c1"># step: 2</span>
<spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">):</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">)</span>
<spanclass="c1"># prints: 1.. 3.. 5.. 7.. 9</span>
</pre></div>
</div>
<hrclass="docutils" />
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># A very common use case!</span>
<spanclass="c1"># start: 0 (default, included)</span>
<spanclass="c1"># stop: 5 (excluded)</span>
<spanclass="c1"># step: 1 (default)</span>
<spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">5</span><spanclass="p">):</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">)</span>
<spanclass="c1"># prints: 0.. 1.. 2.. 3.. 4</span>
</pre></div>
</div>
<p>Because <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> is a generator, the command <codeclass="docutils literal notranslate"><spanclass="pre">range(5)</span></code> will simply store the instructions needed to produce the sequence of numbers 0-4, whereas the list <codeclass="docutils literal notranslate"><spanclass="pre">[0,</span><spanclass="pre">1,</span><spanclass="pre">2,</span><spanclass="pre">3,</span><spanclass="pre">4]</span></code> stores all of these items in memory at once. For short sequences, this seems to be a rather paltry savings; this is not the case for long sequences. The following graph compares the memory consumption used when defining a generator for the sequence of numbers <spanclass="math notranslate nohighlight">\(0-N\)</span> using <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code>, compared to storing the sequence
in a list:</p>
<p><imgalt="Memory consumption figure" src="../_images/Mem_Consumption_Generator.png" /></p>
<p>Given our discussion of generators, it should make sense that the memory consumed simply by defining <codeclass="docutils literal notranslate"><spanclass="pre">range(N)</span></code> is independent of <spanclass="math notranslate nohighlight">\(N\)</span>, whereas the memory consumed by the list grows linearly with <spanclass="math notranslate nohighlight">\(N\)</span> (for large <spanclass="math notranslate nohighlight">\(N\)</span>).</p>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Takeaway</strong>:</p>
<p><codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> is a built-in generator, which generates sequences of integers.</p>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Using ``range``</strong>:</p>
<p>Using <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> in a for-loop, print the numbers 10-1, in sequence.</p>
</div>
</div>
</div>
<divclass="section" id="Creating-your-own-generator:-generator-comprehensions">
<h2>Creating your own generator: generator comprehensions<aclass="headerlink" href="#Creating-your-own-generator:-generator-comprehensions" title="Permalink to this headline"></a></h2>
<p>Python provides a sleek syntax for defining a simple generator in a single line of code; this expression is known as a <strong>generator comprehension</strong>. The following syntax is extremely useful and will appear very frequently in Python code:</p>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Definition</strong>:</p>
<p>The syntax <codeclass="docutils literal notranslate"><spanclass="pre">(<expression></span><spanclass="pre">for</span><spanclass="pre"><var></span><spanclass="pre">in</span><spanclass="pre"><iterable></span><spanclass="pre">[if</span><spanclass="pre"><condition>])</span></code> specifies the general form for a <strong>generator comprehension</strong>. This produces a generator, whose instructions for generating its members are provided within the parenthetical statement.</p>
</div>
<p>Written in a long form, the pseudo-code for</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>(<expression> for <var> in <iterable> if <condition>)
</pre></div>
</div>
<p>is:</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>for <var> in <iterable>:
if bool(<condition>):
yield <expression>
</pre></div>
</div>
<p>The following expression defines a generator for all the even numbers in 0-99:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># when iterated over, `even_gen` will generate 0.. 2.. 4.. ... 98</span>
<spanclass="n">even_gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">)</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">%</span><spanclass="mi">2</span><spanclass="o">==</span><spanclass="mi">0</span><spanclass="p">)</span>
</pre></div>
</div>
<p>The <codeclass="docutils literal notranslate"><spanclass="pre">if</span><spanclass="pre"><condition></span></code> clause in the generator expression is optional. The generator comprehension</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>(<expression> for <var> in <iterable>)
</pre></div>
</div>
<p>corresponds to:</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>for <var> in <iterable>:
yield <expression>
</pre></div>
</div>
<p>For example:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># when iterated over, `example_gen` will generate 0/2.. 9/2.. 21/2.. 32/2</span>
<spanclass="n">example_gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">/</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">9</span><spanclass="p">,</span><spanclass="mi">21</span><spanclass="p">,</span><spanclass="mi">32</span><spanclass="p">])</span>
<spanclass="k">for</span><spanclass="n">item</span><spanclass="ow">in</span><spanclass="n">example_gen</span><spanclass="p">:</span>
<spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">item</span><spanclass="p">)</span>
<spanclass="c1"># prints: 0.0.. 4.5.. 10.5.. 16.0</span>
</pre></div>
</div>
<p><codeclass="docutils literal notranslate"><spanclass="pre"><expression></span></code> can be any valid single-line of Python code that returns an object:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="p">((</span><spanclass="n">i</span><spanclass="p">,</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="p">,</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">3</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">10</span><spanclass="p">))</span>
<spanclass="c1"># will generate:</span>
<spanclass="c1"># (0, 0, 0)</span>
<spanclass="c1"># (1, 1, 1)</span>
<spanclass="c1"># (2, 4, 8)</span>
<spanclass="c1"># (3, 9, 27)</span>
<spanclass="c1"># (4, 16, 64)</span>
<spanclass="c1"># (5, 25, 125)</span>
<spanclass="c1"># (6, 36, 216)</span>
<spanclass="c1"># (7, 49, 343)</span>
<spanclass="c1"># (8, 64, 512)</span>
<spanclass="c1"># (9, 81, 729)</span>
</pre></div>
</div>
<p>This means that <codeclass="docutils literal notranslate"><spanclass="pre"><expression></span></code> can even involve inline if-else statements!</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="p">((</span><spanclass="s2">"apple"</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o"><</span><spanclass="mi">3</span><spanclass="k">else</span><spanclass="s2">"pie"</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">6</span><spanclass="p">))</span>
<spanclass="c1"># will generate:</span>
<spanclass="c1"># 'apple'..</span>
<spanclass="c1"># 'apple'..</span>
<spanclass="c1"># 'apple'..</span>
<spanclass="c1"># 'pie'..</span>
<spanclass="c1"># 'pie'..</span>
<spanclass="c1"># 'pie'</span>
</pre></div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Takeaway</strong>:</p>
<p>A generator comprehension is a single-line specification for defining a generator in Python. It is absolutely essential to learn this syntax in order to write simple and readable code.</p>
</div>
<divclass="admonition warning">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Note</strong>:</p>
<p>Generator comprehensions are <strong>not</strong> the only method for defining generators in Python. One can define a generator similar to the way one can define a function (which we will encounter soon). <aclass="reference external" href="https://docs.python.org/3/tutorial/classes.html#generators">See this section of the official Python tutorial</a> if you are interested in diving deeper into generators.</p>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Writing a Generator Comprehension</strong>:</p>
<p>Using a generator comprehension, define a generator for the series:</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>(0, 2).. (1, 3).. (2, 4).. (4, 6).. (5, 7)
</pre></div>
</div>
<p>Note that (3, 5) is <em>not</em> in the series.</p>
<p>Iterate over the generator and print its contents to verify your solution.</p>
</div>
<divclass="section" id="Storing-generators">
<h3>Storing generators<aclass="headerlink" href="#Storing-generators" title="Permalink to this headline"></a></h3>
<p>Just like we saw with the <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> generator, defining a generator using a comprehension does <em>not</em> perform any computations or consume any memory beyond defining the rules for producing the sequence of data. See what happens when we try to print this generator:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># will generate 0, 1, 4, 9, 25, ..., 9801</span>
<spanclass="o">>>></span><spanclass="n">gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">))</span>
<spanclass="o">>>></span><spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span>
<spanclass="o"><</span><spanclass="n">generator</span><spanclass="nb">object</span><spanclass="o"><</span><spanclass="n">genexpr</span><spanclass="o">></span><spanclass="n">at</span><spanclass="mh">0x000001E768FE8A40</span><spanclass="o">></span>
</pre></div>
</div>
<p>This output simply indicates that <codeclass="docutils literal notranslate"><spanclass="pre">gen</span></code> stores a generator-expression at the memory address <codeclass="docutils literal notranslate"><spanclass="pre">0x000001E768FE8A40</span></code>; this is simply where the instructions for generating our sequence of squared numbers is stored. <codeclass="docutils literal notranslate"><spanclass="pre">gen</span></code> will not produce any results until we iterate over it. For this reason, generators cannot be inspected in the same way that lists and other sequences can be. You <strong>cannot</strong> do the following:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># you **cannot** do the following...</span>
<spanclass="o">>>></span><spanclass="n">gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">))</span>
<spanclass="c1"># query the length of a generator</span>
<spanclass="o">>>></span><spanclass="nb">len</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span>
<spanclass="ne">TypeError</span><spanclass="p">:</span><spanclass="nb">object</span><spanclass="n">of</span><spanclass="nb">type</span><spanclass="s1">'generator'</span><spanclass="n">has</span><spanclass="n">no</span><spanclass="nb">len</span><spanclass="p">()</span>
<spanclass="c1"># index into a generator</span>
<spanclass="o">>>></span><spanclass="n">gen</span><spanclass="p">[</span><spanclass="mi">2</span><spanclass="p">]</span>
<spanclass="ne">TypeError</span><spanclass="p">:</span><spanclass="s1">'generator'</span><spanclass="nb">object</span><spanclass="ow">is</span><spanclass="ow">not</span><spanclass="n">subscriptable</span>
</pre></div>
</div>
<p>The sole exception to this is the <codeclass="docutils literal notranslate"><spanclass="pre">range</span></code> generator, for which all of these inspections are valid.</p>
</div>
<divclass="section" id="Consuming-generators">
<h3>Consuming generators<aclass="headerlink" href="#Consuming-generators" title="Permalink to this headline"></a></h3>
<p>We can feed this to any function that accepts iterables. For instance, we can feed <codeclass="docutils literal notranslate"><spanclass="pre">gen</span></code> to the built-in <codeclass="docutils literal notranslate"><spanclass="pre">sum</span></code> function, which sums the contents of an iterable:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">))</span>
<spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span><spanclass="c1"># computes the sum 0 + 1 + 4 + 9 + 25 + ... + 9801</span>
<spanclass="go">328350</span>
</pre></div>
</div>
<p>This computes the sum of the sequence of numbers <em>without ever storing the full sequence of numbers in memory</em>. In fact, only two numbers need be stored during any given iteration of the sum: the current value of the sum, and the number being added to it.</p>
<p>What happens if we run this command a second time:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># computes the sum of ... nothing!</span>
<spanclass="c1"># `gen` has already been consumed!</span>
<spanclass="o">>>></span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span>
<spanclass="mi">0</span>
</pre></div>
</div>
<p>It may be surprising to see that the sum now returns 0. This is because <strong>a generator is exhausted after it is iterated over in full</strong>. You must redefine the generator if you want to iterate over it again; fortunately, defining a generator requires very few resources, so this is not a point of concern.</p>
<p>You can also check for membership in a generator, but this also consumes the generator:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># checking for membership consumes a generator until</span>
<spanclass="c1"># it finds that item (consuming the entire generator</span>
<spanclass="c1"># if the item is not contained within it)</span>
<spanclass="o">>>></span><spanclass="n">gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">11</span><spanclass="p">))</span>
<spanclass="o">>>></span><spanclass="mi">5</span><spanclass="ow">in</span><spanclass="n">gen</span><spanclass="c1"># first 5 elements are consumed</span>
<spanclass="kc">True</span>
<spanclass="c1"># 1-5 are no longer contained in gen</span>
<spanclass="c1"># this check consumes the entire generator!</span>
<spanclass="o">>>></span><spanclass="mi">5</span><spanclass="ow">in</span><spanclass="n">gen</span>
<spanclass="kc">False</span>
<spanclass="o">>>></span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span>
<spanclass="mi">0</span>
</pre></div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Takeaway</strong>:</p>
<p>A generator can only be iterated over once, after which it is exhausted and must be re-defined in order to be iterated over again.</p>
</div>
</div>
<divclass="section" id="Chaining-comprehensions">
<h3>Chaining comprehensions<aclass="headerlink" href="#Chaining-comprehensions" title="Permalink to this headline"></a></h3>
<p>Because generators are iterables, they can be fed into subsequent generator comprehensions. That is, they can be “chained” together.</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># chaining two generator comprehensions</span>
<spanclass="c1"># generates 400.. 100.. 0.. 100.. 400</span>
<spanclass="o">>>></span><spanclass="n">gen_1</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="o">-</span><spanclass="mi">20</span><spanclass="p">,</span><spanclass="o">-</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">20</span><spanclass="p">])</span>
<spanclass="c1"># iterates through gen_1, excluding any numbers whose absolute value is greater than 150</span>
<spanclass="o">>>></span><spanclass="n">gen_2</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">j</span><spanclass="k">for</span><spanclass="n">j</span><spanclass="ow">in</span><spanclass="n">gen_1</span><spanclass="k">if</span><spanclass="nb">abs</span><spanclass="p">(</span><spanclass="n">j</span><spanclass="p">)</span><spanclass="o"><=</span><spanclass="mi">150</span><spanclass="p">)</span>
<spanclass="c1"># computing 100 + 0 + 100</span>
<spanclass="o">>>></span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">gen_2</span><spanclass="p">)</span>
<spanclass="mi">200</span>
</pre></div>
</div>
<p>This is equivalent to:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="n">total</span><spanclass="o">=</span><spanclass="mi">0</span>
<spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="o">-</span><spanclass="mi">20</span><spanclass="p">,</span><spanclass="o">-</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">20</span><spanclass="p">]:</span>
<spanclass="n">j</span><spanclass="o">=</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span>
<spanclass="k">if</span><spanclass="n">j</span><spanclass="o"><=</span><spanclass="mi">150</span><spanclass="p">:</span>
<spanclass="n">total</span><spanclass="o">+=</span><spanclass="n">j</span>
<spanclass="c1"># total is now 200</span>
</pre></div>
</div>
</div>
<divclass="section" id="Using-generator-comprehensions-on-the-fly">
<h3>Using generator comprehensions on the fly<aclass="headerlink" href="#Using-generator-comprehensions-on-the-fly" title="Permalink to this headline"></a></h3>
<p>A feature of Python, that can make your code supremely readable and intuitive, is that generator comprehensions can be fed <em>directly</em> into functions that operate on iterables. That is,</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">))</span>
<spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">gen</span><spanclass="p">)</span>
<spanclass="go">328350</span>
</pre></div>
</div>
<p>can be simplified as:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">))</span>
<spanclass="go">328350</span>
</pre></div>
</div>
<p>If you want your code to compute the finite harmonic series: <spanclass="math notranslate nohighlight">\(\sum_{k=1}^{100} \frac{1}{n} = 1 + \frac{1}{2} + ... + \frac{1}{100}\)</span>, you can simply write:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="o">/</span><spanclass="n">n</span><spanclass="k">for</span><spanclass="n">n</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">101</span><spanclass="p">))</span>
<spanclass="go">5.187377517639621</span>
</pre></div>
</div>
<p>This convenient syntax works for any function that expects an iterable as an argument, such as the <codeclass="docutils literal notranslate"><spanclass="pre">list</span></code> function and <codeclass="docutils literal notranslate"><spanclass="pre">all</span></code> function:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># providing generator expressions as arguments to functions</span>
<spanclass="c1"># that operate on iterables</span>
<spanclass="o">>>></span><spanclass="nb">list</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">10</span><spanclass="p">))</span>
<spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">4</span><spanclass="p">,</span><spanclass="mi">9</span><spanclass="p">,</span><spanclass="mi">16</span><spanclass="p">,</span><spanclass="mi">25</span><spanclass="p">,</span><spanclass="mi">36</span><spanclass="p">,</span><spanclass="mi">49</span><spanclass="p">,</span><spanclass="mi">64</span><spanclass="p">,</span><spanclass="mi">81</span><spanclass="p">]</span>
<spanclass="o">>>></span><spanclass="nb">all</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o"><</span><spanclass="mi">10</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">3</span><spanclass="p">,</span><spanclass="mi">5</span><spanclass="p">,</span><spanclass="mi">7</span><spanclass="p">])</span>
<spanclass="kc">True</span>
<spanclass="o">>>></span><spanclass="s2">", "</span><spanclass="o">.</span><spanclass="n">join</span><spanclass="p">(</span><spanclass="nb">str</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">200</span><spanclass="p">,</span><spanclass="mi">4000</span><spanclass="p">,</span><spanclass="mi">80000</span><spanclass="p">])</span>
<spanclass="s1">'10, 200, 4000, 80000'</span>
</pre></div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Takeaway</strong>:</p>
<p>A generator comprehension can be specified directly as an argument to a function, wherever a single iterable is expected as an input to that function.</p>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Using Generator Comprehensions on the Fly</strong>:</p>
<p>In a single line, compute the sum of all of the odd-numbers in 0-100.</p>
</div>
</div>
</div>
<divclass="section" id="Iterating-over-generators-using-next">
<h2>Iterating over generators using <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code><aclass="headerlink" href="#Iterating-over-generators-using-next" title="Permalink to this headline"></a></h2>
<p>The built-in function <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code> allows you manually “request” the next member of a generator, or more generally, any kind of <em>iterator</em>. Calling <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code> on an exhausted iterator will raise a <codeclass="docutils literal notranslate"><spanclass="pre">StopIteration</span></code> signal.</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># consuming an iterator using `next`</span>
<spanclass="o">>>></span><spanclass="n">short_gen</span><spanclass="o">=</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">/</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="p">[</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">,</span><spanclass="mi">3</span><spanclass="p">])</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">short_gen</span><spanclass="p">)</span>
<spanclass="mf">0.5</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">short_gen</span><spanclass="p">)</span>
<spanclass="mf">1.0</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">short_gen</span><spanclass="p">)</span>
<spanclass="mf">1.5</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">short_gen</span><spanclass="p">)</span>
<spanclass="ne">StopIteration</span>
<spanclass="n">Traceback</span><spanclass="p">(</span><spanclass="n">most</span><spanclass="n">recent</span><spanclass="n">call</span><spanclass="n">last</span><spanclass="p">)</span>
<spanclass="o"><</span><spanclass="n">ipython</span><spanclass="o">-</span><spanclass="nb">input</span><spanclass="o">-</span><spanclass="mi">5</span><spanclass="o">-</span><spanclass="n">ed60a54ccf0b</span><spanclass="o">></span><spanclass="ow">in</span><spanclass="o"><</span><spanclass="n">module</span><spanclass="o">></span><spanclass="p">()</span>
<spanclass="o">----></span><spanclass="mi">1</span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">short_gen</span><spanclass="p">)</span>
<spanclass="ne">StopIteration</span><spanclass="p">:</span>
</pre></div>
</div>
<p>This is a great tool for retrieving content from a generator, or any iterator, without having to perform a for-loop over it.</p>
<divclass="section" id="Iterables-vs. Iterators">
<h3>Iterables vs. Iterators<aclass="headerlink" href="#Iterables-vs. Iterators" title="Permalink to this headline"></a></h3>
<p>This subsection is <em>not</em> essential to your basic understanding of the material. I am including it to prevent this text from being misleading to those who already know quite a bit about Python. <strong>This is a bit advanced, feel free to skip it…</strong></p>
<p>There is a bit of confusing terminology to be cleared up: an iterable is not the same thing as an iterator.</p>
<p>An <em>iterator</em> object stores its current state of iteration and “yields” each of its members in order, on demand via <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code>, until it is exhausted. As we’ve seen, a generator is an example of an iterator. We now must understand that every iterator is an iterable, but not every iterable is an iterator.</p>
<p>An <em>iterable</em> is an object that <em>can</em> be iterated over but does not necessarily have all the machinery of an iterator. For example, sequences (e.g lists, tuples, and strings) and other containers (e.g. dictionaries and sets) do not keep track of their own state of iteration. Thus you cannot call <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code> on one of these outright:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># a list is an example of an iterable that is *not*</span>
<spanclass="c1"># an iterator - you cannot call `next` on it.</span>
<spanclass="o">>>></span><spanclass="n">x</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">,</span><spanclass="mi">3</span><spanclass="p">]</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">x</span><spanclass="p">)</span>
<spanclass="o">---------------------------------------------------------------------------</span>
<spanclass="ne">TypeError</span><spanclass="n">Traceback</span><spanclass="p">(</span><spanclass="n">most</span><spanclass="n">recent</span><spanclass="n">call</span><spanclass="n">last</span><spanclass="p">)</span>
<spanclass="o"><</span><spanclass="n">ipython</span><spanclass="o">-</span><spanclass="nb">input</span><spanclass="o">-</span><spanclass="mi">19</span><spanclass="o">-</span><spanclass="n">b9d20096048c</span><spanclass="o">></span><spanclass="ow">in</span><spanclass="o"><</span><spanclass="n">module</span><spanclass="o">></span><spanclass="p">()</span>
<spanclass="o">----></span><spanclass="mi">1</span><spanclass="nb">next</span><spanclass="p">([</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">])</span>
<spanclass="ne">TypeError</span><spanclass="p">:</span><spanclass="s1">'list'</span><spanclass="nb">object</span><spanclass="ow">is</span><spanclass="ow">not</span><spanclass="n">an</span><spanclass="n">iterator</span>
</pre></div>
</div>
<p>In order to iterate over, say, a list you must first pass it to the built-in <codeclass="docutils literal notranslate"><spanclass="pre">iter</span></code> function. This function will return an <em>iterator</em> for that list, which stores its state of iteration and the instructions to yield each one of the list’s members:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># any iterable can be fed to `iter`</span>
<spanclass="c1"># to produce an iterator for that object</span>
<spanclass="o">>>></span><spanclass="n">x</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">,</span><spanclass="mi">3</span><spanclass="p">]</span>
<spanclass="o">>>></span><spanclass="n">x_it</span><spanclass="o">=</span><spanclass="nb">iter</span><spanclass="p">(</span><spanclass="n">x</span><spanclass="p">)</span><spanclass="c1"># `x_it` is an iterator</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">x_it</span><spanclass="p">)</span>
<spanclass="mi">1</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">x_it</span><spanclass="p">)</span>
<spanclass="mi">2</span>
<spanclass="o">>>></span><spanclass="nb">next</span><spanclass="p">(</span><spanclass="n">x_it</span><spanclass="p">)</span>
<spanclass="mi">3</span>
</pre></div>
</div>
<p>In this way, a list is an <em>iterable</em> but not an <em>iterator</em>, which is also the case for tuples, strings, sets, and dictionaries.</p>
<p>Python actually creates an iterator “behind the scenes”, whenever you perform a for-loop over an iterable like a list. It feeds that iterable to <codeclass="docutils literal notranslate"><spanclass="pre">iter</span></code>, and then proceeds to call <codeclass="docutils literal notranslate"><spanclass="pre">next</span></code> on the resulting iterator for each of the for-loop’s iterations.</p>
</div>
</div>
<divclass="section" id="List-&-Tuple-Comprehensions">
<h2>List & Tuple Comprehensions<aclass="headerlink" href="#List-&-Tuple-Comprehensions" title="Permalink to this headline"></a></h2>
<p>Using generator comprehensions to initialize lists is so useful that Python actually reserves a specialized syntax for it, known as the list comprehension. A <strong>list comprehension</strong> is a syntax for constructing a list, which exactly mirrors the generator comprehension syntax:</p>
<divclass="highlight-none notranslate"><divclass="highlight"><pre><span></span>[<expression> for <var> in <iterable> {if <condition}]
</pre></div>
</div>
<p>For example, if we want to create a list of square-numbers, we can simply write:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># a simple list comprehension</span>
<spanclass="o">>>></span><spanclass="p">[</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">10</span><spanclass="p">)]</span>
<spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">4</span><spanclass="p">,</span><spanclass="mi">9</span><spanclass="p">,</span><spanclass="mi">16</span><spanclass="p">,</span><spanclass="mi">25</span><spanclass="p">,</span><spanclass="mi">36</span><spanclass="p">,</span><spanclass="mi">49</span><spanclass="p">,</span><spanclass="mi">64</span><spanclass="p">,</span><spanclass="mi">81</span><spanclass="p">]</span>
</pre></div>
</div>
<p>This produces the exact same result as feeding the <codeclass="docutils literal notranslate"><spanclass="pre">list</span></code> function a generator comprehension. However, using a list comprehension is slightly more efficient than is feeding the <codeclass="docutils literal notranslate"><spanclass="pre">list</span></code> function a generator comprehension.</p>
<p>Let’s appreciate how economical list comprehensions are. The following code stores words that contain the letter “o”, in a list:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="n">words_with_o</span><spanclass="o">=</span><spanclass="p">[]</span>
<spanclass="n">word_collection</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="s1">'Python'</span><spanclass="p">,</span><spanclass="s1">'Like'</span><spanclass="p">,</span><spanclass="s1">'You'</span><spanclass="p">,</span><spanclass="s1">'Mean'</span><spanclass="p">,</span><spanclass="s1">'It'</span><spanclass="p">]</span>
<spanclass="k">for</span><spanclass="n">word</span><spanclass="ow">in</span><spanclass="n">word_collection</span><spanclass="p">:</span>
<spanclass="k">if</span><spanclass="s2">"o"</span><spanclass="ow">in</span><spanclass="n">word</span><spanclass="o">.</span><spanclass="n">lower</span><spanclass="p">():</span>
<spanclass="n">words_with_o</span><spanclass="o">.</span><spanclass="n">append</span><spanclass="p">(</span><spanclass="n">word</span><spanclass="p">)</span>
</pre></div>
</div>
<p>This can be written in a single line, using a list comprehension:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">word_collection</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="s1">'Python'</span><spanclass="p">,</span><spanclass="s1">'Like'</span><spanclass="p">,</span><spanclass="s1">'You'</span><spanclass="p">,</span><spanclass="s1">'Mean'</span><spanclass="p">,</span><spanclass="s1">'It'</span><spanclass="p">]</span>
<spanclass="gp">>>> </span><spanclass="n">words_with_o</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="n">word</span><spanclass="k">for</span><spanclass="n">word</span><spanclass="ow">in</span><spanclass="n">word_collection</span><spanclass="k">if</span><spanclass="s2">"o"</span><spanclass="ow">in</span><spanclass="n">word</span><spanclass="o">.</span><spanclass="n">lower</span><spanclass="p">()]</span>
<spanclass="gp">>>> </span><spanclass="n">words_with_o</span>
<spanclass="go">['Python', 'You']</span>
</pre></div>
</div>
<p>Tuples can be created using comprehension expressions too, but we must explicitly invoke the <codeclass="docutils literal notranslate"><spanclass="pre">tuple</span></code> constructor since parentheses are already reserved for defining a generator-comprehension.</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># creating a tuple using a comprehension expression</span>
<spanclass="o">>>></span><spanclass="nb">tuple</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="o">**</span><spanclass="mi">2</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">5</span><spanclass="p">))</span>
<spanclass="p">(</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">4</span><spanclass="p">,</span><spanclass="mi">9</span><spanclass="p">,</span><spanclass="mi">16</span><spanclass="p">)</span>
</pre></div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Takeaway</strong>:</p>
<p>The comprehensions-statement is an extremely useful syntax for creating simple and complicated lists and tuples alike.</p>
</div>
<divclass="section" id="Nesting-comprehensions">
<h3>Nesting comprehensions<aclass="headerlink" href="#Nesting-comprehensions" title="Permalink to this headline"></a></h3>
<p>It can be useful to nest comprehension expressions within one another, although this should be used sparingly.</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># Nested list comprehensions.</span>
<spanclass="c1"># This creates a 3x4 "matrix" (list of lists) of zeros.</span>
<spanclass="o">>>></span><spanclass="p">[[</span><spanclass="mi">0</span><spanclass="k">for</span><spanclass="n">col</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">4</span><spanclass="p">)]</span><spanclass="k">for</span><spanclass="n">row</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">3</span><spanclass="p">)]</span>
<spanclass="p">[[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">],</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">],</span><spanclass="p">[</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">]]</span>
</pre></div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: List Comprehensions</strong>:</p>
<p>Use a list comprehension to create a list that contains the string “hello” 100 times.</p>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Fancier List Comprehensions</strong>:</p>
<p>Use the inline <codeclass="docutils literal notranslate"><spanclass="pre">if-else</span></code> statement (discussed earlier in this module), along with a list comprehension, to create the list:</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="p">[</span><spanclass="s1">'hello'</span><spanclass="p">,</span>
<spanclass="s1">'goodbye'</span><spanclass="p">,</span>
<spanclass="s1">'hello'</span><spanclass="p">,</span>
<spanclass="s1">'goodbye'</span><spanclass="p">,</span>
<spanclass="s1">'hello'</span><spanclass="p">,</span>
<spanclass="s1">'goodbye'</span><spanclass="p">,</span>
<spanclass="s1">'hello'</span><spanclass="p">,</span>
<spanclass="s1">'goodbye'</span><spanclass="p">,</span>
<spanclass="s1">'hello'</span><spanclass="p">,</span>
<spanclass="s1">'goodbye'</span><spanclass="p">]</span>
</pre></div>
</div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Tuple Comprehensions</strong>:</p>
<p>Use a tuple-comprehension to extract comma-separated numbers from a string, converting them into a tuple of floats. I.e. <codeclass="docutils literal notranslate"><spanclass="pre">"3.2,2.4,99.8"</span></code> should become <codeclass="docutils literal notranslate"><spanclass="pre">(3.2,</span><spanclass="pre">2.4,</span><spanclass="pre">99.8)</span></code>. You will want to use the built-in string function <aclass="reference external" href="https://docs.python.org/3/library/stdtypes.html#str.split">str.split</a>.</p>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Translating a For-Loop</strong>:</p>
<p>Replicate the functionality of the the following code by writing a list comprehension.</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># skip all non-lowercased letters (including punctuation)</span>
<spanclass="c1"># append 1 if lowercase letter is equal to "o"</span>
<spanclass="c1"># append 0 if lowercase letter is not "o"</span>
<spanclass="n">out</span><spanclass="o">=</span><spanclass="p">[]</span>
<spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="s2">"Hello. How Are You?"</span><spanclass="p">:</span>
<spanclass="k">if</span><spanclass="n">i</span><spanclass="o">.</span><spanclass="n">islower</span><spanclass="p">():</span>
<spanclass="n">out</span><spanclass="o">.</span><spanclass="n">append</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">==</span><spanclass="s2">"o"</span><spanclass="k">else</span><spanclass="mi">0</span><spanclass="p">)</span>
</pre></div>
</div>
</div>
<divclass="admonition note">
<pclass="admonition-title fa fa-exclamation-circle"><strong>Reading Comprehension: Memory Efficiency</strong>:</p>
<p>Is there any difference in performance between the following expressions?</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># feeding `sum` a generator comprehension</span>
<spanclass="nb">sum</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="o">/</span><spanclass="n">n</span><spanclass="k">for</span><spanclass="n">n</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">101</span><spanclass="p">))</span>
</pre></div>
</div>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># feeding `sum` a list comprehension</span>
<spanclass="nb">sum</span><spanclass="p">([</span><spanclass="mi">1</span><spanclass="o">/</span><spanclass="n">n</span><spanclass="k">for</span><spanclass="n">n</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">101</span><spanclass="p">)])</span>
</pre></div>
</div>
<p>Is one expression preferable over the other? Why?</p>
</div>
</div>
</div>
<divclass="section" id="Links-to-Official-Documentation">
<h2>Links to Official Documentation<aclass="headerlink" href="#Links-to-Official-Documentation" title="Permalink to this headline"></a></h2>
<ulclass="simple">
<li><p><aclass="reference external" href="https://docs.python.org/3/glossary.html#term-generator">Generator definition</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/library/stdtypes.html#typesseq-range">range</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/tutorial/classes.html#generator-expressions">Generator comprehension expressions</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/glossary.html#term-iterator">Iterator definition</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/library/functions.html#next">next</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/library/functions.html#iter">iter</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions">List comprehensions</a></p></li>
<li><p><aclass="reference external" href="https://docs.python.org/3/tutorial/datastructures.html#nested-list-comprehensions">Nested list comprehensions</a></p></li>
</ul>
</div>
<divclass="section" id="Reading-Comprehension-Exercise-Solutions:">
<h2>Reading Comprehension Exercise Solutions:<aclass="headerlink" href="#Reading-Comprehension-Exercise-Solutions:" title="Permalink to this headline"></a></h2>
<p><strong>Using range: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="c1"># start=10, stop=0 (excluded), step-size=-1</span>
<spanclass="o">>>></span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">10</span><spanclass="p">,</span><spanclass="mi">0</span><spanclass="p">,</span><spanclass="o">-</span><spanclass="mi">1</span><spanclass="p">):</span>
<spanclass="o">>>></span><spanclass="nb">print</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">,</span><spanclass="n">end</span><spanclass="o">=</span><spanclass="s2">" "</span><spanclass="p">)</span><spanclass="c1"># the "end" parameter is to avoid each value taking up a new line</span>
<spanclass="mi">10</span><spanclass="mi">9</span><spanclass="mi">8</span><spanclass="mi">7</span><spanclass="mi">6</span><spanclass="mi">5</span><spanclass="mi">4</span><spanclass="mi">3</span><spanclass="mi">2</span><spanclass="mi">1</span>
</pre></div>
</div>
<p><strong>Writing a Generator Comprehension: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">g</span><spanclass="o">=</span><spanclass="p">((</span><spanclass="n">n</span><spanclass="p">,</span><spanclass="n">n</span><spanclass="o">+</span><spanclass="mi">2</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">n</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">6</span><spanclass="p">)</span><spanclass="k">if</span><spanclass="n">n</span><spanclass="o">!=</span><spanclass="mi">3</span><spanclass="p">)</span>
<spanclass="gp">>>> </span><spanclass="nb">list</span><spanclass="p">(</span><spanclass="n">g</span><spanclass="p">)</span><spanclass="c1"># convert into a list to print values</span>
<spanclass="go">[(0, 2), (1, 3), (2, 4), (4, 6), (5, 7)]</span>
</pre></div>
</div>
<p><strong>Using Generator Comprehensions on the Fly: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">1</span><spanclass="p">,</span><spanclass="mi">101</span><spanclass="p">,</span><spanclass="mi">2</span><spanclass="p">))</span>
<spanclass="go">2500</span>
</pre></div>
</div>
<p>or</p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="nb">sum</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">101</span><spanclass="p">)</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">%</span><spanclass="mi">2</span><spanclass="o">!=</span><spanclass="mi">0</span><spanclass="p">)</span>
<spanclass="go">2500</span>
</pre></div>
</div>
<p><strong>List Comprehensions: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="p">[</span><spanclass="s2">"hello"</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">100</span><spanclass="p">)]</span>
<spanclass="go">['hello', 'hello', ..., 'hello', 'hello'] # 100 hello's</span>
</pre></div>
</div>
<p><strong>Fancier List Comprehensions: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="p">[(</span><spanclass="s2">"hello"</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">%</span><spanclass="mi">2</span><spanclass="o">==</span><spanclass="mi">0</span><spanclass="k">else</span><spanclass="s2">"goodbye"</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="nb">range</span><spanclass="p">(</span><spanclass="mi">10</span><spanclass="p">)]</span>
<spanclass="go">['hello', 'goodbye', 'hello', 'goodbye', 'hello', 'goodbye', 'hello', 'goodbye', 'hello', 'goodbye']</span>
</pre></div>
</div>
<p><strong>Tuple Comprehension: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">string_of_nums</span><spanclass="o">=</span><spanclass="s2">"3.2, 2.4, 99.8"</span>
<spanclass="gp">>>> </span><spanclass="nb">tuple</span><spanclass="p">(</span><spanclass="nb">float</span><spanclass="p">(</span><spanclass="n">i</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="n">string_of_nums</span><spanclass="o">.</span><spanclass="n">split</span><spanclass="p">(</span><spanclass="s2">","</span><spanclass="p">))</span>
<spanclass="go">(3.2, 2.4, 99.8)</span>
</pre></div>
</div>
<p><strong>Translating a For-Loop: Solution</strong></p>
<divclass="highlight-python notranslate"><divclass="highlight"><pre><span></span><spanclass="gp">>>> </span><spanclass="n">out</span><spanclass="o">=</span><spanclass="p">[(</span><spanclass="mi">1</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">==</span><spanclass="s2">"o"</span><spanclass="k">else</span><spanclass="mi">0</span><spanclass="p">)</span><spanclass="k">for</span><spanclass="n">i</span><spanclass="ow">in</span><spanclass="s2">"Hello. How Are You?"</span><spanclass="k">if</span><spanclass="n">i</span><spanclass="o">.</span><spanclass="n">islower</span><spanclass="p">()]</span>
<spanclass="gp">>>> </span><spanclass="n">out</span>
<spanclass="go">[0, 0, 0, 1, 1, 0, 0, 0, 1, 0]</span>
</pre></div>
</div>
<p><strong>Memory Efficiency: Solution</strong></p>
<p>It is preferable to use the generator expression <codeclass="docutils literal notranslate"><spanclass="pre">sum(1/n</span><spanclass="pre">for</span><spanclass="pre">n</span><spanclass="pre">in</span><spanclass="pre">range(1,</span><spanclass="pre">101))</span></code>, rather than the list comprehension <codeclass="docutils literal notranslate"><spanclass="pre">sum([1/n</span><spanclass="pre">for</span><spanclass="pre">n</span><spanclass="pre">in</span><spanclass="pre">range(1,</span><spanclass="pre">101)])</span></code>. Using a list comprehension unnecessarily creates a list of the one hundred numbers, in memory, before feeding the list to <codeclass="docutils literal notranslate"><spanclass="pre">sum</span></code>. The generator expression need only produce a single value at a time, as <codeclass="docutils literal notranslate"><spanclass="pre">sum</span></code> iterates over it.</p>
</div>
</div>
</div>
</div>
<footer><divclass="rst-footer-buttons" role="navigation" aria-label="Footer">
<ahref="Iterables.html" class="btn btn-neutral float-left" title="Iterables" accesskey="p" rel="prev"><spanclass="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<ahref="Itertools.html" class="btn btn-neutral float-right" title="Python’s “Itertools”" accesskey="n" rel="next">Next <spanclass="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<divrole="contentinfo">
<p>© Copyright 2021, Ryan Soklaski.</p>
</div>
Built with <ahref="https://www.sphinx-doc.org/">Sphinx</a> using a
<ahref="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <ahref="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function(){
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
