Premature ending of generator in list comprehension
I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?
The generator I created is as follows:
def inc_range(a,b):
for i in range(min(a,b), max(a,b) + 1):
yield i
The first way of calling is as follows:
[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)]
This gives me the following result:
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The second way of calling it is as follows:
a = inc_range(1,3)
b = inc_range(4,6)
[(i,j) for i in a for j in b]
This gives me the following:
[(1, 4), (1, 5), (1, 6)]
Experimenting around, the following two examples gave me the first result:
a = range(1,4)
b = range(4,7)
[(i,j) for i in a for j in b]
a = (i for i in range(1,4))
b = (i for i in range(4,7))
a = list(a)
b = list(b)
[(i,j) for i in a for j in b]
While the following gave me the second result again.
a = (i for i in range(1,4))
b = (i for i in range(4,7))
[(i,j) for i in a for j in b]
What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?
ANSWERS
Check out the following answers which helped me understand what is occurring here:
Alex Yu
mkrieger1
python generator list-comprehension
New contributor
|
show 3 more comments
I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?
The generator I created is as follows:
def inc_range(a,b):
for i in range(min(a,b), max(a,b) + 1):
yield i
The first way of calling is as follows:
[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)]
This gives me the following result:
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The second way of calling it is as follows:
a = inc_range(1,3)
b = inc_range(4,6)
[(i,j) for i in a for j in b]
This gives me the following:
[(1, 4), (1, 5), (1, 6)]
Experimenting around, the following two examples gave me the first result:
a = range(1,4)
b = range(4,7)
[(i,j) for i in a for j in b]
a = (i for i in range(1,4))
b = (i for i in range(4,7))
a = list(a)
b = list(b)
[(i,j) for i in a for j in b]
While the following gave me the second result again.
a = (i for i in range(1,4))
b = (i for i in range(4,7))
[(i,j) for i in a for j in b]
What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?
ANSWERS
Check out the following answers which helped me understand what is occurring here:
Alex Yu
mkrieger1
python generator list-comprehension
New contributor
unrelated - do not usefor x in ...: yield ..
- useyield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
@PatrickArtner didn't know aboutyield from
, thanks for the tip.
– bbminerva
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:j in inc_range(3,6)
vsb = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with justrange
and get results equal to "first way" or "second way". What was the point of all this? I'm confused
– Alex Yu
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
1
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago
|
show 3 more comments
I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?
The generator I created is as follows:
def inc_range(a,b):
for i in range(min(a,b), max(a,b) + 1):
yield i
The first way of calling is as follows:
[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)]
This gives me the following result:
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The second way of calling it is as follows:
a = inc_range(1,3)
b = inc_range(4,6)
[(i,j) for i in a for j in b]
This gives me the following:
[(1, 4), (1, 5), (1, 6)]
Experimenting around, the following two examples gave me the first result:
a = range(1,4)
b = range(4,7)
[(i,j) for i in a for j in b]
a = (i for i in range(1,4))
b = (i for i in range(4,7))
a = list(a)
b = list(b)
[(i,j) for i in a for j in b]
While the following gave me the second result again.
a = (i for i in range(1,4))
b = (i for i in range(4,7))
[(i,j) for i in a for j in b]
What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?
ANSWERS
Check out the following answers which helped me understand what is occurring here:
Alex Yu
mkrieger1
python generator list-comprehension
New contributor
I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?
The generator I created is as follows:
def inc_range(a,b):
for i in range(min(a,b), max(a,b) + 1):
yield i
The first way of calling is as follows:
[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)]
This gives me the following result:
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The second way of calling it is as follows:
a = inc_range(1,3)
b = inc_range(4,6)
[(i,j) for i in a for j in b]
This gives me the following:
[(1, 4), (1, 5), (1, 6)]
Experimenting around, the following two examples gave me the first result:
a = range(1,4)
b = range(4,7)
[(i,j) for i in a for j in b]
a = (i for i in range(1,4))
b = (i for i in range(4,7))
a = list(a)
b = list(b)
[(i,j) for i in a for j in b]
While the following gave me the second result again.
a = (i for i in range(1,4))
b = (i for i in range(4,7))
[(i,j) for i in a for j in b]
What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?
ANSWERS
Check out the following answers which helped me understand what is occurring here:
Alex Yu
mkrieger1
python generator list-comprehension
python generator list-comprehension
New contributor
New contributor
edited 1 hour ago
bbminerva
New contributor
asked 2 hours ago
bbminervabbminerva
414
414
New contributor
New contributor
unrelated - do not usefor x in ...: yield ..
- useyield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
@PatrickArtner didn't know aboutyield from
, thanks for the tip.
– bbminerva
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:j in inc_range(3,6)
vsb = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with justrange
and get results equal to "first way" or "second way". What was the point of all this? I'm confused
– Alex Yu
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
1
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago
|
show 3 more comments
unrelated - do not usefor x in ...: yield ..
- useyield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
@PatrickArtner didn't know aboutyield from
, thanks for the tip.
– bbminerva
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:j in inc_range(3,6)
vsb = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with justrange
and get results equal to "first way" or "second way". What was the point of all this? I'm confused
– Alex Yu
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
1
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago
unrelated - do not use
for x in ...: yield ..
- use yield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
unrelated - do not use
for x in ...: yield ..
- use yield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
@PatrickArtner didn't know about
yield from
, thanks for the tip.– bbminerva
1 hour ago
@PatrickArtner didn't know about
yield from
, thanks for the tip.– bbminerva
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:
j in inc_range(3,6)
vs b = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with just range
and get results equal to "first way" or "second way". What was the point of all this? I'm confused– Alex Yu
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:
j in inc_range(3,6)
vs b = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with just range
and get results equal to "first way" or "second way". What was the point of all this? I'm confused– Alex Yu
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
1
1
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago
|
show 3 more comments
3 Answers
3
active
oldest
votes
And the answer is in:
PEP 289 -- Generator Expressions. The details
Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:
g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4)
is equivalent to:
def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen
Clarification. Why PEP-289 and not PEP-202?
Because comprehensions on lists works in another way.
This would be example for comprehensions on lists:
a = list(inc_range(1,3))
b = list(inc_range(4,6))
[(i,j) for i in a for j in b]
And there will be no "wonders": everything will be as expected.
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And usinggenerator
-functions changes way of evaluation from first to last (not outermost only)
– Alex Yu
1 hour ago
add a comment |
To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.
But, after the first run, the "inner" generator is exhausted and cannot be run again.
Adding a print
illustrates this (simplifying the example):
>>> def inc(a, b):
... for i in range(a, b):
... print(i)
... yield i
...
>>> a = inc(1, 4)
>>> b = inc(4, 7)
>>> [(i, j) for i in a for j in b]
1 # <-- a begins to run
4 # <-- b begins to run
5
6 # <-- b exhausted here
2 # <-- a continued, but not resulting in list item, because lacking value from b
3
[(1, 4), (1, 5), (1, 6)]
The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:
>>> def inc(a, b):
... print('started', a, b)
... for i in range(a, b):
... yield i
...
>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]
started 1 4
started 4 7
started 4 7
started 4 7
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The reason why using range
objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
add a comment |
A generator is an iterable object hence when you call it outside of a list it returns the next item alone.
a = inc_range(1,3)
b = inc_range(4,6)
c = inc_range(7,9)
[(i,j,k) for i in a for j in b for k in c]
This will only yield the elements for k in c when run
Hence you need to iterate through all its objects as you have when defining it as an array.
[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]
This forces the generator to yield all values in each of the iterations.
I don't think I quite understand the example. All three are iterators, and should iterate until they raise aStopIteration
. I thought thefor i in a
would keep callinga
until aStopIteration
is thrown, much like what occurs when I turn it into a list first.
– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
bbminerva is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54895114%2fpremature-ending-of-generator-in-list-comprehension%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
And the answer is in:
PEP 289 -- Generator Expressions. The details
Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:
g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4)
is equivalent to:
def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen
Clarification. Why PEP-289 and not PEP-202?
Because comprehensions on lists works in another way.
This would be example for comprehensions on lists:
a = list(inc_range(1,3))
b = list(inc_range(4,6))
[(i,j) for i in a for j in b]
And there will be no "wonders": everything will be as expected.
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And usinggenerator
-functions changes way of evaluation from first to last (not outermost only)
– Alex Yu
1 hour ago
add a comment |
And the answer is in:
PEP 289 -- Generator Expressions. The details
Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:
g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4)
is equivalent to:
def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen
Clarification. Why PEP-289 and not PEP-202?
Because comprehensions on lists works in another way.
This would be example for comprehensions on lists:
a = list(inc_range(1,3))
b = list(inc_range(4,6))
[(i,j) for i in a for j in b]
And there will be no "wonders": everything will be as expected.
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And usinggenerator
-functions changes way of evaluation from first to last (not outermost only)
– Alex Yu
1 hour ago
add a comment |
And the answer is in:
PEP 289 -- Generator Expressions. The details
Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:
g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4)
is equivalent to:
def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen
Clarification. Why PEP-289 and not PEP-202?
Because comprehensions on lists works in another way.
This would be example for comprehensions on lists:
a = list(inc_range(1,3))
b = list(inc_range(4,6))
[(i,j) for i in a for j in b]
And there will be no "wonders": everything will be as expected.
And the answer is in:
PEP 289 -- Generator Expressions. The details
Only the outermost for-expression is evaluated immediately, the other expressions are deferred until the generator is run:
g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4)
is equivalent to:
def __gen(bound_exp):
for var1 in bound_exp:
if exp2:
for var2 in exp3:
if exp4:
yield tgtexp
g = __gen(iter(exp1))
del __gen
Clarification. Why PEP-289 and not PEP-202?
Because comprehensions on lists works in another way.
This would be example for comprehensions on lists:
a = list(inc_range(1,3))
b = list(inc_range(4,6))
[(i,j) for i in a for j in b]
And there will be no "wonders": everything will be as expected.
edited 1 hour ago
answered 1 hour ago
Alex YuAlex Yu
1,9991522
1,9991522
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And usinggenerator
-functions changes way of evaluation from first to last (not outermost only)
– Alex Yu
1 hour ago
add a comment |
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And usinggenerator
-functions changes way of evaluation from first to last (not outermost only)
– Alex Yu
1 hour ago
This would be the answer! Thanks!
– bbminerva
1 hour ago
This would be the answer! Thanks!
– bbminerva
1 hour ago
1
1
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
You asked excellent question. I was somewhat confused at first. What you asked - is actually question about generator-expressions not list-comprehensions
– Alex Yu
1 hour ago
2
2
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
Good findings. The important point, IMO, is that the inner expression not only is deferred, but it is evaluated again in each iteration of the outer expression.
– mkrieger1
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And using
generator
-functions changes way of evaluation from first to last (not outermost only)– Alex Yu
1 hour ago
@mkrieger1 As I read now Early Binding versus Late Binding there was discussion with pro- and contra- of such behaviour. And using
generator
-functions changes way of evaluation from first to last (not outermost only)– Alex Yu
1 hour ago
add a comment |
To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.
But, after the first run, the "inner" generator is exhausted and cannot be run again.
Adding a print
illustrates this (simplifying the example):
>>> def inc(a, b):
... for i in range(a, b):
... print(i)
... yield i
...
>>> a = inc(1, 4)
>>> b = inc(4, 7)
>>> [(i, j) for i in a for j in b]
1 # <-- a begins to run
4 # <-- b begins to run
5
6 # <-- b exhausted here
2 # <-- a continued, but not resulting in list item, because lacking value from b
3
[(1, 4), (1, 5), (1, 6)]
The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:
>>> def inc(a, b):
... print('started', a, b)
... for i in range(a, b):
... yield i
...
>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]
started 1 4
started 4 7
started 4 7
started 4 7
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The reason why using range
objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
add a comment |
To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.
But, after the first run, the "inner" generator is exhausted and cannot be run again.
Adding a print
illustrates this (simplifying the example):
>>> def inc(a, b):
... for i in range(a, b):
... print(i)
... yield i
...
>>> a = inc(1, 4)
>>> b = inc(4, 7)
>>> [(i, j) for i in a for j in b]
1 # <-- a begins to run
4 # <-- b begins to run
5
6 # <-- b exhausted here
2 # <-- a continued, but not resulting in list item, because lacking value from b
3
[(1, 4), (1, 5), (1, 6)]
The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:
>>> def inc(a, b):
... print('started', a, b)
... for i in range(a, b):
... yield i
...
>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]
started 1 4
started 4 7
started 4 7
started 4 7
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The reason why using range
objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
add a comment |
To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.
But, after the first run, the "inner" generator is exhausted and cannot be run again.
Adding a print
illustrates this (simplifying the example):
>>> def inc(a, b):
... for i in range(a, b):
... print(i)
... yield i
...
>>> a = inc(1, 4)
>>> b = inc(4, 7)
>>> [(i, j) for i in a for j in b]
1 # <-- a begins to run
4 # <-- b begins to run
5
6 # <-- b exhausted here
2 # <-- a continued, but not resulting in list item, because lacking value from b
3
[(1, 4), (1, 5), (1, 6)]
The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:
>>> def inc(a, b):
... print('started', a, b)
... for i in range(a, b):
... yield i
...
>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]
started 1 4
started 4 7
started 4 7
started 4 7
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The reason why using range
objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.
To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.
But, after the first run, the "inner" generator is exhausted and cannot be run again.
Adding a print
illustrates this (simplifying the example):
>>> def inc(a, b):
... for i in range(a, b):
... print(i)
... yield i
...
>>> a = inc(1, 4)
>>> b = inc(4, 7)
>>> [(i, j) for i in a for j in b]
1 # <-- a begins to run
4 # <-- b begins to run
5
6 # <-- b exhausted here
2 # <-- a continued, but not resulting in list item, because lacking value from b
3
[(1, 4), (1, 5), (1, 6)]
The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:
>>> def inc(a, b):
... print('started', a, b)
... for i in range(a, b):
... yield i
...
>>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)]
started 1 4
started 4 7
started 4 7
started 4 7
[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)]
The reason why using range
objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.
answered 1 hour ago
mkrieger1mkrieger1
4,63821932
4,63821932
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
add a comment |
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Thanks for the detailed explanation!
– bbminerva
1 hour ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
Your point about "again" - is great. I remember now that I had similar confusion as OP many years ago and then forgot till today
– Alex Yu
57 mins ago
add a comment |
A generator is an iterable object hence when you call it outside of a list it returns the next item alone.
a = inc_range(1,3)
b = inc_range(4,6)
c = inc_range(7,9)
[(i,j,k) for i in a for j in b for k in c]
This will only yield the elements for k in c when run
Hence you need to iterate through all its objects as you have when defining it as an array.
[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]
This forces the generator to yield all values in each of the iterations.
I don't think I quite understand the example. All three are iterators, and should iterate until they raise aStopIteration
. I thought thefor i in a
would keep callinga
until aStopIteration
is thrown, much like what occurs when I turn it into a list first.
– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
add a comment |
A generator is an iterable object hence when you call it outside of a list it returns the next item alone.
a = inc_range(1,3)
b = inc_range(4,6)
c = inc_range(7,9)
[(i,j,k) for i in a for j in b for k in c]
This will only yield the elements for k in c when run
Hence you need to iterate through all its objects as you have when defining it as an array.
[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]
This forces the generator to yield all values in each of the iterations.
I don't think I quite understand the example. All three are iterators, and should iterate until they raise aStopIteration
. I thought thefor i in a
would keep callinga
until aStopIteration
is thrown, much like what occurs when I turn it into a list first.
– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
add a comment |
A generator is an iterable object hence when you call it outside of a list it returns the next item alone.
a = inc_range(1,3)
b = inc_range(4,6)
c = inc_range(7,9)
[(i,j,k) for i in a for j in b for k in c]
This will only yield the elements for k in c when run
Hence you need to iterate through all its objects as you have when defining it as an array.
[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]
This forces the generator to yield all values in each of the iterations.
A generator is an iterable object hence when you call it outside of a list it returns the next item alone.
a = inc_range(1,3)
b = inc_range(4,6)
c = inc_range(7,9)
[(i,j,k) for i in a for j in b for k in c]
This will only yield the elements for k in c when run
Hence you need to iterate through all its objects as you have when defining it as an array.
[(i,j,k) for i in inc_range(1,3) for j in inc_range(3,6) for k in inc_range(7,9)]
This forces the generator to yield all values in each of the iterations.
answered 1 hour ago
James CarronJames Carron
618
618
I don't think I quite understand the example. All three are iterators, and should iterate until they raise aStopIteration
. I thought thefor i in a
would keep callinga
until aStopIteration
is thrown, much like what occurs when I turn it into a list first.
– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
add a comment |
I don't think I quite understand the example. All three are iterators, and should iterate until they raise aStopIteration
. I thought thefor i in a
would keep callinga
until aStopIteration
is thrown, much like what occurs when I turn it into a list first.
– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
I don't think I quite understand the example. All three are iterators, and should iterate until they raise a
StopIteration
. I thought the for i in a
would keep calling a
until a StopIteration
is thrown, much like what occurs when I turn it into a list first.– bbminerva
1 hour ago
I don't think I quite understand the example. All three are iterators, and should iterate until they raise a
StopIteration
. I thought the for i in a
would keep calling a
until a StopIteration
is thrown, much like what occurs when I turn it into a list first.– bbminerva
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
@Alex Yu describes this well
– James Carron
1 hour ago
add a comment |
bbminerva is a new contributor. Be nice, and check out our Code of Conduct.
bbminerva is a new contributor. Be nice, and check out our Code of Conduct.
bbminerva is a new contributor. Be nice, and check out our Code of Conduct.
bbminerva is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54895114%2fpremature-ending-of-generator-in-list-comprehension%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
unrelated - do not use
for x in ...: yield ..
- useyield from range(min(a,b), max(a,b) + 1)
– Patrick Artner
1 hour ago
@PatrickArtner didn't know about
yield from
, thanks for the tip.– bbminerva
1 hour ago
Maybe I'm missing something. Your first way and second way are different only in one thing:
j in inc_range(3,6)
vsb = inc_range(4,6)
. Obviously you get different results. Then you make analogous operations with justrange
and get results equal to "first way" or "second way". What was the point of all this? I'm confused– Alex Yu
1 hour ago
@AlexYu that was a typo copying over, they're both inc_range(4,6)
– bbminerva
1 hour ago
1
@smac89 a typo from the question being simplified -- I'll fix it.
– bbminerva
1 hour ago