Skip to content

BUG: index of group not returned correctly in groupby.apply #22541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
h-vetinari opened this issue Aug 30, 2018 · 3 comments · Fixed by #33096
Closed

BUG: index of group not returned correctly in groupby.apply #22541

h-vetinari opened this issue Aug 30, 2018 · 3 comments · Fixed by #33096
Labels
Apply Apply, Aggregate, Transform, Map Bug Groupby
Milestone
1.1

Comments

@h-vetinari
Copy link
Contributor

h-vetinari commented Aug 30, 2018 •
edited
Loading

Returning the index of a group is admittedly a somewhat unusual use of apply (since the information is available in groups), but it's clearly legal and shouldn't be wrong.

N = 10
df = pd.DataFrame(np.random.randint(0, int(N/3), (N,)) + 10, columns=['id'])
df
#    id
# 0  11
# 1  11
# 2  11
# 3  10
# 4  11
# 5  12
# 6  12
# 7  12
# 8  10
# 9  12

The issue is that the result of last group gets wrongly broadcast to all groups

df.groupby('id', as_index=True).apply(lambda gr: gr.index)
# id
# 10    Int64Index([5, 6, 7, 9], dtype='int64')
# 11    Int64Index([5, 6, 7, 9], dtype='int64')
# 12    Int64Index([5, 6, 7, 9], dtype='int64')
# dtype: object

Interestingly, with adding any operation I've tried, the behaviour is correct again:

df.groupby('id', as_index=True).apply(lambda gr: gr.index + 1 - 1)
# id
# 10          Int64Index([3, 8], dtype='int64')
# 11    Int64Index([0, 1, 2, 4], dtype='int64')
# 12    Int64Index([5, 6, 7, 9], dtype='int64')
# dtype: object
@WillAyd
Copy link
Member

WillAyd commented Aug 30, 2018

Think you may have a typo in your example - can you double check?

@WillAyd WillAyd added Groupby Needs Info Clarification about behavior needed to assess issue labels Aug 30, 2018
@h-vetinari
Copy link
Contributor Author

h-vetinari commented Aug 30, 2018 •
edited
Loading

@WillAyd Sorry, kept simplifying the example and messed up. Code working now, issue persists.

@WillAyd WillAyd added Bug and removed Needs Info Clarification about behavior needed to assess issue labels Aug 30, 2018
@WillAyd
Copy link
Member

WillAyd commented Aug 30, 2018

OK thanks. As you mentioned this is a pretty unusual use of apply but nonetheless PRs and investigation are always welcome

@h-vetinari h-vetinari mentioned this issue Aug 30, 2018
Open
@h-vetinari h-vetinari mentioned this issue Nov 18, 2018
Closed
@h-vetinari h-vetinari mentioned this issue Jan 28, 2019
Closed
@jbrockmendel jbrockmendel added the Apply Apply, Aggregate, Transform, Map label Dec 1, 2019
phofl added a commit to phofl/pandas that referenced this issue Mar 28, 2020
@phofl
TEST: Add test for pandas-dev#22541 to ensure that error does not occ…
5a381c3 
…ur again
@phofl phofl mentioned this issue Mar 28, 2020
Merged
4 tasks
@jreback jreback added this to the 1.1 milestone Mar 30, 2020
jreback pushed a commit that referenced this issue Mar 30, 2020
@phofl
TEST: Add test for #22541 to ensure that error does not occur again (#…
4f89c26 
…33096)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Apply Apply, Aggregate, Transform, Map Bug Groupby
Projects
None yet
Milestone
1.1
Development

Successfully merging a pull request may close this issue.

TEST: Add test for #22541 to ensure that error does not occur again phofl/pandas
4 participants
@WillAyd @jreback @jbrockmendel @h-vetinari