feat: Support `axis=1` in `df.apply` for scalar outputs #629

shobsi · 2024-04-22T17:26:19Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)
- Note in remote_function doc https://screenshot.googleplex.com/4c6ZojpEYf7Wymj
- Overloading of the arg input_type https://screenshot.googleplex.com/8TZGXJLw8rewd6W
- Note and description update for DataFrame.apply https://screenshot.googleplex.com/9jLWmQNUYpoVrTA
- Code sample and arg update in DataFrame.apply https://screenshot.googleplex.com/7nHi5ZjsGyTruTo

Fixes internal issue 320674749 🦕

…-arr-in

bigframes/core/blocks.py

bigframes/functions/remote_function.py

bigframes/core/blocks.py

tests/system/small/test_remote_function.py

…-arr-in

bigframes/core/blocks.py

bigframes/dataframe.py

bigframes/functions/remote_function.py

…ailure

…-arr-in

bigframes/functions/remote_function.py

bigframes/core/blocks.py

bigframes/functions/remote_function.py

…-arr-in

deserialization

…-arr-in

tswast · 2024-05-17T19:12:58Z

bigframes/dataframe.py

+            # in the remote function
+            # NOTE: Keep in sync with the value converters used in the gcf code
+            # generated in generate_cloud_function_main_code in remote_function.py
+            remote_function_supported_dtypes = (


Ideally we'd document these limitations, too, right?

Adding in #800

feat: Support axis=1 in df.apply for scalar outputs

4d3200c

shobsi requested review from a team as code owners April 22, 2024 17:26

shobsi requested a review from DevStephanie April 22, 2024 17:26

blunderbuss-gcf bot assigned chelsea-lin Apr 22, 2024

shobsi removed the request for review from DevStephanie April 22, 2024 17:45

shobsi marked this pull request as draft April 22, 2024 17:46

shobsi added 2 commits April 22, 2024 23:14

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

ceb7dbc

…-arr-in

avoid mixing other changes in the input_types param

22b7f32

product-auto-label bot added the size: m Pull request size is medium. label Apr 22, 2024

use guid instead of hard coded column name

5049170

product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. label Apr 23, 2024

shobsi added 2 commits April 23, 2024 08:28

check_exact=False to avoid failing system_prerelease

4f28f56

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

fb92f34

…-arr-in

product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Apr 23, 2024

shobsi added 2 commits April 24, 2024 17:54

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

2e2f32c

…-arr-in

handle index in remote function, add large system tests

99e3e7b

shobsi marked this pull request as ready for review April 24, 2024 21:21

shobsi requested a review from TrevorBergeron April 24, 2024 21:22

shobsi added 2 commits April 25, 2024 02:24

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

85efbb7

…-arr-in

make the test case more robust

7153db8

TrevorBergeron requested changes Apr 25, 2024

View reviewed changes

shobsi added 7 commits April 26, 2024 00:15

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

13c2e62

…-arr-in

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

c3dddd8

…-arr-in

handle non-string column names, add unsupported dtype tests

5fb8148

fix import

edbac1b

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

74aeaea

…-arr-in

use _cached in df.apply to catch any rf execution errors early

d3c07e9

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

ed03d28

…-arr-in

shobsi added 4 commits May 2, 2024 19:14

introduce early check on the supported dtypes

eb383f3

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

d520337

…-arr-in

asjust test after early dtype handling

7a3aa5f

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

4d39204

…-arr-in

TrevorBergeron reviewed May 2, 2024

View reviewed changes

GarrettWu mentioned this pull request May 3, 2024

Error when applying remote function with multiple parameters #646

Closed

shobsi added 11 commits May 4, 2024 04:03

address review comments

7383faf

Merge remote-tracking branch 'refs/remotes/github/main'

a8f036a

user NameError for column name parsing issue, address test coverage f…

84d719c

…ailure

address nan return handling in the gcf code

4e96b96

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

4ce3cc9

…-arr-in

handle (nan, inf, -inf)

612055d

replace "row" by bpd.Series for input types

1c58ded

make the bq parity assert more readable

bede078

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

56a8236

…-arr-in

fix the series name before assert

3409bc3

fix docstring for args

ea7e28e

TrevorBergeron reviewed May 8, 2024

View reviewed changes

bigframes/functions/remote_function.py Outdated Show resolved Hide resolved

bigframes/core/blocks.py Outdated Show resolved Hide resolved

bigframes/functions/remote_function.py Show resolved Hide resolved

shobsi added 4 commits May 8, 2024 23:40

move more low level string logic in sql module

14602f8

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

7f5f2a3

…-arr-in

raise explicit error when a column name cannot be supported

3bf5bee

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

0149d59

…-arr-in

TrevorBergeron approved these changes May 10, 2024

View reviewed changes

shobsi added 2 commits May 10, 2024 17:22

keep literal_eval check on the serialization side to match

b5f3232

deserialization

Merge remote-tracking branch 'refs/remotes/github/main' into shobs-rf…

bad6df6

…-arr-in

shobsi merged commit f6bdc4a into main May 10, 2024
20 of 21 checks passed

shobsi deleted the shobs-rf-arr-in branch May 10, 2024 19:50

release-please bot mentioned this pull request May 10, 2024

chore(main): release 1.6.0 #667

Merged

tswast reviewed May 17, 2024

View reviewed changes

shobsi mentioned this pull request Jun 26, 2024

docs: document dtype limitation on row processing remote_function #800

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Support `axis=1` in `df.apply` for scalar outputs #629

feat: Support `axis=1` in `df.apply` for scalar outputs #629

Uh oh!

shobsi commented Apr 22, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast May 17, 2024

Uh oh!

shobsi Jun 26, 2024

Uh oh!

Uh oh!

feat: Support axis=1 in df.apply for scalar outputs #629

feat: Support axis=1 in df.apply for scalar outputs #629

Uh oh!

Conversation

shobsi commented Apr 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tswast May 17, 2024

Choose a reason for hiding this comment

Uh oh!

shobsi Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feat: Support `axis=1` in `df.apply` for scalar outputs #629

feat: Support `axis=1` in `df.apply` for scalar outputs #629

shobsi commented Apr 22, 2024 •

edited

Loading