feat: handle auto pagination for BigQuery v2 #3829

PhongChuong · 2025-06-06T14:53:35Z

Thank you for opening a Pull Request! Before submitting your PR, please read our contributing guidelines.

There are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes # b/360114900 & b/352331075

PhongChuong · 2025-06-06T15:57:02Z

I'm unsure which checks are relevant/needed. Please let me know,
Thank you,

blakeli0 · 2025-06-09T18:56:57Z

/gcbrun

blakeli0 · 2025-06-10T02:50:47Z

...com/google/api/generator/gapic/composer/common/AbstractServiceStubSettingsClassComposer.java

+      PAGINATE_MAX_RESULT_TYPES =
+          ImmutableMap.of(
+              "com.google.cloud.bigquery.v2",
+              ImmutableMap.of(


I guess there are some legacy RPCs use max_results field and are not of wrapper types?

Yes, there are total of 4 types. 2 wrapper types [UINT32VALUE, INT32VALUE] and 2 non wrapper type [uint32, int32].

Thanks! I think my question is more like do we need the value of the map to be another map (compared to a set) that contains the actual types? Looks like all the legacy RPCs are using wrapper types.

gapic-generator-java/src/test/proto/echo.proto

gapic-generator-java/src/test/proto/bigquery_jobs.proto

PhongChuong

Thanks for the review. PTAL.

gapic-generator-java/src/test/proto/echo.proto

PhongChuong · 2025-06-11T15:22:50Z

...com/google/api/generator/gapic/composer/common/AbstractServiceStubSettingsClassComposer.java

+      PAGINATE_MAX_RESULT_TYPES =
+          ImmutableMap.of(
+              "com.google.cloud.bigquery.v2",
+              ImmutableMap.of(


Yes, there are total of 4 types. 2 wrapper types [UINT32VALUE, INT32VALUE] and 2 non wrapper type [uint32, int32].

blakeli0 · 2025-06-16T14:31:16Z

...-generator-java/src/test/java/com/google/api/generator/test/protoloader/TestProtoLoader.java

+    Path serviceYamlPath = Paths.get(testFilesDirectory, serviceYamlFilename);
+    Optional<com.google.api.Service> serviceYamlOpt =
+        ServiceYamlParser.parse(serviceYamlPath.toString());
+    assertTrue(serviceYamlOpt.isPresent());


I think service yaml and service config json may not be required in this case? Can we pass empty ones to simplify the test cas?

Deleted service.yaml but it seems like config json is still needed.

blakeli0 · 2025-06-16T15:00:25Z

...com/google/api/generator/gapic/composer/common/AbstractServiceStubSettingsClassComposer.java

+      PAGINATE_MAX_RESULT_TYPES =
+          ImmutableMap.of(
+              "com.google.cloud.bigquery.v2",
+              ImmutableMap.of(


Thanks! I think my question is more like do we need the value of the map to be another map (compared to a set) that contains the actual types? Looks like all the legacy RPCs are using wrapper types.

blakeli0 · 2025-06-16T15:01:44Z

...com/google/api/generator/gapic/composer/common/AbstractServiceStubSettingsClassComposer.java

@@ -149,6 +149,23 @@ public abstract class AbstractServiceStubSettingsClassComposer implements ClassC

  private final TransportContext transportContext;

+  // Maps of package->methodName to types for pagination with max_results field.
+  private static final ImmutableMap<String, ImmutableMap<String, TypeNode>>
+      PAGINATE_MAX_RESULT_TYPES =


Can we make the name of this static field more bigquery specific? Same comment for PAGINATION_MAX_RESULTS_SERVICES_ALLOWLIST below.

Updated. PTAL.
I've also updated the dictionary to be a single map of types instead of map to map. Using a set is not possible because unwrapping requires to differentiate the 2 wrapper types.

gapic-generator-java/src/test/proto/echo.proto

blakeli0 · 2025-06-16T15:06:32Z

gapic-generator-java/src/test/proto/bigquery_jobs.proto

+  // in reverse chronological order, by job creation time. Requires the Can View
+  // project role, or the Is Owner project role if you set the allUsers
+  // property.
+  rpc ListJobs(ListJobsRequest) returns (JobList) {


Do you mind adding another test case here that is outside of the hardcoded list? To prove that other RPCs are not affected even if they are under the bigquery namespace?

Added deleteJob rpc as additional test case. PTAL.

PhongChuong · 2025-06-17T16:36:14Z

@blakeli0 , thank you for the review. I've updated the PR to include your suggestions. PTAL.

blakeli0 · 2025-06-17T17:54:03Z

/gcbrun

blakeli0 · 2025-06-17T19:50:11Z

/gcbrun

blakeli0 · 2025-06-18T15:41:37Z

/gcbrun

🤖 I have created a release *beep* *boop* --- <details><summary>2.60.0</summary> ## [2.60.0](v2.59.2...v2.60.0) (2025-06-23) ### Features * handle auto pagination for BigQuery v2 ([#3829](#3829)) ([025c84c](025c84c)) ### Dependencies * update google auth library dependencies to v1.37.1 ([#3846](#3846)) ([ea1d9e5](ea1d9e5)) * update google http client dependencies to v1.47.1 ([#3848](#3848)) ([a9a39d7](a9a39d7)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

feat: handle auto pagination for BigQuery v2

841aee8

product-auto-label bot added the size: l Pull request size is large. label Jun 6, 2025

PhongChuong marked this pull request as draft June 6, 2025 14:54

fix lint

b5498a0

PhongChuong marked this pull request as ready for review June 6, 2025 15:56

blakeli0 reviewed Jun 10, 2025

View reviewed changes

add golden unit tests

cfa71b7

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Jun 11, 2025

PhongChuong commented Jun 11, 2025

View reviewed changes

blakeli0 reviewed Jun 16, 2025

View reviewed changes

addressed PR comments

e429787

PhongChuong and others added 2 commits June 17, 2025 13:19

Merge branch 'main' into main

836bef6

Merge branch 'main' into main

e5b5691

blakeli0 approved these changes Jun 17, 2025

View reviewed changes

Merge branch 'main' into main

69ca436

blakeli0 merged commit 025c84c into googleapis:main Jun 20, 2025
37 of 39 checks passed

release-please bot mentioned this pull request Jun 20, 2025

chore(main): release 2.60.0 #3844

Merged

feat: handle auto pagination for BigQuery v2 #3829

feat: handle auto pagination for BigQuery v2 #3829

Uh oh!

Conversation

PhongChuong commented Jun 6, 2025

Uh oh!

PhongChuong commented Jun 6, 2025

Uh oh!

blakeli0 commented Jun 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

PhongChuong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PhongChuong Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PhongChuong commented Jun 17, 2025

Uh oh!

blakeli0 commented Jun 17, 2025

Uh oh!

blakeli0 commented Jun 17, 2025

Uh oh!

blakeli0 commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

PhongChuong Jun 17, 2025 •

edited

Loading