Skip to content

chore(deps): update apache beam packages to v2.34.0 #5875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

chore(deps): update apache beam packages to v2.34.0 #5875

wants to merge 2 commits into from

Conversation

renovate-bot
Copy link
Contributor

@renovate-bot renovate-bot commented Aug 26, 2021 •
edited
Loading

WhiteSource Renovate

This PR contains the following updates:

Package Change Age Adoption Passing Confidence
org.apache.beam:beam-sdks-java-extensions-sql (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-sdks-java-io-google-cloud-platform (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-runners-google-cloud-dataflow-java (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-runners-direct-java (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-sdks-java-core (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-sdks-java-io-kafka (source) 2.31.0 -> 2.34.0 age adoption passing confidence
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core (source) 2.31.0 -> 2.34.0 age adoption passing confidence

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

  • If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by WhiteSource Renovate. View repository job log here.

@forking-renovate forking-renovate bot added the automerge Merge the pull request once unit tests and other checks pass. label Aug 26, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Aug 26, 2021
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Aug 26, 2021
@trusted-contributions-gcf trusted-contributions-gcf bot added kokoro:force-run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Aug 26, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 26, 2021
@gcf-merge-on-green
Copy link
Contributor

Merge-on-green attempted to merge your PR for 6 hours, but it was not mergeable because either one of your required status checks failed, one of your required reviews was not approved, or there is a do not merge label. Learn more about your required status checks here: https://help.github.com/en/github/administering-a-repository/enabling-required-status-checks. You can remove and reapply the label to re-run the bot.

@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Aug 26, 2021
@lesv
Copy link
Contributor

lesv commented Aug 27, 2021

Java 8

#
# dataflow/transforms
#

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.417 s <<< FAILURE! - in com.example.AvroToCsvTest
[ERROR] com.example.AvroToCsvTest.testAvroToCsv  Time elapsed: 3.417 s  <<< ERROR!
java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.427 s <<< FAILURE! - in com.example.CsvToAvroTest
[ERROR] com.example.CsvToAvroTest.testCsvToAvro  Time elapsed: 3.427 s  <<< ERROR!
java.lang.ExceptionInInitializerError

Caused by: java.lang.IllegalStateException: No logging platforms found:
com.google.common.flogger.backend.system.DefaultPlatform: java.lang.NoClassDefFoundError: com/google/common/flogger/backend/NoOpContextDataProvider
	at com.google.common.flogger.backend.Platform$LazyHolder.loadFirstAvailablePlatform(Platform.java:96)

@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from 408d9ea to 1b6efe8 Compare August 28, 2021 00:00
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 28, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 28, 2021
@eaball35 eaball35 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 7, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from 1b6efe8 to dcdf536 Compare September 13, 2021 19:41
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 13, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from dcdf536 to b896ca1 Compare September 20, 2021 22:11
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 20, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from b896ca1 to c1f6f8e Compare October 6, 2021 21:28
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 6, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 6, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from c1f6f8e to dcf7441 Compare October 7, 2021 02:49
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 7, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 7, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from dcf7441 to ca5b6d4 Compare October 7, 2021 10:14
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 7, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 7, 2021
@stephaniewang526 stephaniewang526 removed their assignment Oct 12, 2021
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 19, 2021
@kokoro-team kokoro-team removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Nov 3, 2021
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 7, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 7, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from b2fe7b2 to 09ef8ed Compare December 8, 2021 00:36
@renovate-bot renovate-bot changed the title chore(deps): update apache beam packages to v2.32.0 chore(deps): update apache beam packages to v2.34.0 Dec 8, 2021
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 8, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 8, 2021
@renovate-bot renovate-bot force-pushed the renovate/apache-beam-packages branch from 09ef8ed to b834548 Compare December 13, 2021 22:30
@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2021
@anguillanneuf
Copy link
Member

@davidcavazos Could you take a look at SpannerWriteIT.testEndToEnd?

@anguillanneuf anguillanneuf self-requested a review December 14, 2021 00:53
@davidcavazos
Copy link
Contributor

#6579 removed the dataflow/transforms sample, so that shouldn't be an issue anymore. The current failures are both a null pointer exception, so at least it looks like it's failing consistently in both Java 8 and 11.

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerWriteIT.testEndToEnd(SpannerWriteIT.java:116)
Caused by: java.lang.NullPointerException

@davidcavazos
Copy link
Contributor 

[ERROR] Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 80.594 s <<< FAILURE! - in com.example.dataflow.SpannerReadIT
[ERROR] com.example.dataflow.SpannerReadIT.readDbEndToEnd  Time elapsed: 23.789 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerReadIT.readDbEndToEnd(SpannerReadIT.java:165)
Caused by: java.lang.NullPointerException

[ERROR] com.example.dataflow.SpannerReadIT.reaTransactionalReadEndToEnd  Time elapsed: 17.293 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerReadIT.reaTransactionalReadEndToEnd(SpannerReadIT.java:215)
Caused by: java.lang.NullPointerException

[ERROR] com.example.dataflow.SpannerReadIT.readTableEndToEnd  Time elapsed: 27.383 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerReadIT.readTableEndToEnd(SpannerReadIT.java:181)
Caused by: java.lang.NullPointerException

[ERROR] com.example.dataflow.SpannerReadIT.readApiEndToEnd  Time elapsed: 11.689 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerReadIT.readApiEndToEnd(SpannerReadIT.java:198)
Caused by: java.lang.NullPointerException

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 18.53 s <<< FAILURE! - in com.example.dataflow.SpannerGroupWriteIT
[ERROR] com.example.dataflow.SpannerGroupWriteIT.testEndToEnd  Time elapsed: 18.515 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerGroupWriteIT.testEndToEnd(SpannerGroupWriteIT.java:134)
Caused by: java.lang.NullPointerException

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 24.291 s <<< FAILURE! - in com.example.dataflow.SpannerWriteIT
[ERROR] com.example.dataflow.SpannerWriteIT.testEndToEnd  Time elapsed: 24.286 s  <<< ERROR!
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.NullPointerException
	at dataflow.spanner@1.0-SNAPSHOT/com.example.dataflow.SpannerWriteIT.testEndToEnd(SpannerWriteIT.java:116)
Caused by: java.lang.NullPointerException

[ERROR] Errors:
[ERROR]   SpannerGroupWriteIT.testEndToEnd:134 » PipelineExecution java.lang.NullPointer...
[ERROR]   SpannerReadIT.reaTransactionalReadEndToEnd:215 » PipelineExecution java.lang.N...
[ERROR]   SpannerReadIT.readApiEndToEnd:198 » PipelineExecution java.lang.NullPointerExc...
[ERROR]   SpannerReadIT.readDbEndToEnd:165 » PipelineExecution java.lang.NullPointerExce...
[ERROR]   SpannerReadIT.readTableEndToEnd:181 » PipelineExecution java.lang.NullPointerE...
[ERROR]   SpannerWriteIT.testEndToEnd:116 » PipelineExecution java.lang.NullPointerExcep...
[ERROR] Tests run: 6, Failures: 0, Errors: 6, Skipped: 0

@davidcavazos
Copy link
Contributor

All errors seem to be failing when calling the class main method, which makes me think the sample objects themselves are null.

@davidcavazos
Copy link
Contributor

davidcavazos commented Dec 14, 2021 •
edited
Loading

The Java tests at the Beam repo seem to be passing, including the SpannerIO tests.

@jsimonweb, since you're more familiar with these samples and SpannerIO, do you have any idea what's happening? Do you think Beam 2.34 might be bringing some other dependency version incompatibility?

@jsimonweb
Copy link
Contributor

Looping in @skuruppu - would you please take a look to see if you can help to resolve these test errors?

@skuruppu
Copy link
Contributor

@ansh0l will take a look.

@trusted-contributions-gcf trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 15, 2021
@kokoro-team kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 15, 2021
@anguillanneuf
Copy link
Member

Thank you @ansh0l - I recall mvn dependency:tree with beam 2.31.0 still prints out log4j-api as a direct dependency but beam 2.34.0 no longer does. We should aim to update these samples a.s.a.p.

@anguillanneuf
Copy link
Member

anguillanneuf commented Dec 17, 2021 •
edited
Loading

New comms from Dataflow team as of Dec 17, 2021:

Apache Beam versions do not depend on the impacted Log4j 2 dependency (log4j-core).
The following recommendation provided in our prior communication is no longer necessary based on our assessment.

Users using Apache Beam version 2.31.0 or older should update all Dataflow pipelines to Apache Beam version 2.32.0 or newer.

@davidcavazos davidcavazos mentioned this pull request Dec 21, 2021
Merged
9 tasks
@lesv
Copy link
Contributor

lesv commented Dec 21, 2021

Let's wait for next time.

@lesv lesv closed this Dec 21, 2021
@forking-renovate
Copy link

Renovate Ignore Notification

As this PR has been closed unmerged, Renovate will now ignore this update (2.34.0). You will still receive a PR once a newer version is released, so if you wish to permanently ignore this dependency, please add it to the ignoreDeps array of your renovate config.

If this PR was closed by mistake or you changed your mind, you can simply rename this PR and you will soon get a fresh replacement PR opened.

@renovate-bot renovate-bot deleted the renovate/apache-beam-packages branch December 21, 2021 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anguillanneuf anguillanneuf Awaiting requested review from anguillanneuf

Assignees

@chingor13 chingor13

Labels
cla: yes This human has signed the Contributor License Agreement. owlbot:run Add this label to trigger the Owlbot post processor. samples Issues that are directly related to samples.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@renovate-bot @lesv @anguillanneuf @davidcavazos @jsimonweb @skuruppu @chingor13 @stephaniewang526 @kokoro-team @eaball35