Data orchestration blog sample #5686

rachael-ds · 2021-04-12T10:46:12Z

Adding new example Composer DAG sample to accompany my blog on data orchestration.
In draft state for review by @leahecole thanks!

google-cla · 2021-04-12T10:46:17Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T10:53:06Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T13:14:36Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T13:51:27Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T13:55:17Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T14:54:56Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T14:56:40Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T14:57:20Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T15:15:14Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-12T15:16:47Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

rachael-ds · 2021-04-12T15:22:23Z

Fixed GPG signing of old commits

snippet-bot · 2021-05-20T15:24:00Z

No region tags are edited in this PR.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

Refresh this comment

Signed-off-by: rachael-ds <deaconsmith@google.com>

leahecole · 2021-05-21T05:20:26Z

composer/blog/gcp-tech-blog/data-orchestration-with-composer/requirements-test.txt

@@ -0,0 +1,2 @@
+pytest==6.2.1
+git+https://github.com/GoogleCloudPlatform/python-docs-samples.git#egg=dag_test_utils&subdirectory=composer/dag_test_utils # DAG unit test utility


Suggested change

git+https://github.com/GoogleCloudPlatform/python-docs-samples.git#egg=dag_test_utils&subdirectory=composer/dag_test_utils # DAG unit test utility

cloud-composer-dag-test-utils==0.0.1 # DAG unit test utility

I released this as a real PyPI package now so no more ugly import needed 😄

composer/blog/gcp-tech-blog/data-orchestration-with-composer/requirements.txt

composer/blog/gcp-tech-blog/data-orchestration-with-composer/constraints.txt

...p-tech-blog/data-orchestration-with-composer/dags/data_orchestration_blog_sample_dag_test.py

...og/gcp-tech-blog/data-orchestration-with-composer/dags/data_orchestration_blog_sample_dag.py

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

leahecole

The nits I've left are non-blocking - mostly just converting to f-strings. If you choose to accept the changes, just double check the dag runs in your environment before sending to publication

...og/gcp-tech-blog/data-orchestration-with-composer/dags/data_orchestration_blog_sample_dag.py

leahecole · 2021-06-14T22:29:47Z

...og/gcp-tech-blog/data-orchestration-with-composer/dags/data_orchestration_blog_sample_dag.py

+    slack_msg = """
+            :red_circle: Task Failed.
+            *Task*: {task}
+            *Dag*: {dag}
+            *Execution Time*: {exec_date}
+            *Log Url*: {log_url}
+            """.format(
+            task=ti.task_id,
+            dag=ti.dag_id,
+            log_url=ti.log_url,
+            exec_date=context.get('execution_date'),
+        )


Suggested change

slack_msg = """

:red_circle: Task Failed.

*Task*: {task}

*Dag*: {dag}

*Execution Time*: {exec_date}

*Log Url*: {log_url}

""".format(

task=ti.task_id,

dag=ti.dag_id,

log_url=ti.log_url,

exec_date=context.get('execution_date'),

)

slack_msg = f"""

:red_circle: Task Failed.

*Task*: {ti.task_id}

*Dag*: {ti.dag_id}

*Execution Time*: {context.get('execution_date')}

*Log Url*: {ti.log_url}

"""

done - thank you!

dinagraves · 2021-06-15T17:07:19Z

Looks good! Nit: some of the formatting is inconsistent, eg the mixing of single and double quotes in one dict. Could you run black on these files to clean it up?

product-auto-label bot added the samples Issues that are directly related to samples. label Apr 12, 2021

google-cla bot added the cla: no This human has *not* signed the Contributor License Agreement. label Apr 12, 2021

rachael-ds force-pushed the data-orchestration-blog-sample branch from aee35da to 82c7df7 Compare April 12, 2021 13:51

rachael-ds force-pushed the data-orchestration-blog-sample branch from 82c7df7 to 4e514ad Compare April 12, 2021 13:55

rachael-ds closed this Apr 12, 2021

rachael-ds reopened this Apr 12, 2021

rachael-ds force-pushed the data-orchestration-blog-sample branch from 4e514ad to 82c7df7 Compare April 12, 2021 14:54

rachael-ds force-pushed the data-orchestration-blog-sample branch from 82c7df7 to 29da218 Compare April 12, 2021 14:56

adding new gcp blog directory and sample dag for my blog

f33f295

rachael-ds force-pushed the data-orchestration-blog-sample branch from e0914db to 0f2b96b Compare April 12, 2021 15:15

added required files and moved to gcp tech blog folder

5cb325b

rachael-ds force-pushed the data-orchestration-blog-sample branch from 0f2b96b to a866173 Compare April 12, 2021 15:16

adding licence to example dag

732a6e9

rachael-ds force-pushed the data-orchestration-blog-sample branch from a866173 to 732a6e9 Compare April 12, 2021 15:18

google-cla bot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Apr 12, 2021

leahecole self-assigned this Apr 12, 2021

leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 12, 2021

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 12, 2021

Merge branch 'master' into data-orchestration-blog-sample

718bd01

rachael-ds closed this Apr 26, 2021

rachael-ds reopened this Apr 26, 2021

rachael-ds and others added 2 commits April 26, 2021 18:03

trigger build

ff62adb

Merge branch 'master' into data-orchestration-blog-sample

a6e2170

leahecole added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 27, 2021

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 27, 2021

tmatsuo added the snippet-bot:force-run Force snippet-bot runs its logic label May 20, 2021

snippet-bot bot removed the snippet-bot:force-run Force snippet-bot runs its logic label May 20, 2021

tmatsuo added the snippet-bot:force-run Force snippet-bot runs its logic label Jun 1, 2021

snippet-bot bot removed the snippet-bot:force-run Force snippet-bot runs its logic label Jun 1, 2021

rachael-ds and others added 3 commits June 7, 2021 11:03

Merge branch 'master' into data-orchestration-blog-sample

0011c56

updated to use Airflow 2.0

ba8e02b

Signed-off-by: rachael-ds <deaconsmith@google.com>

Fixed conftest.py file header and lint issues

edd4aac

Signed-off-by: rachael-ds <deaconsmith@google.com>

rachael-ds marked this pull request as ready for review June 8, 2021 17:38

rachael-ds requested a review from a team as a code owner June 8, 2021 17:38

leahecole requested changes Jun 8, 2021

View reviewed changes

Apply suggestions from code review

72a80fb

Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com>

leahecole approved these changes Jun 14, 2021

View reviewed changes

leahecole added the blunderbuss: assign Instruct blunderbuss to assign someone label Jun 15, 2021

blunderbuss-gcf bot removed the blunderbuss: assign Instruct blunderbuss to assign someone label Jun 15, 2021

blunderbuss-gcf bot assigned dinagraves Jun 15, 2021

Merge branch 'master' into data-orchestration-blog-sample

65a4059

dinagraves approved these changes Jun 15, 2021

View reviewed changes

rachael-ds added 3 commits June 16, 2021 15:16

formatting changes after running nox black

e6e3a6f

updated to use f-strings

0422943

updated slack connection formatting

35c3734

leahecole merged commit e415b2d into GoogleCloudPlatform:master Jun 16, 2021

		@@ -0,0 +1,2 @@
		pytest==6.2.1
		git+https://github.com/GoogleCloudPlatform/python-docs-samples.git#egg=dag_test_utils&subdirectory=composer/dag_test_utils # DAG unit test utility

	git+https://github.com/GoogleCloudPlatform/python-docs-samples.git#egg=dag_test_utils&subdirectory=composer/dag_test_utils # DAG unit test utility
	cloud-composer-dag-test-utils==0.0.1 # DAG unit test utility

Data orchestration blog sample #5686

Data orchestration blog sample #5686

Uh oh!

Conversation

rachael-ds commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

google-cla bot commented Apr 12, 2021

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

google-cla bot commented Apr 12, 2021

Uh oh!

rachael-ds commented Apr 12, 2021

Uh oh!

snippet-bot bot commented May 20, 2021

No region tags are edited in this PR.

Uh oh!

leahecole May 21, 2021

Choose a reason for hiding this comment

Uh oh!

leahecole May 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leahecole left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leahecole Jun 14, 2021

Choose a reason for hiding this comment

Uh oh!

dinagraves Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

rachael-ds Jun 16, 2021

Choose a reason for hiding this comment

Uh oh!

dinagraves commented Jun 15, 2021

Uh oh!

Uh oh!