Skip to content

Commit e415b2d

Browse files
rachael-dsleahecoledinagraves
authored
Data orchestration blog sample (GoogleCloudPlatform#5686)
* adding new gcp blog directory and sample dag for my blog * added required files and moved to gcp tech blog folder * adding licence to example dag * Apply suggestions from code review Commits from Leah Cole - replace my specific environment details with generic environment strings Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * Update composer/blog/gcp-tech-blog/data-orchestration-with-composer/requirements.txt Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * updated dataflow template and variables * updated source destinations and added init.py * added beam package requirement and lint changes * updated slack package and further lint changes * lint fix * trigger build * updated to use Airflow 2.0 Signed-off-by: rachael-ds <deaconsmith@google.com> * Fixed conftest.py file header and lint issues Signed-off-by: rachael-ds <deaconsmith@google.com> * Apply suggestions from code review Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> * formatting changes after running nox black * updated to use f-strings * updated slack connection formatting Co-authored-by: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Co-authored-by: Dina Graves Portman <dinagraves@google.com>
1 parent 1f9501c commit e415b2d

File tree

10 files changed

+729
-0
lines changed

10 files changed

+729
-0
lines changed
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Data Orchestration with Cloud Composer
2+
3+
Code found in this directory is part of a [blog post](https://cloud.google.com/blog/topics/developers-practitioners) published in June 2021

composer/blog/gcp-tech-blog/data-orchestration-with-composer/__init__.py

Whitespace-only changes.
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Copyright 2021 Google LLC
2+
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
7+
# https://www.apache.org/licenses/LICENSE-2.0
8+
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
16+
import os
17+
18+
import pytest
19+
20+
21+
# this fixture initializes the Airflow DB once per session
22+
# it is used by DAGs in both the blogs and workflows directories,
23+
# unless there exists a conftest at a lower level
24+
@pytest.fixture(scope="session")
25+
def airflow_database():
26+
import airflow.utils.db
27+
28+
# We use separate directory for local db path per session
29+
# by setting AIRFLOW_HOME env var, which is done in noxfile_config.py.
30+
31+
assert "AIRFLOW_HOME" in os.environ
32+
33+
airflow_home = os.environ["AIRFLOW_HOME"]
34+
airflow_db = f"{airflow_home}/airflow.db"
35+
36+
# reset both resets and initializes a new database
37+
airflow.utils.db.resetdb()
38+
39+
# Making sure we are using a data file there.
40+
assert os.path.isfile(airflow_db)

0 commit comments

Comments
 (0)