MINOR: Add tutorial of Airflow AWS S3 Sensor

coder2j · coder2j · commit 6594b1b39f0c · 2022-02-19T15:10:30.000+01:00
diff --git a/README.md b/README.md
@@ -1,22 +1,36 @@
-# airflow-docker
+# Apache Airflow  Tutorial Series [YouTube](https://www.youtube.com/watch?v=z7xyNOF8tak&list=PLwFJcsJ61oujAqYpMp1kdUBcPG0sE0QMT)
+## Updated Tutorial Episode
+1. [Introduction and Local Installation](https://youtu.be/z7xyNOF8tak)
+2. [Get Airflow running in Docker](https://youtu.be/J6azvFhndLg)
+3. [Airflow Core Concepts in 5 mins](https://youtu.be/mtJHMdoi_Gg)
+4. [Airflow Task Lifecycle and Basic Architecture](https://youtu.be/UFsCvWjQT4w)
+5. [Airflow DAG with BashOperator](https://youtu.be/CLkzXrjrFKg)
+6. [Airflow DAG with PythonOperator and XComs](https://youtu.be/IumQX-mm20Y)
+7. [Airflow TaskFlow API](https://youtu.be/9y0mqWsok_4)
+8. [Airflow Catchup and Backfill](https://youtu.be/OXOiUeHOQ-0)
+9. [Schedule Airflow DAG with Cron Expression](https://youtu.be/tpuovQFUByk)
+10. [Airflow Connection and PostgresOperator](https://youtu.be/S1eapG6gjLU)
+11. [Add Python Dependencies via Airflow Docker Image Extending and Customizing](https://youtu.be/0UepvC9X4HY)
+12. [AWS S3 Key Sensor Operator](https://youtu.be/vuxrhipJMCk)
+
 ## Running apache airflow 2.0 in docker with local executor.
 Here are the steps to take to get airflow 2.0 running with docker on your machine. 
 1. Clone this repo
-2. Create dags, logs and plugins folder inside the project directory
+1. Create dags, logs and plugins folder inside the project directory
 ```bash
 mkdir ./dags ./logs ./plugins
 ```
-3. Install docker desktop application if you don't have docker running on your machine
+1. Install docker desktop application if you don't have docker running on your machine
 - [Download Docker Desktop Application for Mac OS](https://hub.docker.com/editions/community/docker-ce-desktop-mac)
 - [Download Docker Desktop Application for Windows](https://hub.docker.com/editions/community/docker-ce-desktop-windows)
-4. Launch airflow by docker-compose
+1. Launch airflow by docker-compose
 ```bash
 docker-compose up -d
 ```
-5. Check the running containers
+1. Check the running containers
 ```bash
 docker ps
 ```
-6. Open browser and type http://0.0.0.0:800 to launch the airflow webserver
+1. Open browser and type http://0.0.0.0:800 to launch the airflow webserver
 
 ![](images/screenshot_airflow_docker.png)
diff --git a/dags/dag_with_minio_s3.py b/dags/dag_with_minio_s3.py
@@ -0,0 +1,28 @@
+from datetime import datetime, timedelta
+
+from airflow import DAG
+from airflow.providers.amazon.aws.sensors.s3_key import S3KeySensor
+
+
+default_args = {
+    'owner': 'coder2j',
+    'retries': 5,
+    'retry_delay': timedelta(minutes=10)
+}
+
+
+with DAG(
+    dag_id='dag_with_minio_s3_v02',
+    start_date=datetime(2022, 2, 12),
+    schedule_interval='@daily',
+    default_args=default_args
+) as dag:
+    task1 = S3KeySensor(
+        task_id='sensor_minio_s3',
+        bucket_name='airflow',
+        bucket_key='data.csv',
+        aws_conn_id='minio_conn',
+        mode='poke',
+        poke_interval=5,
+        timeout=30
+    )
diff --git a/data/data.csv b/data/data.csv
@@ -0,0 +1,3 @@
+product_id, delivery_dt
+10005, 2022-01-01
+10001, 2021-12-30

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+product_id, delivery_dt`
	`2`	`+10005, 2022-01-01`
	`3`	`+10001, 2021-12-30`