Add tutorial for file preprocessing index pipeline (deepset-ai#274)

Tilde Thurium · TuanaCelik · bilgeyucel · web-flow · commit ce7178c56d6c · 2024-02-01T13:31:20.000+01:00
* add file type preprocessing tutorial

* update index.toml

* update workflows and README

* Update tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb

Co-authored-by: Tuana Çelik &lt;tuana.celik@deepset.ai&gt;

* Update index.toml

Co-authored-by: Bilge Yücel &lt;bilgeyucel96@gmail.com&gt;

* Update index.toml

Co-authored-by: Bilge Yücel &lt;bilgeyucel96@gmail.com&gt;

* Update README.md

Co-authored-by: Bilge Yücel &lt;bilgeyucel96@gmail.com&gt;

* address feedback

* remove commented out code and add telemetry text

* adding a mini explanation on DocumentJoiner

* moving pipeline drawing up

---------

Co-authored-by: Tuana Çelik &lt;tuana.celik@deepset.ai&gt;
Co-authored-by: Bilge Yücel &lt;bilgeyucel96@gmail.com&gt;
diff --git a/.github/workflows/run_tutorials.yml b/.github/workflows/run_tutorials.yml
@@ -70,6 +70,7 @@ jobs:
               "tutorials/27_First_RAG_Pipeline.ipynb"
               "tutorials/28_Structured_Output_With_Loop.ipynb"
               "tutorials/29_Serializing_Pipelines.ipynb"
+              "tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb"
               "tutorials/31_Metadata_Filtering.ipynb"
           )
           for changed_file in ${{ steps.files.outputs.all_changed_files }}; do
diff --git a/README.md b/README.md
@@ -47,5 +47,6 @@ To contribute to the tutorials, please check out our [Contributing Guidelines](.
 | [Your First QA Pipeline with Retrieval-Augmentation](./tutorials/27_First_RAG_Pipeline.ipynb)                | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/main/tutorials/27_First_RAG_Pipeline.ipynb) |
 | [Generating Structured Output with Loop-Based Auto-Correction](./tutorials/28_Structured_Output_With_Loop.ipynb)                | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/main/tutorials/28_Structured_Output_With_Loop.ipynb) |
 | [Serializing Pipelines](./tutorials/29_Serializing_Pipelines.ipynb)                | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/main/tutorials/29_Serializing_Pipelines.ipynb) |
+| [Preprocessing Different File Types](./tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb)                | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/main/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb) |
 | [Metadata Filtering](./tutorials/31_Metadata_Filtering.ipynb)                | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/main/tutorials/31_Metadata_Filtering.ipynb) |
 
diff --git a/index.toml b/index.toml
@@ -298,13 +298,24 @@ completion_time = "10 min"
 created_at = 2024-01-29
 haystack_2 = true
 
+[[tutorial]]
+title = "Preprocessing Different File Types"
+description = "Learn how to build an indexing pipeline that will preprocess files based on their file type"
+level = "beginner"
+weight = 72
+notebook = "30_File_Type_Preprocessing_Index_Pipeline.ipynb"
+aliases = []
+completion_time = "10 min"
+created_at = 2024-01-30
+haystack_2 = true
+
 [[tutorial]]
 title = "Filtering Documents with Metadata"
 description = "Learn how to filter down to specific documents at retrieval time using metadata"
 level = "beginner"
 weight = 6
 notebook = "31_Metadata_Filtering.ipynb"
 aliases = []
-completion_time = "10 min"
+completion_time = "15 min"
 created_at = 2024-01-30
-haystack_2 = true
+haystack_2 = true
diff --git a/tutorials/29_Serializing_Pipelines.ipynb b/tutorials/29_Serializing_Pipelines.ipynb
@@ -396,6 +396,20 @@
       "source": [
         "new_pipeline.run(data={\"builder\": {\"sentence\": \"I love capybaras\"}})"
       ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## What's next\n",
+        "\n",
+        "🎉 Congratulations! You've serialzed a pipeline into YAML, edited it and ran it again!\n",
+        "\n",
+        "If you liked this tutorial, you may also enjoy:\n",
+        "-  [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline)\n",
+        "\n",
+        "To stay up to date on the latest Haystack developments, you can [sign up for our newsletter](https://landing.deepset.ai/haystack-community-updates?utm_campaign=developer-relations&utm_source=tutorial&utm_medium=serialization). Thanks for reading!"
+      ]
     }
   ],
   "metadata": {
diff --git a/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb b/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb

Original file line number	Diff line number	Diff line change
`@@ -70,6 +70,7 @@ jobs:`
`70`	`70`	`"tutorials/27_First_RAG_Pipeline.ipynb"`
`71`	`71`	`"tutorials/28_Structured_Output_With_Loop.ipynb"`
`72`	`72`	`"tutorials/29_Serializing_Pipelines.ipynb"`
	`73`	`+ "tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb"`
`73`	`74`	`"tutorials/31_Metadata_Filtering.ipynb"`
`74`	`75`	`)`
`75`	`76`	`for changed_file in ${{ steps.files.outputs.all_changed_files }}; do`