-
Notifications
You must be signed in to change notification settings - Fork 365
dbt and Git Extensions Public Beta [DOC-862] #6573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
4ef3138
Add structure
pwseg 2080c4c
dbt sync first pass
pwseg f48f686
Add Reverse ETL steps
pwseg c4db284
More changes
pwseg cc73892
Update main page
pwseg b6b3f78
More cleanup [netlify-build]
pwseg 8d6c1a8
Extensions, not integrations
pwseg 67efe18
Add resource info
pwseg 50b27d9
workspace not Workspace
pwseg 546a3af
Add to side navigation [netlify-build]
pwseg c70a001
small fixes
pwseg 5222ad5
Add Reverse ETL links
pwseg File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
--- | ||
title: dbt Extension | ||
--- | ||
|
||
Segment's dbt extension lets you use [Reverse ETL](/docs/connections/reverse-etl/) with your existing dbt labs models and syncs to help centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. | ||
|
||
With Segment's dbt extension, you can: | ||
|
||
- Securely connect Segment to the GitHub repository that stores your dbt models. | ||
- Use centralized dbt models to set up Reverse ETL. | ||
- Trigger Reverse ETL syncs from dbt jobs. | ||
|
||
This page explains how to set up a dbt Model and then use the model with Reverse ETL. | ||
|
||
> info "" | ||
> Extensions, including dbt sync, is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. | ||
|
||
## Before you begin | ||
|
||
Keep the following in mind as you set up the dbt extension: | ||
|
||
- The extension supports [dbt Core v1.7](https://docs.getdbt.com/docs/dbt-versions/core-upgrade/upgrading-to-v1.7){:target="_blank"}. | ||
- You can use [Snowflake](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/snowflake-setup/), [Databricks](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/databricks-setup/), [Redshift](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/redshift-setup/), [Postgres](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/postgres-setup/), and [BigQuery](/docs/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup/) as Reverse ETL sources. | ||
- dbt models aren't synchronized from the dbt cloud. The model sync connects to a Git repository that loads models into Segment for use with Reverse ETL. | ||
|
||
## Set up Git dbt Models and dbt Cloud | ||
|
||
To set up the dbt extension, you'll need: | ||
|
||
- an existing dbt account with a Git repository | ||
- (for job syncs:) dbt cloud with jobs already created | ||
|
||
### Git repository and dbt Models setup | ||
pwseg marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Follow these steps to connect the Git repository that stores your dbt Models: | ||
|
||
1. In your Segment workspace, navigate to **Settings > Extensions**. | ||
2. Click **Set up Git sync**. | ||
3. On the **Configure service credentials** page, select a service and protocol, add your SSH private key or GitHub token, then click **Next**. | ||
4. In the **Connect source** window, select a Reverse ETL warehouse source from the dropdown, then click **Save**. | ||
|
||
After you've saved your setup, you can configure your Git repository's settings to your needs by changing the repository, branch, dbt version, default schema, and project path. | ||
|
||
### dbt Cloud setup | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. passive voice |
||
|
||
You can also use dbt Cloud to schedule Reverse ETL syncs after a dbt Cloud job successfully runs. | ||
|
||
To set up dbt Cloud: | ||
|
||
1. In your Segment workspace, navigate to **Settings > Extensions**. | ||
2. Click **Manage dbt Cloud**. | ||
3. Add your dbt Cloud API key, and, optionally, a custom subdomain. Click **Save**. | ||
|
||
### Model syncs | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. passive voice |
||
|
||
After you set up dbt, Segment runs an initial sync to load models from your connected Git repository. This initial sync lets you use the most recent models when you set up Reverse ETL. In addition to Segment's initial dbt sync, you can also trigger manual dbt model syncs. | ||
|
||
### Use a model with Reverse ETL | ||
|
||
After you've successfully set up dbt with a warehouse and connected to your Git repository, you can select dbt models for use with Reverse ETL by following these steps: | ||
|
||
1. In your Segment workspace, navigate to **Connections > Sources** and select the Reverse ETL tab. | ||
2. Click **+Add Reverse ETL source** , select your source, then click **Add Model**. | ||
3. Click **dbt Models** as your modeling method, then select and preview a model from the dbt model dropdown. | ||
4. Add a primary key, then click **Preview your model**. | ||
5. Click **Next**. | ||
6. Enter your **Model Name**, then click **Create Model**. | ||
|
||
To change a connected model, ensure that you've removed it from all active Reverse ETL syncs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
title: Git Sync Extension | ||
--- | ||
|
||
Segment's Git extension lets you manage versioning by syncing changes you make in Sources and Destinations from your Segment workspace to a Git repository. | ||
|
||
> info "" | ||
> Extensions, including Git sync, is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. | ||
|
||
## Set up Git sync | ||
|
||
Follow these steps to set up Git sync: | ||
|
||
1. In your Segment workspace, navigate to **Settings > Extensions**. | ||
2. Click **Set up Git sync**. | ||
3. On the **Configure service credentials** page, select a service and protocol, add your SSH private key or GitHub token, then click **Next**. | ||
|
||
## Working with Git syncs | ||
|
||
The Git sync extension syncs [Connections](/docs/connections/) ([Sources](/docs/connections/sources/) and [Destinations](/docs/connections/destinations/)) from Segment to your Git repository. | ||
|
||
After you set up the Git sync extension for the first time, Segment performs an initial sync that sends source and destination information in your Segment workspace to the Git repository you connected. | ||
|
||
You can run syncs at any time by clicking **Full resync** on the Git sync page. To disable Git sync from the Git sync page, switch the **Enabled** toggle to off. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
--- | ||
title: Extensions | ||
--- | ||
|
||
Extensions integrate third-party tools into your existing Segment workspace to help you automate tasks. | ||
|
||
> info "" | ||
> Extensions is currently in public beta and is governed by Segment's [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. During Public Beta, Extensions is available for Team and Developer plans only. [Reach out to Segment](mailto:friends@segment.com) if you're on a Business Tier plan and would like to participate in the Public Beta. | ||
|
||
During public beta, Segment offers the following extensions: | ||
|
||
- [dbt models and dbt Cloud](/docs/segment-app/extensions/dbt), which lets you sync models with dbt Labs. | ||
- [Git sync](/docs/segment-app/extensions/git), which you can use to manage versioning and track changes in your Segment workspace. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.