Skip to content

Update the auto YAML Generation #7725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open

Update the auto YAML Generation #7725

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
1df036d
some changes for connector YAML
royendo Aug 4, 2025
3ff888e
remove
royendo Aug 4, 2025
a381593
separated YAML schema
royendo Aug 4, 2025
b6b987f
nit
royendo Aug 4, 2025
d7d4b11
link fixes
royendo Aug 4, 2025
a3784a0
Update generate_project.go
royendo Aug 4, 2025
f2ed2c4
add sec warning
royendo Aug 4, 2025
d57d430
motherduck live connector
royendo Aug 4, 2025
e5434ee
remove local file from connector
royendo Aug 4, 2025
cd0c819
nit
royendo Aug 4, 2025
d3396eb
matching file names
royendo Aug 4, 2025
16c99c6
reordered files and added sources YAML and model SQL
royendo Aug 4, 2025
6ba0dcd
nit, fix links
royendo Aug 4, 2025
4363fff
comparing with current YAML changes
royendo Aug 5, 2025
64677e0
Merge branch 'main' into docs/auto-gen-YAML
royendo Aug 5, 2025
77f0ca9
adding annotatons back
royendo Aug 5, 2025
7c35559
gofmt, golint
royendo Aug 5, 2025
0ff11e2
fix
royendo Aug 5, 2025
11b2710
first pass fix, need to review a few other items
royendo Aug 6, 2025
db75a6b
remove toplevel driver
royendo Aug 6, 2025
95ca49a
single project file
royendo Aug 6, 2025
98baf56
Update generate_project.go
royendo Aug 6, 2025
7065bcc
good old cursor wiped my functoni
royendo Aug 6, 2025
dba395a
missing parameters
royendo Aug 6, 2025
6e32ddd
midding parameters and inline examples
royendo Aug 6, 2025
c55eff3
god so many false positives from asking cursor to merge files
royendo Aug 6, 2025
6ffe7f8
adding links
royendo Aug 6, 2025
f69c89a
fixed oneOf
royendo Aug 6, 2025
3b6517e
split data properties to have api excamples
royendo Aug 6, 2025
e67f159
gonig through files
royendo Aug 7, 2025
df54d71
returning original format
royendo Aug 7, 2025
f94396e
?
royendo Aug 7, 2025
c58c634
Merge remote-tracking branch 'origin/main' into docs/auto-gen-YAML
royendo Aug 8, 2025
e8c04e9
fixing urls,
royendo Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
split data properties to have api excamples
  • Loading branch information
@royendo
royendo committed Aug 6, 2025
commit 3b6517e6e5b05de5ad3f0dfb5e568ac7e5905166
56 changes: 0 additions & 56 deletions docs/docs/hidden/yaml/advanced-models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,50 +94,22 @@ state:


- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we remove the options? I think it was clearer before, as it showed that the user could choose any option like sql, metric_sql, api, etc.

```yaml
type: api

connector: "duckdb"
sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.
```yaml
type: api

metrics_sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.
```yaml
type: api

api: "user_analytics_api"
args:
start_date: "2024-01-01"
limit: 10
```


- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.
```yaml
type: api

glob: "data/*.csv"
```


- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

Expand Down Expand Up @@ -174,50 +146,22 @@ partitions:


- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.
```yaml
type: api

connector: "duckdb"
sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.
```yaml
type: api

metrics_sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.
```yaml
type: api

api: "user_analytics_api"
args:
start_date: "2024-01-01"
limit: 10
```


- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.
```yaml
type: api

glob: "data/*.csv"
```


- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

Expand Down
28 changes: 0 additions & 28 deletions docs/docs/hidden/yaml/alerts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,50 +45,22 @@ _[string]_ - Description for the alert
_[oneOf]_ - Data source for the alert _(required)_

- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here. we should not remove options.

```yaml
type: api

connector: "duckdb"
sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.
```yaml
type: api

metrics_sql: "SELECT * FROM user_metrics WHERE date >= '2024-01-01'"
```


- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.
```yaml
type: api

api: "user_analytics_api"
args:
start_date: "2024-01-01"
limit: 10
```


- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.
```yaml
type: api

glob: "data/*.csv"
```


- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

Expand Down
83 changes: 81 additions & 2 deletions runtime/parser/schema/project.schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1119,7 +1119,7 @@ oneOf:
sample: false
required:
- type
- $ref: '#/definitions/data_properties'
- $ref: '#/definitions/api_data_properties'
# - $ref: '#/definitions/common_properties'

# Themes
Expand Down Expand Up @@ -1389,7 +1389,7 @@ definitions:
# - name
# - type
# additionalProperties: false
data_properties:
api_data_properties:
oneOf:
- title: SQL Query
type: object
Expand Down Expand Up @@ -1476,6 +1476,85 @@ definitions:
required:
- glob

- title: Resource Status Check
type: object
description: Uses the status of a resource as data.
properties:
resource_status:
type: object
description: Based on resource status
properties:
where_error:
type: boolean
description: Indicates whether the condition should trigger when the resource is in an error state.
sample: true
additionalProperties: true
required:
- resource_status
examples: |
```yaml
resource_status:
where_error: true
```
data_properties:
oneOf:
- title: SQL Query
type: object
description: Executes a raw SQL query against the project's data models.
properties:
sql:
type: string
description: Raw SQL query to run against existing models in the project.
connector:
type: string
description: specifies the connector to use when running SQL or glob queries.
required:
- sql


- title: Metrics View Query
type: object
description: Executes a SQL query that targets a defined metrics view.
properties:
metrics_sql:
type: string
description: SQL query that targets a metrics view in the project
required:
- metrics_sql

- title: Custom API Call
type: object
description: Calls a custom API defined in the project to compute data.
properties:
api:
type: string
description: Name of a custom API defined in the project.
args:
type: object
description: Arguments to pass to the custom API.
additionalProperties: true
required:
- api

- title: File Glob Query
type: object
description: Uses a file-matching pattern (glob) to query data from a connector.
properties:
glob:
description: Defines the file path or pattern to query from the specified connector.
anyOf:
- type: string
description: A simple file path/glob pattern as a string.
sample: "data/*.csv"
- type: object
description: An object-based configuration for specifying a file path/glob pattern with advanced options.
additionalProperties: true
connector:
type: string
description: Specifies the connector to use with the glob input.
required:
- glob

- title: Resource Status Check
type: object
description: Uses the status of a resource as data.
Expand Down