Skip to content

Update the auto YAML Generation #7725

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
1df036d
some changes for connector YAML
royendo Aug 4, 2025
3ff888e
remove
royendo Aug 4, 2025
a381593
separated YAML schema
royendo Aug 4, 2025
b6b987f
nit
royendo Aug 4, 2025
d7d4b11
link fixes
royendo Aug 4, 2025
a3784a0
Update generate_project.go
royendo Aug 4, 2025
f2ed2c4
add sec warning
royendo Aug 4, 2025
d57d430
motherduck live connector
royendo Aug 4, 2025
e5434ee
remove local file from connector
royendo Aug 4, 2025
cd0c819
nit
royendo Aug 4, 2025
d3396eb
matching file names
royendo Aug 4, 2025
16c99c6
reordered files and added sources YAML and model SQL
royendo Aug 4, 2025
6ba0dcd
nit, fix links
royendo Aug 4, 2025
4363fff
comparing with current YAML changes
royendo Aug 5, 2025
64677e0
Merge branch 'main' into docs/auto-gen-YAML
royendo Aug 5, 2025
77f0ca9
adding annotatons back
royendo Aug 5, 2025
7c35559
gofmt, golint
royendo Aug 5, 2025
0ff11e2
fix
royendo Aug 5, 2025
11b2710
first pass fix, need to review a few other items
royendo Aug 6, 2025
db75a6b
remove toplevel driver
royendo Aug 6, 2025
95ca49a
single project file
royendo Aug 6, 2025
98baf56
Update generate_project.go
royendo Aug 6, 2025
7065bcc
good old cursor wiped my functoni
royendo Aug 6, 2025
dba395a
missing parameters
royendo Aug 6, 2025
6e32ddd
midding parameters and inline examples
royendo Aug 6, 2025
c55eff3
god so many false positives from asking cursor to merge files
royendo Aug 6, 2025
6ffe7f8
adding links
royendo Aug 6, 2025
f69c89a
fixed oneOf
royendo Aug 6, 2025
3b6517e
split data properties to have api excamples
royendo Aug 6, 2025
e67f159
gonig through files
royendo Aug 7, 2025
df54d71
returning original format
royendo Aug 7, 2025
f94396e
?
royendo Aug 7, 2025
c58c634
Merge remote-tracking branch 'origin/main' into docs/auto-gen-YAML
royendo Aug 8, 2025
e8c04e9
fixing urls,
royendo Aug 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
gonig through files
  • Loading branch information
royendo committed Aug 7, 2025
commit e67f1590fd78d7bcd6b66d825c1d3545ca443c4f
39 changes: 34 additions & 5 deletions cli/cmd/docs/generate_project.go
Original file line number Diff line number Diff line change
Expand Up @@ -479,10 +479,9 @@ func generateDoc(sidebarPosition, level int, node *yaml.Node, indent string, req


}

} else {
// only for APIs, no other oneOf uses this, so we can be specific
} else if id == "apis" {
// Skip oneOf processing for connectors and data_properties at level 1 since we handle it above
if !(id == "connectors") {
if len(oneOf.Content) == 1 {
doc.WriteString(generateDoc(sidebarPosition, level, oneOf.Content[0], indent, getRequiredMapFromNode(oneOf.Content[0]), rootSchema, id))
} else {
Expand All @@ -491,10 +490,10 @@ func generateDoc(sidebarPosition, level int, node *yaml.Node, indent string, req
doc.WriteString("\n\n## One of Properties Options")
}

for i, item := range oneOf.Content {
for _, item := range oneOf.Content {
if hasType(item) || hasProperties(item) || hasCombinators(item) {
doc.WriteString(fmt.Sprintf("\n\n%s- **option %d** - %s - %s", indent, i+1, getPrintableType(item), getPrintableDescription(item, indent, "(no description)")))
doc.WriteString(generateDoc(sidebarPosition, level, item, indent+" ", getRequiredMapFromNode(item), rootSchema, id))

}

// Handle examples for oneOf items
Expand All @@ -515,6 +514,36 @@ func generateDoc(sidebarPosition, level int, node *yaml.Node, indent string, req
}
}
}
} else {
if !(id == "connectors" || id == "apis") {
// Check if this is a field_selector_properties definition (used in explore dashboards)
title := getScalarValue(node, "title")
if title == "Wildcard(*) selector" || title == "Explicit list of fields" || title == "Regex matching" {
// This is field_selector_properties, use numbered options format
for i, item := range oneOf.Content {
if hasType(item) || hasProperties(item) || hasCombinators(item) {
itemTitle := getScalarValue(item, "title")
if itemTitle != "" {
doc.WriteString(fmt.Sprintf("\n\n%s- **option %d** - _[%s]_ - %s", indent, i+1, getPrintableType(item), itemTitle))
} else {
doc.WriteString(fmt.Sprintf("\n\n%s- **option %d** - %s - %s", indent, i+1, getPrintableType(item), getPrintableDescription(item, indent, "(no description)")))
}
doc.WriteString(generateDoc(sidebarPosition, level, item, indent+" ", getRequiredMapFromNode(item), rootSchema, id))
}
}
} else {
// Regular oneOf processing
if len(oneOf.Content) == 1 {
doc.WriteString(generateDoc(sidebarPosition, level, oneOf.Content[0], indent, getRequiredMapFromNode(oneOf.Content[0]), rootSchema, id))
} else {
for _, item := range oneOf.Content {
if hasType(item) || hasProperties(item) || hasCombinators(item) {
doc.WriteString(generateDoc(sidebarPosition, level, item, indent+" ", getRequiredMapFromNode(item), rootSchema, id))
}
}
}
}
}
}
}
}
Expand Down
70 changes: 37 additions & 33 deletions docs/docs/hidden/yaml/advanced-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,24 +93,16 @@ state:
```


- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we remove the options? I think it was clearer before, as it showed that the user could choose any option like sql, metric_sql, api, etc.


- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.

- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.

- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.

- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

- **option 1** - _[string]_ - A simple file path/glob pattern as a string.
Expand All @@ -119,18 +111,10 @@ state:

- **`connector`** - _[string]_ - Specifies the connector to use with the glob input.

- **option 5** - _[object]_ - Uses the status of a resource as data.

- **`resource_status`** - _[object]_ - Based on resource status _(required)_

- **`where_error`** - _[boolean]_ - Indicates whether the condition should trigger when the resource is in an error state.

```yaml
resource_status:
where_error: true
```


### `partitions`

_[oneOf]_ - Refers to the how your data is partitioned, cannot be used with state. (optional)
Expand All @@ -145,24 +129,16 @@ partitions:
```


- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.

- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.

- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.

- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.

- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

- **option 1** - _[string]_ - A simple file path/glob pattern as a string.
Expand All @@ -171,18 +147,10 @@ partitions:

- **`connector`** - _[string]_ - Specifies the connector to use with the glob input.

- **option 5** - _[object]_ - Uses the status of a resource as data.

- **`resource_status`** - _[object]_ - Based on resource status _(required)_

- **`where_error`** - _[boolean]_ - Indicates whether the condition should trigger when the resource is in an error state.

```yaml
resource_status:
where_error: true
```


### `materialize`

_[boolean]_ - models will be materialized in olap
Expand Down Expand Up @@ -225,6 +193,38 @@ _[object]_ - to define the properties of output

- **`partition_by`** - _[string]_ - Column or expression to partition the table by

**Additional properties for `output` when `connector` is `clickhouse`**

- **`type`** - _[string]_ - Type to materialize the model into. Can be 'TABLE', 'VIEW' or 'DICTIONARY'

- **`columns`** - _[string]_ - Column names and types. Can also include indexes. If unspecified, detected from the query.

- **`engine_full`** - _[string]_ - Full engine definition in SQL format. Can include partition keys, order, TTL, etc.

- **`engine`** - _[string]_ - Table engine to use. Default is MergeTree

- **`order_by`** - _[string]_ - ORDER BY clause.

- **`partition_by`** - _[string]_ - Partition BY clause.

- **`primary_key`** - _[string]_ - PRIMARY KEY clause.

- **`sample_by`** - _[string]_ - SAMPLE BY clause.

- **`ttl`** - _[string]_ - TTL settings for the table or columns.

- **`table_settings`** - _[string]_ - Table-specific settings.

- **`query_settings`** - _[string]_ - Settings used in insert/create table as select queries.

- **`distributed_settings`** - _[string]_ - Settings for distributed table.

- **`distributed_sharding_key`** - _[string]_ - Sharding key for distributed table.

- **`dictionary_source_user`** - _[string]_ - User for accessing the source dictionary table (used if type is DICTIONARY).

- **`dictionary_source_password`** - _[string]_ - Password for the dictionary source user.

## Common Properties

### `name`
Expand All @@ -241,4 +241,8 @@ _[object]_ - Overrides any properties in development environment.

### `prod`

_[object]_ - Overrides any properties in production environment.
_[object]_ - Overrides any properties in production environment.

## Depending on the connector, additional properties may be required

Depending on the connector, additional properties may be required, for more information see the [connectors](./connectors.md) documentation
68 changes: 18 additions & 50 deletions docs/docs/hidden/yaml/alerts.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,28 +40,38 @@ _[string]_ - Display name for the alert

_[string]_ - Description for the alert

### `intervals`

_[object]_ - define the interval of the alert to check

- **`duration`** - _[string]_ - a valid ISO8601 duration to define the interval duration

- **`limit`** - _[integer]_ - maximum number of intervals to check for on invocation

- **`check_unclosed`** - _[boolean]_ - boolean, whether unclosed intervals should be checked

### `watermark`

_[string]_ - Specifies how the watermark is determined for incremental processing. Use 'trigger_time' to set it at runtime or 'inherit' to use the upstream model's watermark.

### `timeout`

_[string]_ - define the timeout of the alert in seconds (optional).

### `data`

_[oneOf]_ - Data source for the alert _(required)_

- **option 1** - _[object]_ - Executes a raw SQL query against the project's data models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here. we should not remove options.


- **`sql`** - _[string]_ - Raw SQL query to run against existing models in the project. _(required)_

- **`connector`** - _[string]_ - specifies the connector to use when running SQL or glob queries.

- **option 2** - _[object]_ - Executes a SQL query that targets a defined metrics view.

- **`metrics_sql`** - _[string]_ - SQL query that targets a metrics view in the project _(required)_

- **option 3** - _[object]_ - Calls a custom API defined in the project to compute data.

- **`api`** - _[string]_ - Name of a custom API defined in the project. _(required)_

- **`args`** - _[object]_ - Arguments to pass to the custom API.

- **option 4** - _[object]_ - Uses a file-matching pattern (glob) to query data from a connector.

- **`glob`** - _[anyOf]_ - Defines the file path or pattern to query from the specified connector. _(required)_

- **option 1** - _[string]_ - A simple file path/glob pattern as a string.
Expand All @@ -70,50 +80,18 @@ _[oneOf]_ - Data source for the alert _(required)_

- **`connector`** - _[string]_ - Specifies the connector to use with the glob input.

- **option 5** - _[object]_ - Uses the status of a resource as data.

- **`resource_status`** - _[object]_ - Based on resource status _(required)_

- **`where_error`** - _[boolean]_ - Indicates whether the condition should trigger when the resource is in an error state.

```yaml
resource_status:
where_error: true
```


### `intervals`

_[object]_ - define the interval of the alert to check

- **`duration`** - _[string]_ - a valid ISO8601 duration to define the interval duration

- **`limit`** - _[integer]_ - maximum number of intervals to check for on invocation

- **`check_unclosed`** - _[boolean]_ - boolean, whether unclosed intervals should be checked

### `watermark`

_[string]_ - Specifies how the watermark is determined for incremental processing. Use 'trigger_time' to set it at runtime or 'inherit' to use the upstream model's watermark.

### `timeout`

_[string]_ - define the timeout of the alert in seconds (optional).

### `for`

_[oneOf]_ - Specifies how user identity or attributes should be evaluated for security policy enforcement.

- **option 1** - _[object]_ - Specifies a unique user identifier for applying security policies.

- **`user_id`** - _[string]_ - The unique user ID used to evaluate security policies. _(required)_

- **option 2** - _[object]_ - Specifies a user's email address for applying security policies.

- **`user_email`** - _[string]_ - The user's email address used to evaluate security policies. _(required)_

- **option 3** - _[object]_ - Specifies a set of arbitrary user attributes for applying security policies.

- **`attributes`** - _[object]_ - A dictionary of user attributes used to evaluate security policies. _(required)_

### `on_recover`
Expand All @@ -136,16 +114,6 @@ _[boolean]_ - Enable repeated notifications for unresolved alerts. Defaults to f

_[string]_ - Defines the re-notification interval for the alert (e.g., '10m','24h'), equivalent to snooze duration in UI, defaults to 'Off'

### `condition`

_[object]_ - Condition that triggers the alert

- **`operator`** - _[string]_ - Comparison operator (gt, lt, eq, etc.)

- **`threshold`** - _[no type]_ - Threshold value for the condition

- **`measure`** - _[string]_ - Measure to compare against the threshold

### `notify`

_[object]_ - Notification configuration _(required)_
Expand Down
Loading