diff --git a/docs/blog/0.49.md b/docs/blog/0.49.md index 2abaa7ca31f..3f4ddd715ed 100644 --- a/docs/blog/0.49.md +++ b/docs/blog/0.49.md @@ -30,7 +30,7 @@ You can now run Rill on ARM based Linux machines! ## Bug Fixes and Misc -- Starting in v0.49, configured source refreshes will *only* be applied to Rill Cloud by default. If you would like source refreshes to still apply to Rill Developer (or are hosting Rill Developer), you can add `run_in_dev`: true to your source YAML. More information can be found in [our documentation](https://docs.rilldata.com/build/build/models/source-refresh#running-scheduled-refreshes-in-development). +- Starting in v0.49, configured source refreshes will *only* be applied to Rill Cloud by default. If you would like source refreshes to still apply to Rill Developer (or are hosting Rill Developer), you can add `run_in_dev`: true to your source YAML. More information can be found in [our documentation](https://docs.rilldata.com/build/build/models/data-refresh#running-scheduled-refreshes-in-development). - Added dashboard state (always include time filter) to `public-url`, allows to share URLS with specific filter / pivot table. - Added % of total support to `Alerts` and Added support for `project alerts`. - Added support to disconnect a project from GitHub. diff --git a/docs/docs/build/advanced-models/advanced-models.md b/docs/docs/build/advanced-models/advanced-models.md index 08382475764..acb0c90711f 100644 --- a/docs/docs/build/advanced-models/advanced-models.md +++ b/docs/docs/build/advanced-models/advanced-models.md @@ -12,17 +12,16 @@ Unlike SQL models, YAML file models allow you to fine-tune a model to perform ad ## When to use Advanced Models? -For the majority of users on a DuckDB backed Rill project, advanced models are not required. When a project gets larger and refreshing the whole datasets becomes a time-consuming and costly task, we introduce incremental ingestion to help alleviate the problem. Along with incremental modeling, we use partitions to divide a dataset into smaller, more manageable datasets. When enabling partitions, you are able to refresh individual sections of data if required. - -Another use case is when using multiple OLAP engines. This allows you to specify where your SQL is running. While we do not officially support ClickHouse modeling, this is available behind a feature flag `clickhouseModeling`. When both DuckDB and ClickHouse are enabled in a single environment, you will need to define `connector: duckdb/clickhouse` in the YAML to tell Rill where to run the SQL query. +For the majority of users on a DuckDB backed Rill project, advanced models are not required. When a project gets larger and refreshing the whole datasets becomes a time-consuming and costly task, we introduce incremental ingestion to help alleviate the problem. Along with incremental modeling, we use partitions to divide a dataset into smaller, more manageable partitonis. After enabling partitions, you will be able to refresh individual partitonis of data when required. +Another use case is when using multiple OLAP engines. This allows you to specify where your SQL query is running. When both DuckDB and ClickHouse are enabled in a single environment, you will need to define `connector: duckdb/clickhouse` in the YAML to tell Rill where to run the SQL query, as well as define the `output` location. For more information, refer to the [YAML reference](/reference/project-files/advanced-models) ## Types of Advanced Models 1. [Incremental Models](./incremental-models) 2. [Partitioned Models](./partitions) 3. [Staging Models](./staging) -4. [DuckDB `pre_exec`/`post_exec` Models](#duckdb-models-pre-exec-sql-post-exec) +4. [DuckDB `pre_exec`/`post_exec` Models](#duckdb-models-pre-exec-post-exec-sql) ## Creating an Advanced Model @@ -36,19 +35,23 @@ type: model connector: duckdb sql: select * from + +output: + connector: duckdb + table: output_name ``` Please refer to [our reference documentation](../../reference/project-files/advanced-models) linked above for the available parameters to set in your model. :::note -Currently, there isn't a UI button to start off with an advanced model YAML. Creating a model in Rill will always create a model.sql file. +Currently, there isn't a UI button to start off with an advanced model YAML. Creating a model in Rill via the UI will always create a model.sql file. Instead, start with a blank file and rename it model_name.yaml and add the above sample code. ::: -## DuckDB Model's pre-exec, SQL, post-exec +## DuckDB Model's pre-exec, post-exec SQL While we install a set of core libraries and extensions with our embedded DuckDB, there might be specific use cases where you want to add a different one. In order to do this, you will need to use the pre-exec parameter to ensure that everything is loaded before running your SQL query. diff --git a/docs/docs/build/canvas/customization.md b/docs/docs/build/canvas/customization.md index f9c04038e54..831bc479e2d 100644 --- a/docs/docs/build/canvas/customization.md +++ b/docs/docs/build/canvas/customization.md @@ -72,7 +72,7 @@ defaults: ## Row Access Policies ### Security -Defining security policies for your data is crucial. For more information, please refer to our [Dashboard Access Policies](/manage/security.md). +Defining security policies for your data is crucial. For more information, please refer to our [Dashboard Access Policies](/build/metrics-view/security). ## Changing Themes & Colors diff --git a/docs/docs/build/dashboards/customize.md b/docs/docs/build/dashboards/customize.md index 37e05174c5e..18d68aba727 100644 --- a/docs/docs/build/dashboards/customize.md +++ b/docs/docs/build/dashboards/customize.md @@ -100,7 +100,7 @@ defaults: ## Row Access Policies ### Security -Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/manage/security.md). +Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/build/metrics-view/security). ## Changing Themes & Colors diff --git a/docs/docs/build/debugging/rill-logs.md b/docs/docs/build/debugging/rill-logs.md new file mode 100644 index 00000000000..d6b0f7bf39a --- /dev/null +++ b/docs/docs/build/debugging/rill-logs.md @@ -0,0 +1,145 @@ +--- +title: "Rill Project Logs" +description: Alter dashboard look and feel +sidebar_label: "Rill Project Logs" +sidebar_position: 30 +--- + +Whether you start Rill from the terminal or your favorite IDE, that window will output the project logs. From reconciling items to partition ingestion and beyond, browsing the project logs is a great place to start when troubleshooting errors or slow loading models. + + +## Dissecting the Format of Common Logs + +```bash +Reconciled resource {"name": "commits__ (copy)_metrics_explore", "type": "Explore", "elapsed": "1ms"} +Executed model partition {"model": "CH_incremental_commits_directory", "key": "55454ed4ad31cd3266988fe523103637", "data": {"path":"github-analytics/Clickhouse/2025/08","uri":"gs://rilldata-public/github-analytics/Clickhouse/2025/08"}, "elapsed": "283.188333ms"} +# debug +Executed model partition {"model": "staging_to_CH", "key": "0030406e528b3799c8cbad6bfe609e83", "trace_id": "3073a89ac5cee9e7e3433ce0a34d291a", "span_id": "c3cb402d7b4af9b6", "data": {"day":"2022-12-20T00:00:00Z"}} +# verbose +grpc finished call {"protocol": "grpc", "peer.address": "::1", "grpc.component": "server", "grpc.method_type": "unary", "grpc.method": "/rill.runtime.v1.RuntimeService/GetResource", "instance_id": "default", "args.instance_id": "default", "args.name.kind": "rill.runtime.v1.Theme", "args.name.name": "theme", "args.skip_security_checks": false, "grpc.code": "OK", "duration": "38.125µs"} +``` + +### Generic Logging + +- **`name`** – Filename or YAML-defined name of the Rill object. +- **`type`** – Resource type (e.g., `Connector`, `Model`, `MetricsView`, `Explore`, `API`, `Alert`, `Theme`, `Component`, `Canvas`). +- **`elapsed`** – Time taken to reconcile, execute, or otherwise process the resource. +- **`error`** – Error message generated during reconciliation or execution. +- **`dependency_error`** – Boolean flag indicating that the resource failed due to another resource's error. +- **`deleted`** – Boolean flag indicating that the resource was deleted. +- **`path`** – Filesystem path to the resource YAML file. +- **`logger_name`** – Name of the logger emitting the message. +- **`message`** – Log message content. + +### Partitioning +- **`model`** – Name of the model associated with a partition operation. +- **`partitions`** – Number of partitions resolved for a model. +- **`key`** – Partition key (usually an MD5-like hash). +- **`data`** – Partition-specific metadata or parameters. + +### Embedded ClickHouse +- **`addr`** – Host and port address for the embedded ClickHouse server. + +### Debug +- **`sql`** – SQL statement executed by DuckDB during model evaluation or metrics computation. +- **`args`** – SQL query parameters (if any). +- **`trace_id`** – Unique trace identifier for the operation (used for distributed tracing). +- **`span_id`** – Unique span identifier within the trace. + +### Verbose +- **`protocol`** – Protocol used for gRPC calls (e.g., `"grpc"`). +- **`peer.address`** – Remote address of the gRPC peer. +- **`grpc.component`** – Component type for the gRPC call (usually `"server"`). +- **`grpc.method_type`** – gRPC method type (e.g., `"unary"`). +- **`grpc.method`** – Full gRPC method name being called. +- **`grpc.code`** – Status code returned by the gRPC call. +- **`duration`** – Duration of the gRPC call or query execution. +- **`args.instance_id`** – Instance ID argument passed in a gRPC request. +- **`args.glob`** – Glob pattern argument for file listing operations. +- **`args.kind`** – Kind of Rill resource being listed or retrieved. +- **`args.skip_security_checks`** – Boolean flag indicating if security checks were skipped. +- **`args.path`** – Path argument for file retrieval operations. +- **`args.name.kind`** – Kind of resource name provided in a `GetResource` call. +- **`args.name.name`** – Name of resource provided in a `GetResource` call. + + +## Logging Examples + +### Project Creation + +When you first initialize a Rill project, you'll see Rill reconcile a resource "duckdb" of type "connector". This is expected as we explicitly create this file to initialize a connection to our embedded DuckDB. + +```bash +Rill will create project files in "~/Desktop/GitHub/testing-folder/dsn". Do you want to continue? Yes +2025-08-05T15:45:21.932 INFO Serving Rill on: http://localhost:9009 +2025-08-05T15:45:31.491 INFO Reconciling resource {"name": "duckdb", "type": "Connector"} +2025-08-05T15:45:31.581 INFO Reconciled resource {"name": "duckdb", "type": "Connector", "elapsed": "90ms"} +``` + +### Connecting to a Data Source + +When connecting to a data source via a connector, you'll see a "Connector" being reconciled. In the case of any errors, you'll see this in both the UI and the logs. + +```bash +2025-08-05T16:50:42.393 INFO Reconciling resource {"name": "gcs", "type": "Connector"} +2025-08-05T16:50:42.432 INFO Reconciled resource {"name": "gcs", "type": "Connector", "elapsed": "39ms"} +2025-08-05T16:55:47.725 WARN Reconcile failed {"name": "gcs", "type": "Connector", "elapsed": "1ms", "error": "failed to resolve templated property \"google_application_credentials\": template: :1:6: executing \"\" at <.env.connector.gcs.google_application_credentialsss>: map has no entry for key \"google_application_credentials\""} +``` + +Once connected, you'll likely create a model and see this also reconciling in the logs. Similar to the above, if there are any issues, you'll see it both in the logs and UI. + +```bash +2025-08-05T16:43:06.403 INFO Reconciling resource {"name": "commits__", "type": "Model"} +2025-08-05T16:43:07.348 INFO Reconciled resource {"name": "commits__", "type": "Model", "elapsed": "944ms"} +#or +2025-08-05T16:58:17.137 WARN Reconcile failed {"name": "commits__", "type": "Model", "elapsed": "682ms", "error": "blob (key \"github-analytics/Clickhouse/2025/06/commits_2025_0.parquet\") (code=Unknown): storage: object doesn't exist: googleapi: Error 404: No such object: rilldata-public/github-analytics/Clickhouse/2025/06/commits_2025_0.parquet, notFound", "errorVerbose": "blob (key \"github-analytics/Clickhouse/2025/06/commits_2025_0.parquet\") (code=Unknown):\n gocloud.dev/blob.(*Bucket).Attributes\n /Users/runner/go/pkg/mod/gocloud.dev@v0.36.0/blob/blob.go:913\n - storage: object doesn't exist: googleapi: Error 404: No such object: rilldata-public/github-analytics/Clickhouse/2025/06/commits_2025_0.parquet, notFound"} +``` + +### Creating Rill Objects + +The next section of logs is creating a metrics view and explore dashboard. You'll see some errors are thrown in the metrics view and resolved in Rill Developer. + +```bash +2025-08-05T17:00:08.191 INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} +2025-08-05T17:00:08.206 WARN Reconcile failed {"name": "commits___metrics", "type": "MetricsView", "elapsed": "15ms", "error": "measure \"earliest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type\nmeasure \"latest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type"} +2025-08-05T17:00:24.948 INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} +2025-08-05T17:00:24.969 WARN Reconcile failed {"name": "commits___metrics", "type": "MetricsView", "elapsed": "21ms", "error": "measure \"earliest_commit_date_measure\" is of type CODE_TIMESTAMP, but must be a numeric type"} +2025-08-05T17:00:33.899 INFO Reconciling resource {"name": "commits___metrics", "type": "MetricsView"} +2025-08-05T17:00:33.914 INFO Reconciled resource {"name": "commits___metrics", "type": "MetricsView", "elapsed": "15ms"} +2025-08-05T17:03:21.502 INFO Reconciling resource {"name": "commits___metrics_explore", "type": "Explore"} +2025-08-05T17:03:21.503 INFO Reconciled resource {"name": "commits___metrics_explore", "type": "Explore", "elapsed": "1ms"} +``` + +### Partitioned Models + +The main takeaway for partitioned models is that you'll be able to see the number of partitions that Rill will start ingesting. This is especially important when creating [dev/prod](/deploy/templating) environments and you're trying to avoid ingesting large amounts of data locally. + +```bash +Resolved model partitions {"model": "staging_to_CH", "partitions": 16} +2025-08-05T17:19:09.269 INFO Executed model partition {"model": "staging_to_CH", "key": "0030406e528b3799c8cbad6bfe609e83", "data": {"day":"2022-12-20T00:00:00Z"}} +``` + + +## Debugging Complicated Issues + +Sometimes the default logs don't have enough information to figure out why something isn't working as planned. In these cases, you can set `--debug` or `--verbose` to get more information. + +```bash +rill start --debug +rill start --verbose +``` + + +## Rill Cloud Logs + +Similar to the Rill Developer experience, you can view the logs in Rill Cloud to see the progress of your reconciliation. + +``` +rill project logs +``` + +To continually view the progression of your logs, use `-f` or `--follow`. + +``` +rill project logs -f +``` diff --git a/docs/docs/build/debugging/trace-viewer.md b/docs/docs/build/debugging/trace-viewer.md index a165aeeff71..a2d742588e8 100644 --- a/docs/docs/build/debugging/trace-viewer.md +++ b/docs/docs/build/debugging/trace-viewer.md @@ -1,4 +1,9 @@ -# Trace Viewer in Rill Developer +--- +title: "Trace Viewer in Rill Developer" +description: Alter dashboard look and feel +sidebar_label: "Trace Viewer in Rill Developer" +sidebar_position: 00 +--- Rill Developer provides a built-in trace viewer, enabling users to visually inspect operations performed when reconciling resources or fetching data for dashboards. This helps in diagnosing performance and operational issues. diff --git a/docs/docs/build/metrics-view/customize.md b/docs/docs/build/metrics-view/customize.md index f8f15541adb..0e43796239d 100644 --- a/docs/docs/build/metrics-view/customize.md +++ b/docs/docs/build/metrics-view/customize.md @@ -31,7 +31,7 @@ The first month of the year for time grain aggregation. The valid values are 1 t **`security`** -Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/manage/security.md). +Defining security policies for your data is crucial for security. For more information on this, please refer to our [Dashboard Access Policies](/build/metrics-view/security). **`ai_instructions`** diff --git a/docs/docs/manage/security.md b/docs/docs/build/metrics-view/security.md similarity index 72% rename from docs/docs/manage/security.md rename to docs/docs/build/metrics-view/security.md index 6a50c88e124..91fa89177bc 100644 --- a/docs/docs/manage/security.md +++ b/docs/docs/build/metrics-view/security.md @@ -1,65 +1,81 @@ --- -title: "Dashboard Access Policies" -description: Granular, row-level security for dashboards -sidebar_label: "Dashboard Access Policies" -sidebar_position: 40 +title: Who Can Access Your Data +description: Control who has access to view your metrics and data +sidebar_label: Data Access Control +sidebar_position: 10 --- Rill supports granular access policies for dashboards. They allow the dashboard developer to configure dashboard-level, row-level and column-level restrictions based on user attributes such as email address and domain. Our goal with access to policies is to avoid dashboard sprawl by creating a single configuration of the dashboard that can then be sliced or restricted into multiple views via different policies. Using those access controls, a single dashboard can now serve dozens of teams and use cases to ensure consistent metric definitions and better dashboard findability. -Some of the typical use cases include: +Typical use cases include: -- Restricting access to certain dashboards to admin users -- Limiting dashboards to relevant fields or metrics by team for ease of use (creating a lookup and filter by role) -- Limiting access to sensitive dimensions or measures to users from a specific department -- Partner-filtered dashboards where external users can only see the subset of data that relates to them +- **Granting or Restricting Access** to data and as a result, dashboards +- **Hiding specific dimensions and measures** from specific groups of users, creating a tailored dashboard experience +- **Restricting Access to Internal users** of your organization, allowing specific dashboards to be viewed by internal users only +- **Partner-filtered Dashboards** where external users can only access the subset of their data +- **Embedded** use cases, passing custom attributes to Rill +- **Combination of all the above** + + +:::tip Assuming Access to Project is already given + +Access Policies assume that the user already has access to the project in Rill Cloud. For more information on user management, see our [User Management](/manage/user-management) and [Project Management](/manage/project-management) for more information! -:::note User Access vs. Access Policies -Access policies only apply to users who have been invited to access the project. They provide granular access control for your data, but are not the first layer of security for your project. ::: -## Configuration +## Configurations -You define access policies for dashboards under the `security` key in a dashboard's YAML file. The key properties are: +There are three levels of considerations for access policies. -- **Dashboard-level access:** `access` – a boolean expression that determines if a user can or can't access the dashboard -- **Row-level access:** `row_filter` – a SQL expression that will be injected into the `WHERE` clause of all dashboard queries to restrict access to a subset of rows -- **Column-level access:** `include` or `exclude` – lists of boolean expressions that determine which dimension and measure names will be available to the user +- **Data Access** - `access`– a boolean expression that determines if a user can or can't access the dashboard +- **Row-level access:** `row_filter` – a SQL expression that will be injected into the WHERE clause of all dashboard queries to restrict access to a subset of rows +- **Column-level access**: `include` or `exclude` – lists of boolean expressions that determine which dimension and measure names will be available to the user -![access](../../static/img/manage/security/access.png) + -See the [Dashboard YAML](/reference/project-files/explore-dashboards) reference docs for all the available fields. +```yaml +security: + access: true + row_filter: ... + #include: + #exclude: +``` -See the [Examples](#examples) below for how to set up each type of configuration. +For more information, see our [metric view YAML reference page](/reference/project-files/metrics-views) for more information. -## User attributes -When developing access policies, you can leverage a fixed set of user attributes to resolve access at runtime. The attributes are: +# Setting up Data Access -- `.user.email` – the current user's email address, for example `john.doe@example.com` (string) -- `.user.domain` – the domain of the current user's email address, for example `example.com` (string) -- `.user.name` - the current user's name, for example `John Doe` (string) -- `.user.admin` – a boolean value indicating whether the current user is an org or project admin, for example `true` (bool) -- `.user.groups` - a list of user groups the user belongs to in the project's org (list of strings), e.g. `["marketing","sales","finance"]` +There are two locations that control data access in Rill. -Note: Rill requires users to confirm their email address before letting them interact with the platform so a user cannot fake an email address or email domain. +### Project Level Defaults +By default, when a user is granted access to your project, they have access to all metrics views. This is not always the desired behavior as some organizations will invite partner users to the Rill Cloud UI. In these instances, project level defaults that only give access to internal domain are required. -If you require additional user attributes to enforce access policies, see the [example for custom attributes below](#advanced-example-custom-attributes-embed-dashboards) for more details. +### Metrics View Specific + +## User Attributes +- `.user.email` – the current user's email address, for example john.doe@example.com (string) +- `.user.domain` – the domain of the current user's email address, for example example.com (string) +- `.user.name` - the current user's name, for example John Doe (string) +- `.user.admin`– a boolean value indicating whether the current user is an org or project admin, for example true (bool) +- `.user.groups` - a list of user groups the user belongs to in the project's org (list of strings), e.g. ["marketing","sales","finance"] +- `.user.attribute` - where `attribute` is s custom variables that you can pass via an embedded dashboard from the application. + +Note: Rill requires users to confirm their email address before letting them interact with the platform so a user cannot fake an email address or email domain. ## Templating and expression evaluation When a user loads a dashboard, the policies are resolved in two phases: + 1. The templating engine first replaces expressions like `{{ .user.domain }}` with actual values ([Templating reference](/connect/templating)) 2. The resulting expression is then evaluated contextually: - The `access` and `if` values are evaluated as SQL expressions and resolved to a `true` or `false` value - The `row_filter` value is injected into the `WHERE` clause of the SQL queries used to render the dashboard -## Testing your policies - -### In Rill Developer +## Testing Policies in Rill Developer In development (on `localhost`), you can test your policies by adding "mock users" to your project and viewing the dashboard as one of them. @@ -78,13 +94,19 @@ mock_users: On the dashboard page (provided you've added a policy) you'll see a "View as" button in the top right corner. Click this button and select one of your mock users. You'll see the dashboard as that user would see it. -### In Rill Cloud (Admin only) +### Rill Cloud In case you want to test what your users are seeing in Rill Cloud after deploying, you can find this in the dropdown of your account. You will see the actual users in the dropdown of this list, not the mock users defined in the rill.yaml file. +****
+### Embedded Dashboards + +When [requestimg an embedded dashboard from Rill](/integrate/embedding) from your frontend, you can pass the `attributes` parameter with custom names to ensurer that the resulting dashboard is displaying the correct information. +~For more information, see [our embedding docs](/integrate/embedding#backend-build-an-iframe-url). + ## Examples diff --git a/docs/docs/build/models/source-refresh.md b/docs/docs/build/models/data-refresh.md similarity index 100% rename from docs/docs/build/models/source-refresh.md rename to docs/docs/build/models/data-refresh.md diff --git a/docs/docs/build/models/environments.md b/docs/docs/build/models/environments.md index 111d53b0150..dd7739d1cf8 100644 --- a/docs/docs/build/models/environments.md +++ b/docs/docs/build/models/environments.md @@ -78,7 +78,7 @@ refresh: :::tip Why are source refreshes only enabled by default for Rill Cloud? -Source refreshes are primarily meant to _help keep the data in your deployed dashboards on Rill Cloud up-to-date_ (without needing to manually trigger refreshes). For more details, see our documentation on [configuring source refreshes](/build/models/source-refresh). +Source refreshes are primarily meant to _help keep the data in your deployed dashboards on Rill Cloud up-to-date_ (without needing to manually trigger refreshes). For more details, see our documentation on [configuring source refreshes](/build/models/data-refresh). ::: diff --git a/docs/docs/build/models/models.md b/docs/docs/build/models/models.md index 9794d144caf..cf88fdd7b30 100644 --- a/docs/docs/build/models/models.md +++ b/docs/docs/build/models/models.md @@ -8,7 +8,7 @@ sidebar_position: 00
-In Rill, [data models](/reference/project-files/models.md) are built using SQL `SELECT` statements applied to your source data. They allow you to join, transform, and clean data. +In Rill, [data models](/reference/project-files/models.md) are built using SQL `SELECT` statements applied to your source data. They allow you to join, transform, and clean data. For simple models, we recommend .SQL files, but also provide [YAML based moddels](/build/advanced-models) for more complex setups. :::tip Avoid Pre-aggregated Metrics @@ -23,22 +23,24 @@ By default, data transformations in Rill Developer are powered by DuckDB and its It is possible to change the default [OLAP engine](https://docs.rilldata.com/connect/olap) for [the entire project](https://docs.rilldata.com/reference/project-files/rill-yaml#configuring-the-default-olap-engine) or [a specific metrics view](https://docs.rilldata.com/reference/project-files/metrics-views). You will need to define the connector credentials within your Rill project or via environment variables. -For additional tips on commonly used expressions (either in models or dashboard definitions), visit our [common expressions page](../metrics-view/advanced-expressions/advanced-expressions.md). +:::tip Support OLAP engines for modeling +We support modeling on [ClickHouse\*](/reference/olap-engines/clickhouse), [DuckDB](/reference/olap-engines/duckdb) and [MotherDuck\*](/reference/olap-engines/motherduck). For more information, see each OLAP engine page for further information. +\* indicates some caveats with modeling and encourage you to read the documentation before getting started. +::: + +For additional tips on advanced expressions (either in models or measureß definitions), visit our [advanced expressions page](../metrics-view/advanced-expressions/advanced-expressions.md). -## Adding a data model -### Using the UI -In the UI, add a new data model by clicking the '+' icon next to 'Models' in the left-hand navigation pane. You can now begin typing a DuckDB SQL `SELECT` query for your model in the code editor – with real-time feedback. +## Adding a data model -### Using code -When you add a data model using the UI, a code definition will automatically be created as a `.sql` file in the `models` folder of your Rill project. +Add a new data model by either clicking 'model' in the 'Add' menu or select the '...' in any connector view or existing model. When you add a data, a code definition will automatically be created as a `.sql` file in the `models` folder of your Rill project. You can also create a model outside of the application and add it to Rill by placing a `.sql` file in the `models` directory containing a DuckDB SQL `SELECT` statement. Rill will automatically detect and parse the model the next time you run `rill start`. -:::tip +:::tip reference page -See also our [Model YAML](../../reference/project-files/models) reference page. +See also our [Model SQL](../../reference/project-files/models) reference page. ::: @@ -50,6 +52,16 @@ It is powerful to be able to translate many ad hoc questions into a data framewo To experience the full potential of Rill, model your data sources into "One Big Table" – a granular resource that contains as much information as possible and can be rolled up in a meaningful way. This flexible OBT can be combined with a generalizable [metrics definition](/build/dashboards) to enable ad hoc slice-and-dice discovery through Rill's interactive dashboard. +:::tip materializing metrics powered models + +We recommend materializing the model that powers your [metrics view](/build/metrics-view). You can materialze a SQL model by adding this to the top of the file. This will greatly improve the performance of your dashboards. + +```sql +-- @materialize: true +``` + +::: + ### Intermediate processing Models can also be cross-referenced with each other to produce the final output for your dashboard. The advantage here is that more complex, intermediate data transformations can be utilized to achieve your final source for the dashboard. Example ideas for modeling: @@ -58,6 +70,33 @@ Models can also be cross-referenced with each other to produce the final output - Unnesting and merging complex data types - Combining multiple sources with data cleansing or transformation requirements +## Model Materializaton + +```sql +-- @materialize: true +``` + +Model materialization is something to consider when creating intermediate models. Intermediate models are not, by default, materialized and are views in your underlying database engine. There are some pros and cons of enabling it during the development process. + +The pros include improved performance for downstream models and dashboards, especially with complex logic and/or large data sizes. Some cons are certain edge cases like cross joins might have a degreaded keystroke-by-keystroke experience, and materialized models are billable. + + +If you are seeing degraded performance, the first recommendation you'll hear from us is to materialize the metrics powered model. + +### Default Model Materialization +If you want, you can change the default behavior of all models in Rill by setting the default model behavior in the rill.yaml file. + +```yaml +models: + materialize: true +``` + +To override this on a per-model basis, simply set the specific model.sql to false. +```sql +-- @materialize: false +``` + + ## Working with Pivots Pivots deserve their own section, as using the [Pivot](https://duckdb.org/docs/sql/statements/pivot) statement while modeling requires special consideration. Notably, there are a few existing DuckDB limitations to consider: diff --git a/docs/docs/build/using-ai/_category_.yml b/docs/docs/build/using-ai/_category_.yml new file mode 100644 index 00000000000..84ac1b9ccb4 --- /dev/null +++ b/docs/docs/build/using-ai/_category_.yml @@ -0,0 +1,4 @@ +position: 00 +label: Using AI +collapsible: true +collapsed: true diff --git a/docs/docs/build/using-ai/using-ai.md b/docs/docs/build/using-ai/using-ai.md new file mode 100644 index 00000000000..738f0e54ca3 --- /dev/null +++ b/docs/docs/build/using-ai/using-ai.md @@ -0,0 +1,45 @@ +--- +title: Using AI to Create a Rill Project +sidebar_label: with AI Agent +sidebar_position: 00 +--- + +Given our BI-as-code approach, it is an obvious expectation that users will build Rill projects solely with an AI agent. You may have even seen one of our demo or live discussions showcasing this feature. But, we've all experienced, even the best language models, suggest random and even outlandish properties that doesn't actually exist. How do we fix that? + +## Provide Context to your Agent +The step that is easily and quite often skipped is forgetting to provide context and guidelines for your agent to use. This w + + +``` +You are a helpful engineering assistant working in a BI-as-code environment. + +You are editing Rill project configuration files such as: +- `connector.yaml` +- `model.yaml` +- `_metrics.yaml` +- `_dashboard.yaml` + +These are YAML files used to configure dashboards, metrics, and data models in a declarative way. + +You MUST follow these rules: +1. Do NOT guess or invent YAML keys or values. +2. Only use keys and structures defined in the official Rill documentation and public GitHub repo: + - https://docs.rilldata.com/reference/project-files/ + - https://github.com/rilldata/rill + - https://github.com/rilldata/rill-examples +3. If you are unsure whether a key or value is valid, **either omit it** or include a comment such as: + `# TODO: Confirm if 'xyz' is a valid property` +4. Prefer minimal, working examples over speculative or verbose ones. +5. Always follow the existing structure and indentation of the file. +6. When referencing fields (e.g., dimensions, measures), assume they are defined in the referenced dataset and DO NOT make up field names. + +You are not allowed to: +- Invent properties not found in the schema +- Use placeholder keys like `example: true` +- Generate broken or non-compiling YAML + +Your goal is to produce YAML that will **pass `rill lint`**, assuming schema and project context are correct. + +If there is an issue during YAML file creation, review the logs and fix the issues. If you are unable to figure it out, comment the lines out. +``` + diff --git a/docs/docs/explore/alerts/alerts.md b/docs/docs/explore/alerts/alerts.md index ccea3832562..adc1752fc30 100644 --- a/docs/docs/explore/alerts/alerts.md +++ b/docs/docs/explore/alerts/alerts.md @@ -72,7 +72,7 @@ On the second tab, you will have the opportunity to specify the criteria for whi ### Delivery -On the final tab, you will choose how and where your alert is delivered. By default, the alert will be checked whenever the source data is [refreshed](/build/models/source-refresh). There are a few additional things worth noting: +On the final tab, you will choose how and where your alert is delivered. By default, the alert will be checked whenever the source data is [refreshed](/build/models/data-refresh). There are a few additional things worth noting: 1. To limit the number of alerts, you can set an optional **Snooze** period after an alert is triggered. 2. Depending on the available notification targets (see next section), choose which targets and/or users to subscribe to the alert. diff --git a/docs/docs/explore/canvas.md b/docs/docs/explore/canvas.md new file mode 100644 index 00000000000..b9b475e9ca1 --- /dev/null +++ b/docs/docs/explore/canvas.md @@ -0,0 +1,75 @@ +--- +title: Canvas Dashboard +sidebar_label: "Canvas Dashboard" +sidebar_position: 05 +--- + + +
+ +
+
+ +Prefer video? Check out our [YouTube playlist](https://www.youtube.com/watch?v=wTP46eOzoCk&list=PL_ZoDsg2yFKgi7ud_fOOD33AH8ONWQS7I&index=1) for a quick start! + + +## Overview + +After logging into [Rill Cloud](https://ui.rilldata.com), you should see all projects within your [organization](/manage/organization-management#organization) that is available and/or has been granted permissions to your user profile. Within each project, you'll then be able to access the corresponding individual dashboards that belong to a particular Rill project. + + +
+ + +## Navigating the Dashboard + + +
+ + +Similar to our [Explore dashboards](/explore/dashboard-101), Canvas Dashboards also include a similar navigation bar to control the dashboard components. + +### Navigation Bar + +- _**Time Selector and Time Selector Comparison:**_ You can change the period of analysis to different ranges of time (see `red` box), either by selecting from a pre-defined period (such as last week) or choosing a custom date range. Along with this, you can enable a comparison filter to compare range of dates with 1 click. + +- _**Filtering:**_ Underneath the time selector, you'll also be able to find your filter bar (see `orange` box) where you can [add filters](/explore/filters/filters.md) for metrics (e.g. `campaigns>1000`) or for dimensions (e.g. `campaign_name = Instacart`). + +:::tip identical names in metrics views + + If your dimensions or measures have the same name in your metrics view, filters will apply to all components, regardless if it's in a different file. + ::: + + + +## Component Navigation + +
+ + +If you want to further drill into a component's data, select the top right button to take you to the equivalent explore dashboard. + +:::tip no button? + +If no explore dashboard exists, and/or you don't have permissions to view it, no button will appear and is as designed. + +::: \ No newline at end of file diff --git a/docs/docs/explore/dashboard-101/_category.yml b/docs/docs/explore/dashboard-101/_category.yml index 6d43fe540e2..eb96561dd26 100644 --- a/docs/docs/explore/dashboard-101/_category.yml +++ b/docs/docs/explore/dashboard-101/_category.yml @@ -1,4 +1,4 @@ position: 00 -label: Dashboard Quickstart +label: Explore Dashboard Quickstart collapsible: true collapsed: true diff --git a/docs/docs/explore/dashboard-101/dashboard-101.md b/docs/docs/explore/dashboard-101/dashboard-101.md index 6a88a85eab6..6b16d2e8d76 100644 --- a/docs/docs/explore/dashboard-101/dashboard-101.md +++ b/docs/docs/explore/dashboard-101/dashboard-101.md @@ -1,7 +1,7 @@ --- title: "Dashboard Quickstart" description: Dashboard Quickstart -sidebar_label: "Dashboard Quickstart" +sidebar_label: "Explore Dashboard Quickstart" sidebar_position: 03 --- diff --git a/docs/docs/hidden/yaml/project.md b/docs/docs/hidden/yaml/project.md index 75f353e7966..f897a7e4caa 100644 --- a/docs/docs/hidden/yaml/project.md +++ b/docs/docs/hidden/yaml/project.md @@ -74,7 +74,7 @@ _[object]_ - Defines project-wide default settings for explores. Unless overridd ```yaml # For example, the following YAML configuration below will set a project-wide default for: -# Models - Configure a [source refresh](/build/models/source-refresh). +# Models - Configure a [source refresh](/build/models/data-refresh). # Metrics View - Set the [first day of the week](metrics-view.md) for timeseries aggregations to be Sunday along with setting the smallest_time_grain. # Explore Dashboards - Set the [default](explore-dashboards.md) values when a user opens a dashboard, and available time zones and/or time ranges. models: @@ -162,7 +162,7 @@ ignore_paths: ## Testing access policies -During development, it is always a good idea to check if your [access policies](/manage/security.md) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. +During development, it is always a good idea to check if your [access policies](/build/metrics-view/security) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. :::info The View as selector is not visible in my dashboard, why? This feature is _only_ enabled when you have set a security policy on the dashboard. By default, the dashboard and it's contents is viewable by every user. ::: diff --git a/docs/docs/reference/project-files/rill-yaml.md b/docs/docs/reference/project-files/rill-yaml.md index 7c73f5d4a5b..814f3ebb6f1 100644 --- a/docs/docs/reference/project-files/rill-yaml.md +++ b/docs/docs/reference/project-files/rill-yaml.md @@ -65,7 +65,7 @@ In your `rill.yaml`, the top level property for the resource type needs to be ** ::: For example, the following YAML configuration below will set a project-wide default for: -- **Sources** - Configure a [source refresh](/build/models/source-refresh). +- **Sources** - Configure a [source refresh](/build/models/data-refresh). - **Models** - Automatically materialize the models as tables instead of views (the default behavior if unspecified). - **Metrics View** - Set the [first day of the week](metrics-views.md) for timeseries aggregations to be Sunday along with setting the smallest_time_grain. - **Explore Dashboards** - Set the [default](explore-dashboards.md) values when a user opens a dashboard, and available time zones and/or time ranges. @@ -175,7 +175,7 @@ Don't forget the leading `/` when specifying the path for `ignore_paths` and thi ## Testing access policies -During development, it is always a good idea to check if your [access policies](/manage/security.md) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. +During development, it is always a good idea to check if your [access policies](/build/metrics-view/security) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. ```yaml mock_users: diff --git a/docs/docusaurus.config.js b/docs/docusaurus.config.js index f18235d4107..c36319780fb 100644 --- a/docs/docusaurus.config.js +++ b/docs/docusaurus.config.js @@ -264,7 +264,7 @@ const config = { }, { from: '/deploy/source-refresh', - to: '/build/models/source-refresh' + to: '/build/models/data-refresh' }, { from: '/reference/templating', @@ -290,6 +290,10 @@ const config = { from: '/share/scheduled-reports', to: '/explore/exports' }, + { + from: '/manage/security', + to: '/build/metrics-view/security' + }, // OLAP Engine redirects { from: '/reference/olap-engines/', @@ -417,6 +421,7 @@ const config = { // from: '/old-page', // to: '/new-page', // } + ], }, ], diff --git a/docs/static/img/explore/canvas/canvas-dashboard.png b/docs/static/img/explore/canvas/canvas-dashboard.png new file mode 100644 index 00000000000..4d61db2146a Binary files /dev/null and b/docs/static/img/explore/canvas/canvas-dashboard.png differ diff --git a/docs/static/img/explore/canvas/canvas-navigaton.png b/docs/static/img/explore/canvas/canvas-navigaton.png new file mode 100644 index 00000000000..00bf63dc390 Binary files /dev/null and b/docs/static/img/explore/canvas/canvas-navigaton.png differ diff --git a/runtime/parser/schema/rillyaml.schema.yaml b/runtime/parser/schema/rillyaml.schema.yaml index 8c5a1fd59ae..2e3107e815a 100644 --- a/runtime/parser/schema/rillyaml.schema.yaml +++ b/runtime/parser/schema/rillyaml.schema.yaml @@ -62,7 +62,7 @@ allOf: description: Defines project-wide default settings for explores. Unless overridden, individual explores will inherit these defaults. examples: - # For example, the following YAML configuration below will set a project-wide default for: - # Models - Configure a [source refresh](/build/build/models/source-refresh.md). + # Models - Configure a [source refresh](/build/build/models/data-refresh.md). # Metrics View - Set the [first day of the week](metrics-view.md) for timeseries aggregations to be Sunday along with setting the smallest_time_grain. # Explore Dashboards - Set the [default](explore-dashboards.md) values when a user opens a dashboard, and available time zones and/or time ranges. models: @@ -147,7 +147,7 @@ allOf: type: string - title: Testing access policies description: | - During development, it is always a good idea to check if your [access policies](/manage/security.md) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. + During development, it is always a good idea to check if your [access policies](/build/metrics-view/security) are behaving the way you designed them to before pushing these changes into production. You can set mock users which enables a drop down in the dashboard preview to view as a specific user. :::info The View as selector is not visible in my dashboard, why? This feature is _only_ enabled when you have set a security policy on the dashboard. By default, the dashboard and it's contents is viewable by every user. :::