<div align="center"><h1 align="center">Database Lab Engine (DLE)</h1></div>

<a href="https://twitter.com/intent/tweet?via=Database_Lab&url=https://github.com/postgres-ai/database-lab-engine/&text=Thin%20@PostgreSQL%20clones%20–%20DLE%20provides%20blazing-fast%20database%20cloning%20to%20build%20powerful%20development,%20test,%20QA,%20staging%20environments.">

<strong>:zap: Blazing-fast cloning of PostgreSQL databases :elephant:</strong><br>

Thin clones of PostgreSQL to build powerful development, test, QA, and staging environments.<br>

Available for any PostgreSQL, including AWS RDS^*, GCP CloudSQL^*, Heroku^*, Digital Ocean^*, and self-managed instances.

_{^* For a managed PostgreSQL cloud service such as AWS RDS or Heroku, where physical connection and access to PGDATA are not available, DLE is supposed to be running on a separate VM in the same region, performing periodical automated full refresh of data and serving itself as a database-as-a-service solution providing thin database clones for development and testing purposes.}

- Build dev/QA/staging environments based on full-size production-like databases.

- Automatically test database changes in CI/CD pipelines to avoid incidents in production.

For example, cloning a 1 TiB PostgreSQL database takes ~10 seconds. Dozens of independent clones are up and running on a single machine, supporting lots of development and testing activities, without increasing costs for hardware.

- enter [the Database Lab Platform](https://console.postgres.ai/), join the "Demo" organization, and test cloning of ~1 TiB demo database, or

- check out another demo setup, DLE CE: https://demo.aws.postgres.ai:446/instance, use the token `demo_token` to enter

Thin cloning is fast because it uses [Copy-on-Write (CoW)](https://en.wikipedia.org/wiki/Copy-on-write#In_computer_storage). DLE supports two technologies to enable CoW and thin cloning: [ZFS](https://en.wikipedia.org/wiki/ZFS) (default) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)).

With ZFS, Database Lab Engine periodically creates a new snapshot of the data directory and maintains a set of snapshots, cleaning up old and unused ones. When requesting a new clone, users can choose which snapshot to use.

- [Database Lab tutorial for any PostgreSQL database](https://postgres.ai/docs/tutorials/database-lab-tutorial)

- [Database Lab tutorial for Amazon RDS](https://postgres.ai/docs/tutorials/database-lab-tutorial-amazon-rds)

- [Terraform module template (AWS)](https://postgres.ai/docs/how-to-guides/administration/install-database-lab-with-terraform)

- Qiwi: [How Qiwi Controls the Data to Accelerate Development](https://postgres.ai/resources/case-studies/qiwi)

- Blazing-fast cloning of Postgres databases – a few seconds to create a new clone ready to accept connections and queries, regardless of database size.

- The theoretical maximum number of snapshots and clones is 2⁶⁴ ([ZFS](https://en.wikipedia.org/wiki/ZFS), default).

- The theoretical maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2¹²⁸ bytes ([ZFS](https://en.wikipedia.org/wiki/ZFS), default).

- PostgreSQL major versions supported: 9.6–14.

- Two technologies are supported to enable thin cloning ([CoW](https://en.wikipedia.org/wiki/Copy-on-write)): [ZFS](https://en.wikipedia.org/wiki/ZFS) and [LVM](https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)).

- All components are packaged in Docker containers.

- UI to make manual work more convenient.

- API and CLI to automate the work with DLE snapshots and clones.

- By default, PostgreSQL containers include many popular extensions ([docs](https://postgres.ai/docs/database-lab/supported-databases#extensions-included-by-default)).

- PostgreSQL containers can be customized ([docs](https://postgres.ai/docs/database-lab/supported-databases#how-to-add-more-extensions)).

- Source database can be located anywhere (self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, Timescale Cloud, and so on) and does NOT require any adjustments. There are NO requirements to install ZFS or Docker to the source (production) databases.

- Initial data provisioning can be done at either the physical (pg_basebackup, backup / archiving tools such as WAL-G or pgBackRest) or logical (dump/restore directly from the source or from files stored at AWS S3) level.

- For logical mode, partial data retrieval is supported (specific databases, specific tables).

- For physical mode, a continuously updated state is supported ("sync container"), making DLE a specialized version of standby Postgres.

- For logical mode, periodic full refresh is supported, automated, and controlled by DLE. It is possible to use multiple disks containing different versions of the database, so full refresh won't require downtime.

- Fast Point in Time Recovery (PITR) to the points available in DLE snapshots.

- Unused clones are automatically deleted.

- "Deletion protection" flag can be used to block automatic or manual deletion of clones.

- Snapshot retention policies supported in DLE configuration.

- Persistent clones: clones survive DLE restarts (including full VM reboots).

- The "reset" command can be used to switch to a different version of data.

- DB Migration Checker component collects various artifacts useful for DB testing in CI ([docs](https://postgres.ai/docs/db-migration-checker)).

- SSH port forwarding for API and Postgres connections.

- Docker container config parameters can be specified in the DLE config.

- Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)

- Postgres config parameters can be specified in the DLE config (separately for clones, the "sync" container, and the "promote" container).

- Monitoring: auth-free `/healthz` API endpoint, extended `/status` (requires auth), [Netdata module](https://gitlab.com/postgres-ai/netdata_for_dle).

The easiest way to contribute is to give the project a GitHub/GitLab star:

Post a tweet mentioning [@Database_Lab](https://twitter.com/Database_Lab) or share the link to this repo in your favorite social network.

If you are actively using DLE, tell others about your experience. You can use the logo referenced below and stored in the `./assets` folder. Feel free to put them in your documents, slide decks, application, and website interfaces to show that you use DLE.

Making Database Lab Engine more accessible to engineers around the Globe is a great help for the project. Check details in the [translation section of contributing guide](./CONTRIBUTING.md#Translation).

- [DLE components](https://postgres.ai/docs/reference-guides/database-lab-engine-components)

- [DLE configuration reference](https://postgres.ai/docs/database-lab/config-reference)

- [DLE API reference](https://postgres.ai/swagger-ui/dblab/)

- [How to install Database Lab with Terraform on AWS](https://postgres.ai/docs/how-to-guides/administration/install-database-lab-with-terraform)

- [How to manage DLE](https://postgres.ai/docs/how-to-guides/administration)

- [DLE Docker images](https://hub.docker.com/r/postgresai/dblab-server)

DLE source code is licensed under the OSI-approved open source license GNU Affero General Public License version 3 (AGPLv3).

[](https://app.fossa.io/projects/git%2Bgithub.com%2Fpostgres-ai%2Fdatabase-lab-engine?ref=badge_large)

- If you need to report a security issue, follow instructions in ["Database Lab Engine security guidelines"](./SECURITY.md).

- [German / Deutsch](translations/README.german.md) (🙏 [@ane4ka](https://github.com/ane4ka))

- [Brazilian Portuguese / Português (BR)](translations/README.portuguese-br.md) (🙏 [@Alexand](https://gitlab.com/Alexand))

- [Russian / Pусский](translations/README.russian.md) (🙏 [@Tanya301](https://github.com/Tanya301))

- [Spanish / Español](translations/README.spanish.md) (🙏 [@asotolongo](https://gitlab.com/asotolongo))

- [Ukrainian / Українська](translations/README.ukrainian.md) (🙏 [@denis-boost](https://github.com/denis-boost))