Skip to content

feat: Add built-in PostgreSQL for simple production setup #2345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 15, 2022
Merged

feat: Add built-in PostgreSQL for simple production setup #2345

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dc122b6
feat: Add built-in PostgreSQL for simple production setup
kylecarbs Jun 14, 2022
0450fcb
Merge branch 'main' into embeddeddb
kylecarbs Jun 15, 2022
0856209
Merge branch 'main' into embeddeddb
kylecarbs Jun 15, 2022
7fd3560
Use fork of embedded-postgres for cache path
kylecarbs Jun 15, 2022
690c791
Merge branch 'main' into embeddeddb
kylecarbs Jun 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into embeddeddb
  • Loading branch information
@kylecarbs
kylecarbs committed Jun 15, 2022
commit 0450fcbcef2cb1afae439d14c742db329ba1348a
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ curl -fsSL https://coder.com/install.sh | sh
Once installed, you can start a production deployment with a single command:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but turning the DB off if the single instance of the app goes down is not what a lot of people would consider "production"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do essentially this same thing right now on our dogfood deployment, which I'd consider a production deployment of Coder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @f0ssel here -- I'd be more comfortable with a "traditionally-managed" database if I were setting up a prod deployment than just using the embedded one. Doesn't detract from the value of being able to instantly kick the tyres with a real database!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why though? I believe users should start real, production deployments with this setup. There isn't a technical reason why it's bad practice. Users can just backup the ~/.config/coder/postgres directory on a CRON.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair fair!

Copy link
Contributor

@f0ssel f0ssel Jun 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coupling the lifecycle of the database to the app lifecycle and running the DB on the same host are two pretty big red flags for any engineers that have to actually manage and troubleshoot the app in production.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are the specific red flags? You can still run the database independently.

Copy link
Contributor

@f0ssel f0ssel Jun 15, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm saying that if a customer wanting to run 1000 devs and it be production ready I'd probably advise them:

  • Run the DB on a dedicated host
  • Run regular database backups and have a restore process
  • Run multiple instances of the app server, optionally on multiple hosts behind a LB, optionally across different AZs.
  • Have a way to run new webserver versions without causing downtime

Right now this command not only does none of those things, but also won't even allow you to run multiple instances of the web app since it will conflict on the DB setup.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And similar to why people recommend not putting DB healthchecks in the webserver healthchecks on kubernetes - you don't want your DB taking your webapp down and vice versa, you want their lifecycles independent for obvious reasons.


```sh
# Automatically sets up PostgreSQL and an external access URL on *.try.coder.app
coder server --postgres-builtin --tunnel
# Automatically sets up an external access URL on *.try.coder.app
coder server --tunnel

# Requires a PostgreSQL instance and external access URL
coder server --postgres-url <url> --access-url <url>
Expand Down
7 changes: 4 additions & 3 deletions cli/login.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import (
"github.com/spf13/cobra"
"golang.org/x/xerrors"

"github.com/coder/coder/cli/cliflag"
"github.com/coder/coder/cli/cliui"
"github.com/coder/coder/codersdk"
)
Expand Down Expand Up @@ -230,9 +231,9 @@ func login() *cobra.Command {
return nil
},
}
cmd.Flags().StringVarP(&email, "email", "e", "", "Specifies an email address to authenticate with.")
cmd.Flags().StringVarP(&username, "username", "u", "", "Specifies a username to authenticate with.")
cmd.Flags().StringVarP(&password, "password", "p", "", "Specifies a password to authenticate with.")
cliflag.StringVarP(cmd.Flags(), &email, "email", "e", "CODER_EMAIL", "", "Specifies an email address to authenticate with.")
cliflag.StringVarP(cmd.Flags(), &username, "username", "u", "CODER_USERNAME", "", "Specifies a username to authenticate with.")
cliflag.StringVarP(cmd.Flags(), &password, "password", "p", "CODER_PASSWORD", "", "Specifies a password to authenticate with.")
return cmd
}

Expand Down
31 changes: 6 additions & 25 deletions cli/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ func server() *cobra.Command {
inMemoryDatabase bool
// provisionerDaemonCount is a uint8 to ensure a number > 0.
provisionerDaemonCount uint8
postgresBuiltin bool
postgresURL string
oauth2GithubClientID string
oauth2GithubClientSecret string
Expand Down Expand Up @@ -134,35 +133,19 @@ func server() *cobra.Command {
}

config := createConfig(cmd)
// For in-memory, disable embedded PostgreSQL!
// This is primarily for testing and development.
if inMemoryDatabase {
postgresBuiltin = false
postgresURL = ""
}
// Only use built-in if PostgreSQL URL isn't specified!
if postgresURL == "" && postgresBuiltin {
if postgresURL == "" {
var closeFunc func() error
cmd.Printf("Using built-in PostgreSQL (%s)\n", config.PostgresPath())
postgresURL, closeFunc, err = startBuiltinPostgres(cmd.Context(), config, logger)
if err != nil {
return err
}
cmd.Printf("Using built-in PostgreSQL (%s)\n", config.PostgresPath())
defer func() {
// Gracefully shut PostgreSQL down!
_ = closeFunc()
}()
}
// If the built-in PostgreSQL isn't used and an external URL isn't specified,
// recommend usage of the built-in!
if !inMemoryDatabase && postgresURL == "" {
cmd.PrintErrln(cliui.Styles.Error.Render("PostgreSQL >=13 is required for running Coder.") + "\n")
cmd.PrintErrln("Automatically install and run PostgreSQL (install and runtime data goes to " + config.PostgresPath() + "):")
cmd.PrintErrln(cliui.Styles.Code.Render(strings.Join(os.Args, " ") + " --postgres-builtin"))
cmd.PrintErrln("\nUse an external PostgreSQL deployment:")
cmd.PrintErrln(" coder server --postgres-url <url>")
return xerrors.New("")
}

listener, err := net.Listen("tcp", address)
if err != nil {
Expand Down Expand Up @@ -221,9 +204,9 @@ func server() *cobra.Command {
if isLocal || err != nil {
reason := "could not be resolved"
if isLocal {
reason = "isn't reachable externally"
reason = "isn't externally reachable"
}
cmd.Printf("%s The access URL %s %s. Workspaces must be able to reach Coder from this URL. Optionally, generate a unique *.try.coder.app URL with:\n", cliui.Styles.Warn.Render("Warning:"), cliui.Styles.Field.Render(accessURL), reason)
cmd.Printf("%s The access URL %s %s, this may cause unexpected problems when creating workspaces. Generate a unique *.try.coder.app URL with:\n", cliui.Styles.Warn.Render("Warning:"), cliui.Styles.Field.Render(accessURL), reason)
cmd.Println(cliui.Styles.Code.Render(strings.Join(os.Args, " ") + " --tunnel"))
}
cmd.Printf("View the Web UI: %s\n", accessURL)
Expand Down Expand Up @@ -491,9 +474,7 @@ func server() *cobra.Command {
cliflag.BoolVarP(root.Flags(), &inMemoryDatabase, "in-memory", "", "CODER_INMEMORY", false,
"Specifies whether data will be stored in an in-memory database.")
_ = root.Flags().MarkHidden("in-memory")
cliflag.StringVarP(root.Flags(), &postgresURL, "postgres-url", "", "CODER_PG_CONNECTION_URL", "", "URL of a PostgreSQL database to connect to")
cliflag.BoolVarP(root.Flags(), &postgresBuiltin, "postgres-builtin", "", "CODER_PG_BUILTIN", false,
"Start and run a PostgreSQL database for the Coder deployment. This will download PostgreSQL binaries from Maven (https://repo1.maven.org/maven2) and store all data in the config root. Access this database with \"coder server postgres-builtin-url\"")
cliflag.StringVarP(root.Flags(), &postgresURL, "postgres-url", "", "CODER_PG_CONNECTION_URL", "", "The URL of a PostgreSQL database to connect to. If empty, PostgreSQL binaries will be downloaded from Maven (https://repo1.maven.org/maven2) and store all data in the config root. Access the built-in database with \"coder server postgres-builtin-url\"")
cliflag.Uint8VarP(root.Flags(), &provisionerDaemonCount, "provisioner-daemons", "", "CODER_PROVISIONER_DAEMONS", 3, "The amount of provisioner daemons to create on start.")
cliflag.StringVarP(root.Flags(), &oauth2GithubClientID, "oauth2-github-client-id", "", "CODER_OAUTH2_GITHUB_CLIENT_ID", "",
"Specifies a client ID to use for oauth2 with GitHub.")
Expand Down Expand Up @@ -831,7 +812,7 @@ func startBuiltinPostgres(ctx context.Context, cfg config.Root, logger slog.Logg
)
err = ep.Start()
if err != nil {
return "", nil, xerrors.Errorf("Failed to start built-in PostgreSQL: %w", err)
return "", nil, xerrors.Errorf("Failed to start built-in PostgreSQL. Optionally, specify an external deployment with `--postgres-url`: %w", err)
}
return connectionURL, ep.Stop, nil
}
4 changes: 4 additions & 0 deletions cmd/coder/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,18 @@ package main
import (
"errors"
"fmt"
"math/rand"
"os"
"time"
_ "time/tzdata"

"github.com/coder/coder/cli"
"github.com/coder/coder/cli/cliui"
)

func main() {
rand.Seed(time.Now().UnixMicro())

cmd, err := cli.Root().ExecuteC()
if err != nil {
if errors.Is(err, cliui.Canceled) {
Expand Down
6 changes: 2 additions & 4 deletions coder.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
# Run "coder server --help" for flag information.
CODER_ACCESS_URL=
CODER_ADDRESS=
# Automatically install and run PostgreSQL by setting this to "true"
CODER_PG_BUILTIN=
CODER_PG_CONNECTION_URL=
CODER_TLS_CERT_FILE=
CODER_TLS_ENABLE=
CODER_TLS_KEY_FILE=
# Generate a unique *.try.coder.app access URL by setting this to "true"
CODER_TUNNEL=
# Generate a unique *.try.coder.app access URL
CODER_TUNNEL=false
3 changes: 0 additions & 3 deletions coderd/provisionerdaemons.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"errors"
"fmt"
"io"
"math/rand"
"net/http"
"net/url"
"reflect"
Expand Down Expand Up @@ -63,8 +62,6 @@ func (api *API) ListenProvisionerDaemon(ctx context.Context) (client proto.DRPCP
}
}()

// Required for randomly generated names to not conflict!
rand.Seed(time.Now().UnixMicro())
name := namesgenerator.GetRandomName(1)
daemon, err := api.Database.InsertProvisionerDaemon(ctx, database.InsertProvisionerDaemonParams{
ID: uuid.New(),
Expand Down
4 changes: 2 additions & 2 deletions docs/install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,8 @@ We publish self-contained .zip and .tar.gz archives in [GitHub releases](https:/
1. Start a Coder server

```sh
# Automatically sets up PostgreSQL and an external access URL on *.try.coder.app
coder server --postgres-builtin --tunnel
# Automatically sets up an external access URL on *.try.coder.app
coder server --tunnel

# Requires a PostgreSQL instance and external access URL
coder server --postgres-url <url> --access-url <url>
Expand Down
4 changes: 3 additions & 1 deletion install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,12 @@ $1 package has been installed.

To run Coder as a system service:

# Setup PostgreSQL and an external access URL
# Set up an external access URL or enable CODER_TUNNEL
sudo vim /etc/coder.d/coder.env
# Use systemd to start Coder now and on reboot
sudo systemctl enable --now coder
# View the logs to ensure a successful start
journalctl -u coder.service -b

EOF
}
Expand Down
3 changes: 2 additions & 1 deletion preinstall.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ set -eu

USER="coder"

if ! id -u $USER > /dev/null 2>&1; then
# Add a Coder user to run as in systemd.
if ! id -u $USER >/dev/null 2>&1; then
useradd \
--system \
--user-group \
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.