Skip to content

Use zstd compression for layer cache #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 13, 2023
Merged

Use zstd compression for layer cache #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions envbuilder.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,11 @@ func Run(ctx context.Context, options Options) error {
CacheRunLayers: true,
CacheCopyLayers: true,
CompressedCaching: true,
Compression: config.ZStd,
// Maps to "default" level, ~100-300 MB/sec according to
// benchmarks in klauspost/compress README
// https://github.com/klauspost/compress/blob/67a538e2b4df11f8ec7139388838a13bce84b5d5/zstd/encoder_options.go#L188
CompressionLevel: 3,
CacheOptions: config.CacheOptions{
// Cache for a week by default!
CacheTTL: time.Hour * 24 * 7,
Expand Down
63 changes: 32 additions & 31 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.20

// There are a few options we need added to Kaniko!
// See: https://github.com/GoogleContainerTools/kaniko/compare/main...coder:kaniko:main
replace github.com/GoogleContainerTools/kaniko => github.com/coder/kaniko v0.0.0-20230630144655-c825422ea1d2
replace github.com/GoogleContainerTools/kaniko => github.com/coder/kaniko v0.0.0-20230712162612-efb9a14afa73

// Required to import the codersdk!
replace tailscale.com => github.com/coder/tailscale v0.0.0-20230522123520-74712221d00f
Expand All @@ -14,16 +14,17 @@ require (
github.com/GoogleContainerTools/kaniko v1.9.2
github.com/breml/rootcerts v0.2.10
github.com/coder/coder v0.24.2-0.20230630184129-6015319e9d30
github.com/containerd/containerd v1.7.0
github.com/containerd/containerd v1.7.2
github.com/distribution/distribution/v3 v3.0.0-20230629214736-bac7f02e02a1
github.com/docker/cli v23.0.1+incompatible
github.com/docker/docker v23.0.3+incompatible
github.com/docker/cli v23.0.5+incompatible
github.com/docker/docker v23.0.5+incompatible
github.com/fatih/color v1.15.0
github.com/go-git/go-billy/v5 v5.4.1
github.com/go-git/go-git/v5 v5.6.1
github.com/google/go-containerregistry v0.14.0
github.com/go-git/go-git/v5 v5.7.0
github.com/google/go-containerregistry v0.15.2
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
github.com/mattn/go-isatty v0.0.19
github.com/sirupsen/logrus v1.9.2
github.com/sirupsen/logrus v1.9.3
github.com/spf13/cobra v1.7.0
github.com/stretchr/testify v1.8.4
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2
Expand All @@ -46,7 +47,7 @@ require (
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/OneOfOne/xxhash v1.2.8 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 // indirect
github.com/ProtonMail/go-crypto v0.0.0-20230518184743-7afd39499903 // indirect
github.com/acomagu/bufpipe v1.0.4 // indirect
github.com/agext/levenshtein v1.2.3 // indirect
github.com/agnivade/levenshtein v1.1.1 // indirect
Expand All @@ -55,20 +56,21 @@ require (
github.com/ammario/tlru v0.3.0 // indirect
github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.17.3 // indirect
github.com/aws/aws-sdk-go-v2/config v1.15.5 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.12.0 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.27 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.11 // indirect
github.com/aws/aws-sdk-go-v2/service/ecr v1.15.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.12.0 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.11.4 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.16.4 // indirect
github.com/aws/aws-sdk-go-v2 v1.18.1 // indirect
github.com/aws/aws-sdk-go-v2/config v1.18.27 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.13.26 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.28 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.35 // indirect
github.com/aws/aws-sdk-go-v2/service/ecr v1.18.10 // indirect
github.com/aws/aws-sdk-go-v2/service/ecrpublic v1.16.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.28 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.12.12 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.12 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.19.2 // indirect
github.com/aws/smithy-go v1.13.5 // indirect
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20220228164355-396b2034c795 // indirect
github.com/awslabs/amazon-ecr-credential-helper/ecr-login v0.0.0-20230522190001-adf1bafd791a // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bytedance/sonic v1.9.2 // indirect
github.com/cenkalti/backoff/v4 v4.2.0 // indirect
Expand All @@ -79,7 +81,7 @@ require (
github.com/coder/retry v1.4.0 // indirect
github.com/coder/terraform-provider-coder v0.9.0 // indirect
github.com/containerd/cgroups v1.1.0 // indirect
github.com/containerd/continuity v0.3.0 // indirect
github.com/containerd/continuity v0.4.1 // indirect
github.com/containerd/fifo v1.1.0 // indirect
github.com/containerd/stargz-snapshotter/estargz v0.14.3 // indirect
github.com/containerd/typeurl v1.0.2 // indirect
Expand All @@ -101,7 +103,7 @@ require (
github.com/fxamacker/cbor/v2 v2.4.0 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-chi/chi/v5 v5.0.8 // indirect
github.com/go-git/gcfg v1.5.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-jose/go-jose/v3 v3.0.0 // indirect
github.com/go-logr/logr v1.2.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
Expand Down Expand Up @@ -139,7 +141,7 @@ require (
github.com/hashicorp/yamux v0.1.1 // indirect
github.com/hdevalence/ed25519consensus v0.0.0-20220222234857-c00d1f31bab3 // indirect
github.com/illarion/gonotify v1.0.1 // indirect
github.com/imdario/mergo v0.3.13 // indirect
github.com/imdario/mergo v0.3.15 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/insomniacslk/dhcp v0.0.0-20221215072855-de60144f33f8 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
Expand All @@ -148,7 +150,6 @@ require (
github.com/josharian/native v1.1.1-0.20230202152459-5c7d0dd6ab86 // indirect
github.com/jsimonetti/rtnetlink v1.1.2-0.20220408201609-d380b505068b // indirect
github.com/karrick/godirwalk v1.16.1 // indirect
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.16.5 // indirect
github.com/kortschak/wol v0.0.0-20200729010619-da482cc4850a // indirect
Expand All @@ -167,7 +168,7 @@ require (
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/buildkit v0.11.4 // indirect
github.com/moby/buildkit v0.11.6 // indirect
github.com/moby/locker v1.0.1 // indirect
github.com/moby/patternmatcher v0.5.0 // indirect
github.com/moby/swarmkit/v2 v2.0.0-20230315203717-e28e8ba9bc83 // indirect
Expand All @@ -184,7 +185,7 @@ require (
github.com/opencontainers/runc v1.1.5 // indirect
github.com/opencontainers/runtime-spec v1.1.0-rc.1 // indirect
github.com/opencontainers/selinux v1.11.0 // indirect
github.com/otiai10/copy v1.7.0 // indirect
github.com/otiai10/copy v1.12.0 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
Expand All @@ -195,7 +196,7 @@ require (
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0 // indirect
github.com/rootless-containers/rootlesskit v1.1.0 // indirect
github.com/sergi/go-diff v1.3.1 // indirect
github.com/skeema/knownhosts v1.1.0 // indirect
github.com/skeema/knownhosts v1.1.1 // indirect
github.com/spf13/afero v1.9.5 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tabbed/pqtype v0.1.1 // indirect
Expand Down Expand Up @@ -242,7 +243,7 @@ require (
golang.org/x/net v0.11.0 // indirect
golang.org/x/oauth2 v0.9.0 // indirect
golang.org/x/sync v0.3.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/sys v0.10.0 // indirect
golang.org/x/term v0.9.0 // indirect
golang.org/x/text v0.10.0 // indirect
golang.org/x/time v0.3.0 // indirect
Expand All @@ -253,8 +254,8 @@ require (
google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/grpc v1.56.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
google.golang.org/grpc v1.56.1 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading