feat: Compress and extract slim binaries with zstd #2533

mafredri · 2022-06-20T17:31:21Z

This PR adds support for compressing coder slim binaries. The binaries will be embedded as site/bin/coder.tar.zst and unpacked on coder startup.

Cases:

The bin/coder.tar.zst archive is present -> extracted to $CODER_CACHE_DIR/site/bin and served via http.Dir
There are files other than GITKEEP in embedded fs under bin/ -> serve from embedd fs
There is no bin/ or only bin/GITKEEP in embedd fs -> serve from empty $CODER_CACHE_DIR/site/bin (allows manual placement)

feat: Add support for zstd compression of slim binaries
feat: Add support for extract and serve of zstd bins

Fixes #2202

Changes here may also affect #1547 (i.e. 404 for /bin won't return index.html).

TODO:

Only use compression level 22 for releases, level 6 or 7 should suffice for quick builds
~~Only extract archive once (currently extracted on each startup)~~ (This will be a separate PR)

Notes

Decompression is sufficiently fast at around 0.6 - 1 second and will not delay startup significantly.

The following results show that compression after level 6 or 7 show diminishing returns until "ultra" levels of 20+.

for threads in T1 T0; do
	for level in 3 5 7 9 12 15 16 17 18 19 20 21 22; do
		print "\nZstd ${level} ${threads}:"
		time zstd --ultra --long -$level -$threads -o coder-zstd-level-$level-$threads.tar.zst coder.tar
	done
done

Zstd 3 T1:
coder.tar            : 27.92%   (   271 MiB =>   75.6 MiB, coder-zstd-level-3-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  3.15s user 0.34s system 109% cpu 3.192 total

Zstd 5 T1:
coder.tar            : 27.16%   (   271 MiB =>   73.6 MiB, coder-zstd-level-5-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  4.96s user 0.31s system 106% cpu 4.946 total

Zstd 7 T1:
coder.tar            : 26.16%   (   271 MiB =>   70.9 MiB, coder-zstd-level-7-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  7.12s user 0.33s system 104% cpu 7.105 total

Zstd 9 T1:
coder.tar            : 25.66%   (   271 MiB =>   69.5 MiB, coder-zstd-level-9-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  9.25s user 0.28s system 103% cpu 9.233 total

Zstd 12 T1:
coder.tar            : 25.15%   (   271 MiB =>   68.1 MiB, coder-zstd-level-12-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  20.56s user 0.45s system 101% cpu 20.706 total

Zstd 15 T1:
coder.tar            : 24.91%   (   271 MiB =>   67.5 MiB, coder-zstd-level-15-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  51.20s user 0.47s system 100% cpu 51.433 total

Zstd 16 T1:
coder.tar            : 24.08%   (   271 MiB =>   65.2 MiB, coder-zstd-level-16-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  77.86s user 0.30s system 99% cpu 1:18.18 total

Zstd 17 T1:
coder.tar            : 23.59%   (   271 MiB =>   63.9 MiB, coder-zstd-level-17-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  98.41s user 0.34s system 99% cpu 1:38.88 total

Zstd 18 T1:
coder.tar            : 22.52%   (   271 MiB =>   61.0 MiB, coder-zstd-level-18-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  121.11s user 0.35s system 100% cpu 2:01.44 total

Zstd 19 T1:
coder.tar            : 22.22%   (   271 MiB =>   60.2 MiB, coder-zstd-level-19-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  144.87s user 0.46s system 100% cpu 2:25.16 total

Zstd 20 T1:
coder.tar            : 19.30%   (   271 MiB =>   52.3 MiB, coder-zstd-level-20-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  144.46s user 0.42s system 100% cpu 2:24.55 total

Zstd 21 T1:
coder.tar            : 18.09%   (   271 MiB =>   49.0 MiB, coder-zstd-level-21-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  161.54s user 0.51s system 99% cpu 2:42.07 total

Zstd 22 T1:
coder.tar            : 17.80%   (   271 MiB =>   48.2 MiB, coder-zstd-level-22-T1.tar.zst)
zstd --ultra --long -$level -$threads -o  coder.tar  144.54s user 0.89s system 100% cpu 2:25.36 total

scripts/build_go_slim.sh

deansheather · 2022-06-20T17:36:42Z

scripts/build_go_slim.sh

+if [[ -n $compress ]]; then
+	dependencies tar zstd
+
+	if [[ ! $compress == [0-9]* ]] || ((compress > 22)) || ((compress < 1)); then


Suggested change

if [[ ! $compress == [0-9]* ]] || ((compress > 22)) || ((compress < 1)); then

if [[ "$compress" != [0-9]* ]] || [ "$compress" -gt 22 ] || [ "$compress" -lt 1 ]; then

Just curious why we'd want to avoid (( ))? Neither handles non-numbers gracefully (but we've verified that with the glob match).

Consistency, we use lt and gt in other scripts

Ok, fair enough. I'd like for us to drop the use of [ though. [[ is superior and also supports -gt and -lt. For instance, with [[ there is no need to quote the right-hand variable to avoid errors when it's empty.

Feel free to change it across all scripts 👍

I won't do it in this PR to keep the noise down, but I can do a follow-up tomorrow.

site/site.go

deansheather · 2022-06-20T17:54:56Z

cli/server.go

@@ -254,6 +254,7 @@ func server() *cobra.Command {
 				Logger:               logger.Named("coderd"),
 				Database:             databasefake.New(),
 				Pubsub:               database.NewPubsubInMemory(),
+				CacheDir:             cacheDir,


cacheDir defaults to /tmp/ on my system. We should probably change that to not use /tmp at all and instead use ~/.cache so we're not dumping 300MB into RAM on startup

Indeed, I've hinted at this in #2200 but we should create a separate issue for it.

Co-authored-by: Dean Sheather <dean@deansheather.com>

mafredri

Oh, looks like GitHub left a bunch of my comments pending 😅

mafredri · 2022-06-20T18:05:34Z

scripts/build_go_slim.sh

+if [[ -n $compress ]]; then
+	dependencies tar zstd
+
+	if [[ ! $compress == [0-9]* ]] || ((compress > 22)) || ((compress < 1)); then


Just curious why we'd want to avoid (( ))? Neither handles non-numbers gracefully (but we've verified that with the glob match).

scripts/build_go_slim.sh

site/site.go

mafredri · 2022-06-20T18:18:31Z

cli/server.go

@@ -254,6 +254,7 @@ func server() *cobra.Command {
 				Logger:               logger.Named("coderd"),
 				Database:             databasefake.New(),
 				Pubsub:               database.NewPubsubInMemory(),
+				CacheDir:             cacheDir,


Indeed, I've hinted at this in #2200 but we should create a separate issue for it.

mafredri · 2022-06-20T18:21:11Z

cli/server.go

@@ -254,6 +254,7 @@ func server() *cobra.Command {
 				Logger:               logger.Named("coderd"),
 				Database:             databasefake.New(),
 				Pubsub:               database.NewPubsubInMemory(),
+				CacheDir:             cacheDir,


Makefile

scripts/build_go_slim.sh

mafredri added 2 commits June 20, 2022 20:13

feat: Add support for zstd compression of slim binaries

b262c38

feat: Add support for extract and serve of zstd bins

fb24c02

mafredri requested a review from a team as a code owner June 20, 2022 17:31

mafredri self-assigned this Jun 20, 2022

mafredri requested review from deansheather and a team June 20, 2022 17:31

mafredri added 2 commits June 20, 2022 20:33

chore: Remove unused function

6c7e1f7

Always read body in test

b427108

deansheather reviewed Jun 20, 2022

View reviewed changes

mafredri added 2 commits June 20, 2022 20:59

chore: Log what we are doing

3c68823

chore: Fix PR comments

0c6e367

mafredri mentioned this pull request Jun 20, 2022

The coder server cache dir defaults to /tmp, should be ~/.cache #2534

Closed

mafredri and others added 3 commits June 20, 2022 21:25

chore: Fix PR comments

de8c01f

chore: Fix PR comments

528c34c

Apply suggestions from code review

db959f3

Co-authored-by: Dean Sheather <dean@deansheather.com>

mafredri commented Jun 20, 2022

View reviewed changes

chore: Use pushd/popd vs subshell

bd322c7

deansheather reviewed Jun 20, 2022

View reviewed changes

Makefile Outdated Show resolved Hide resolved

mafredri added 4 commits June 20, 2022 21:38

chore: Switch use of (( to [[

5c070d5

fix: Use compress in github actions

c3e4613

chore: Update build_go_slim.sh usage

d715944

fix: Use compression level 6 for Makefile builds

2db7961

deansheather approved these changes Jun 20, 2022

View reviewed changes

scripts/build_go_slim.sh Show resolved Hide resolved

fix: Explain compress option

e65a528

johnstcn approved these changes Jun 21, 2022

View reviewed changes

fix: Add zstd --force to prevent prompts

650558e

mafredri mentioned this pull request Jun 21, 2022

chore: Prefer [[ over [ in bash build scripts #2543

Merged

mafredri merged commit e2785ad into main Jun 21, 2022

mafredri deleted the mafredri/compress-coder-slim-binaries branch June 21, 2022 16:53

mafredri mentioned this pull request Jun 21, 2022

feat: Check decompressed coder-slim binaries via SHA1 #2556

Merged

mafredri mentioned this pull request Jun 23, 2022

clean site/out and enforce make bin #2604

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Compress and extract slim binaries with zstd #2533

feat: Compress and extract slim binaries with zstd #2533

mafredri commented Jun 20, 2022 •

edited

Loading

deansheather Jun 20, 2022

mafredri Jun 20, 2022

deansheather Jun 20, 2022

mafredri Jun 20, 2022

deansheather Jun 20, 2022

mafredri Jun 20, 2022

deansheather Jun 20, 2022

mafredri Jun 20, 2022

mafredri Jun 20, 2022

mafredri left a comment

mafredri Jun 20, 2022

mafredri Jun 20, 2022

mafredri Jun 20, 2022

	if [[ ! $compress == [0-9]* ]] \|\| ((compress > 22)) \|\| ((compress < 1)); then
	if [[ "$compress" != [0-9]* ]] \|\| [ "$compress" -gt 22 ] \|\| [ "$compress" -lt 1 ]; then

feat: Compress and extract slim binaries with zstd #2533

feat: Compress and extract slim binaries with zstd #2533

Conversation

mafredri commented Jun 20, 2022 • edited Loading

TODO:

Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafredri commented Jun 20, 2022 •

edited

Loading