This came up in #2534 (comment) as well, i.e. we currently don't upload "slim" binaries in our releases -- should we?

The slim binaries are still pretty big though, weighing in at 40-50MB. We already strip debug information from them (via -ldflags "-s -w") so reducing the size means eliminating code and/or libraries.

One last thing I can think of is to pack binaries with UPX.

Unfortunately, Windows AARCH64 doesn't seem to be supported yet (upx/upx#502, upx/upx#551):

upx: /home/maf/src/coder/dist/coder-slim_0.8.5-devel+54b8e794_windows_arm64.exe: CantPackException: can't pack new-exe

We could apply this principle to the whole build pipeline.

Original build:

105M coder_0.8.5-devel+54b8e794_linux_amd64*
 47M coder-slim_0.8.5-devel+54b8e794_darwin_amd64*
 47M coder-slim_0.8.5-devel+54b8e794_darwin_arm64*
 44M coder-slim_0.8.5-devel+54b8e794_linux_amd64*
 42M coder-slim_0.8.5-devel+54b8e794_linux_arm64*
 41M coder-slim_0.8.5-devel+54b8e794_linux_armv7*
 43M coder-slim_0.8.5-devel+54b8e794_windows_amd64.exe*
 42M coder-slim_0.8.5-devel+54b8e794_windows_arm64.exe*

UPX only the "fat" binary:

❯ upx --best --lzma coder_0.8.5-devel+54b8e794_linux_amd64
71M coder_0.8.5-devel+54b8e794_linux_amd64*

Everything built with UPX (skipping Windows AARCH64):

 83M coder_0.8.5-devel+54b8e794_linux_amd64*
 12M coder-slim_0.8.5-devel+54b8e794_darwin_amd64*
 16M coder-slim_0.8.5-devel+54b8e794_darwin_arm64*
 12M coder-slim_0.8.5-devel+54b8e794_linux_amd64*
9.9M coder-slim_0.8.5-devel+54b8e794_linux_arm64*
9.6M coder-slim_0.8.5-devel+54b8e794_linux_armv7*
 12M coder-slim_0.8.5-devel+54b8e794_windows_amd64.exe*

The result is pretty interesting because Zstd does a better job on uncompressed binaries, UPX adds 10MB bloat to the Zstd archive, even with the missing Windows AARCH64 binary.

But then again, in the UPX-ified slim case, we would save ~30MB/binary in the coder cache directory, where slim binaries are extracted to.

Forgot to mention in my previous response, but regarding:

Also, the smaller we make the binary, the more quickly workspaces start.

Interestingly, the agent binary is already plenty small due to the bootstrap script doing curl --compressed which will result in either gzip of deflate compression being used.

Surprisingly, on my local network the result is not at all what I would expect:

❯ time curl --compressed "http://coder.home.my.domain:7080/bin/coder-linux-amd64" --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 16.5M    0 16.5M    0     0  9544k      0 --:--:--  0:00:01 --:--:-- 9579k
curl --compressed "http://coder.home.my.domain:7080/bin/coder-linux-amd64"    0.43s user 0.11s system 30% cpu 1.790 total

❯ time curl "http://coder.home.my.domain:7080/bin/coder-linux-amd64" --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.4M  100 43.4M    0     0   107M      0 --:--:-- --:--:-- --:--:--  109M
curl "http://coder.home.my.domain:7080/bin/coder-linux-amd64" --output /dev/null  0.02s user 0.07s system 20% cpu 0.414 total

I.e. the overhead from compression makes it slower, even though the size is much smaller (16.5M @ 0.43s vs 43.4M @ 0.02s).

But this would be negligible on a slower network and UPX packing will only save us ~4M in transfer size.

Out of interest, did one more test to break out the agent out of the cli package (i.e. move the code to cmd/agent.

This brought the binary size down from 43.4M to 26M. Gzip compressed 9.7M and UPX compressed 7.2M (vs 12M before).

I do like the simplicity of our all-in-one binary, but for size reduction, it's an avenue we could consider pursuing (if we want to prioritize workspace bootstrap speed).

It makes sense to me to ship a stripped agent binary and, in general, watch the size of that binary due to its effect on workspace start times.

I've done some more testing around the use of UPX, and there's an unfortunate side-effect: it adds around 0.5 - 1.5s extra start-up time (e.g. coder --help) which is pretty unacceptable for commands that are run repeatedly. The only place I can see it being acceptable is in a separate agent binary. But even then, gzip compression seems good enough to cover the cost of transfer.

Even in it's lowest setting (upx -1) it's pretty bad, and -1 --lzma (better compression) even worse:

./coder_0.8.6-devel+cee0131d_linux_amd64 --help  0.14s user 0.02s system 111% cpu 0.148 total
./coder_0.8.6-devel+cee0131d_linux_amd64_UPX-1 --help  0.74s user 0.04s system 103% cpu 0.750 total
./coder_0.8.6-devel+cee0131d_linux_amd64_UPX-1--lzma --help  8.69s user 0.06s system 99% cpu 8.748 total

With regards to the original issue: There are problems with separating the agent and coder cli:

• It will prevent us from providing coder inside the workspaces
• It will break Provide a coder upgrade command that pulls coder directly from the server #2744

Or, if we don't want to break the above use-cases, we'll need to ship both coder and agent, inflating the total binary size. (The coder binary will not shrink by removing the agent part.)

As I see it, the best thing we can do without creating a worse experience for most users, is to provide an easy way to install and setup coder with only "slim" binaries. This would mean:

1. Adding slim binaries to our releases
2. Adding an explicit --agent-binaries-dir/CODER_AGENT_BINARIES_DIR to coder server for serving binaries (ease of discovery vs placing in cache dir)
   1. If we want to take extra consideration into size-on-filesystem, we could allow serving binaries from an archive
3. Allow serving the underlying coder server binary if agentbin/coder-linux-amd64 is missing (assuming coder server is running on linux amd64).

The 2i and 3 are kind of mutually exclusive, if our releases contain coder-slim.tar.zstd for all the platforms, then 3 won't help. However, in the extracted case (2), allows saving 40-50MB.

One more thing we can do is to not have the "fat" binary decompress "slim" binaries to the filesystem. This would decrease filesystem storage by ~300MB whilst the "fat" coder binary staying the same size (80+MB). The files would need to be decompressed on-the-fly, though, but Zstd is quite fast (I haven't benchmarked recently, but we're talking sub-second), so we're mostly talking a little bit of extra CPU usage. Maybe a --no-decompress-binaries option.

There's quite a lot of musings here now, so I'll try to distill this all down into a proposal for how to take this forward, but in the meantime, input is welcome.

I think we shouldn't strip binaries of debug information or we will regret it when customers run into problems.

Related to #3188

@deansheather I agree, and that hasn't been suggested here. Currently we already use -ldflags "-s -w" in the build process (takes "slim" from 60MB -> 40MB, ish), and AFAICT, that shouldn't affect any logging, tracing, or use of pprof. But it would hinder the use of e.g. gdb.

For reference:

  -s    disable symbol table
  -w    disable DWARF generation

Good point to reference #3188 👍🏻. It would be interesting to explore how much, if any, savings/binary can be had by separating the server.

I think we should close this for now. It seems like a non-trivial level of effort, and awaiting more feedback would be nice.

@kylecarbs One simple step we could take is to extend the build pipe-line to publish the following additional assets:

• coder_slim_* binaries
• coder_slim.tar.zst (for slim servers to serve agents)

I.e. split out the big binary. This would ~half the size of the binary (and hopefully help with the original issue).

We don't necessarily have to do it for every platform/binary either. We could just always do it for apk/dep/rpm. There's really no need to bundle them in those cases when it's already "one file".

The downsides are that we would either duplicate all our release binaries or we'd introduce inconsistency where some releases use the split, whereas others don't.