ci: Run PostgreSQL with a scratch directory to improve CI durability #89
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When using parallel before, multiple PostgreSQL containers would unintentionally interfere with the other's data. This ensures both containers have separated data, and don't create a volume. 🌮 @bryphe-coder for the idea!
Codecov Report
@@ Coverage Diff @@
## main #89 +/- ##
==========================================
+ Coverage 71.45% 71.71% +0.25%
==========================================
Files 89 89
Lines 3563 3578 +15
Branches 55 55
==========================================
+ Hits 2546 2566 +20
+ Misses 794 790 -4
+ Partials 223 222 -1
Continue to review full report at Codecov.
|
bryphe-coder
approved these changes
Jan 30, 2022
bryphe-coder
added a commit
that referenced
this pull request
Feb 1, 2022
@kylecarbs and I were debugging a gnarly postgres issue over the weekend, and unfortunately it looks like it is still coming up occassionally: https://github.com/coder/coder/runs/5014420662?check_suite_focus=true#step:8:35 - so thought this might be a good testing Monday task. Intermittently, the test would fail with something like a `401` - invalid e-mail, or a `409` - initial user already created. This was quite surprising, because the tests are designed to spin up their own, isolated database. We tried a few things to debug this... ## Attempt 1: Log out the generated port numbers when running the docker image. Based on the errors, it seemed like one test must be connecting to another test's database - that would explain why we'd get these conflicts! However, logging out the port number that came from docker always gave a unique number... and we couldn't find evidence of one database connecting to another. ## Attempt 2: Store the database in unique, temporary folder. @kylecarbs and I found that the there was a [volume](https://github.com/docker-library/postgres/blob/a83005b407ee6d810413500d8a041c957fb10cf0/11/alpine/Dockerfile#L155) for the postgres data... so @kylecarbs implemented mounting the volume to a unique, per-test temporary folder in #89 It sounded really promising... but unfortunately we hit the issue again! ### Attempt 3... this PR After we hit the failure again, we noticed in the `docker ps` logs something quite strange:  When the docker image is run - it creates two port bindings, an IPv4 and an IPv6 one. These _should be the same_ - but surprisingly, they can sometimes be different. It isn't deterministic, and seems to be more common when there are multiple containers running. Importantly, __they can overlap__ as in the above image. Turns out, it seems this is a docker bug: moby/moby#42442 - which may be fixed in newer versions. To work around this bug, we have to manipulate the port bindings (like you would with `-p`) at the command line. We can do this with `docker`/`dockertest`, but it means we have to get a free port ahead of time to know which port to map. With that fix in - the `docker ps` is a little more sane:  ...and hopefully means we can safely run the containers in parallel again.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using parallel before, multiple PostgreSQL containers would
unintentionally interfere with the other's data. This ensures
both containers have separated data, and don't create a volume.
🌮 @bryphe-coder for the idea!