fix: Re-enable parallel run of Postgres-backed tests #119
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bryphe-coder
commented
Feb 1, 2022
Codecov Report
@@ Coverage Diff @@
## main #119 +/- ##
==========================================
- Coverage 74.72% 72.14% -2.59%
==========================================
Files 53 91 +38
Lines 550 3773 +3223
Branches 59 59
==========================================
+ Hits 411 2722 +2311
- Misses 127 831 +704
- Partials 12 220 +208
Continue to review full report at Codecov.
|
kylecarbs
approved these changes
Feb 1, 2022
kylecarbs
approved these changes
Feb 1, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@kylecarbs and I were debugging a gnarly postgres issue over the weekend, and unfortunately it looks like it is still coming up occassionally: https://github.com/coder/coder/runs/5014420662?check_suite_focus=true#step:8:35 - so thought this might be a good testing Monday task.
Intermittently, the test would fail with something like a
401- invalid e-mail, or a409- initial user already created. This was quite surprising, because the tests are designed to spin up their own, isolated database.We tried a few things to debug this...
Attempt 1: Log out the generated port numbers when running the docker image.
Based on the errors, it seemed like one test must be connecting to another test's database - that would explain why we'd get these conflicts! However, logging out the port number that came from docker always gave a unique number... and we couldn't find evidence of one database connecting to another.
Attempt 2: Store the database in unique, temporary folder.
@kylecarbs and I found that the there was a volume for the postgres data... so @kylecarbs implemented mounting the volume to a unique, per-test temporary folder in #89
It sounded really promising... but unfortunately we hit the issue again!
Attempt 3... this PR
After we hit the failure again, we noticed in the
docker pslogs something quite strange:When the docker image is run - it creates two port bindings, an IPv4 and an IPv6 one. These should be the same - but surprisingly, they can sometimes be different. It isn't deterministic, and seems to be more common when there are multiple containers running. Importantly, they can overlap as in the above image.
Turns out, it seems this is a docker bug: moby/moby#42442 - which may be fixed in newer versions.
To work around this bug, we have to manipulate the port bindings (like you would with
-p) at the command line. We can do this withdocker/dockertest, but it means we have to get a free port ahead of time to know which port to map.With that fix in - the
docker psis a little more sane:...and hopefully means we can safely run the containers in parallel again.