Investigating failures like this: https://github.com/coder/coder/runs/5043435263?check_suite_focus=true#step:7:32

where it looks like the job is completed, but the completion condition is never satisfied

Runs

• Run 1: peer failure
• Run 2: peer failure https://github.com/coder/coder/runs/5044293751?check_suite_focus=true#step:7:63
• Run 3: a different data race: https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:83

Issue 1: Data race in p.acquiredJobDone context (provisioners.d)

Failure trace: https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:84

There is a race in the p.acquiredJobDone chan - in particular, there can be a case where we're waiting on the channel to finish (in close) with <-p.acquiredJobDone, but in parallel, an acquireJob could've been started, which would create a new channel for p.acquiredJobDone.

The fix I tried was to also grab the acquiredJobMutex in the Close function. This, at first, caused a deadlock - because there was another case where the mutexes could be grabbed in reverse order (acquiredJobMutex -> then closeMutex). That other place, though, was storing a bool in an atomic, so actually didn't need the mutex guard.

Attempted fix here: 42ce721

Still hit a related race: https://github.com/coder/coder/runs/5044320845?check_suite_focus=true#step:7:84

So tried a second fix here: 84dd68a

The second fix didn't work, trying to switch from chan -> wait group: a8725cd

• Run 1: ✅
• Run 2: peer failure