feat: Add Tailscale networking #3505

kylecarbs · 2022-08-14T20:42:03Z

Contributes to #2779. This adds Tailscale networking in addition to WebRTC. It is disabled by default, clients must opt-in (and it's hidden). The rollout plan is:

Dogfood Tailscale networking for ~1 week.
If all is stable and well, release with it as the default.
Remove WebRTC networking the subsequent week.

The delta should be -3,000 lines of code 😎... not to mention so so so many bugs fixed.

Todo

Add a global toggle to enable Tailscale networking. This should also impact clients looking to create connections.
Refactor the workspace agent retry loop into an agent-client creator.
Make agent.TailscaleConn.Ping() work!
Rename HTTPForTests in the Tailscale fork to ForceHTTP.
The DERPMap should be fetched on agent reconnect.

Might do

Display connected regions with latencies in the UI.
Allow DERP URLs to be specified. This allows Tailscale's hosted DERP servers to be used with Coder, just like Headscale.

This makes for a simpler setup, and reduces the likelihood a user runs into a strange issue.

coadler · 2022-08-24T16:45:40Z

@bpmct I think this is related to websockets being broken for the dev tunnel. Fixing this morning!

kylecarbs · 2022-08-30T03:15:26Z

@bpmct that issue you encountered with WebSockets has been fixed!

bpmct · 2022-08-30T14:58:34Z

Did some tests and confirmed #3316 and the terminal hanging is resolved, the tunnel also works. 🎉 The first time I tried it, I did have to manually reload the pop-up window in order to get things to load, however. I have not been able to reproduce sense. Maybe it was a one-off error?

final_630e1f4ccd265a00839746cf_4775.mp4

I also see somewhat surprising logs, although things seem stable during code-server usage

coder server logs

2022-08-30 14:45:04.950 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: send, starting discovery for [0UbXg] (d:ecee900b5554b13b)
2022-08-30 14:45:04.950 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], 172.31.23.187:44809) sent ping tx=d5440625ea70
2022-08-30 14:45:04.950 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], derp-999) sent call-me-maybe
2022-08-30 14:45:07.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong e1eb51c806ff from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:09.950 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong d5440625ea70 from 172.31.23.187:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:10.101 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v2] [0UbXg] - Handshake did not complete after 20 attempts, giving up
2022-08-30 14:45:10.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong 876e58d1e2fe from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:11.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: send, starting discovery for [0UbXg] (d:ecee900b5554b13b)
2022-08-30 14:45:11.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], derp-999) sent call-me-maybe
2022-08-30 14:45:11.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], 172.31.23.187:44809) sent ping tx=e7c2bc468d13
2022-08-30 14:45:13.417 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong b2b58aa991fd from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:16.418 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong 0a35d2c4b98a from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:16.418 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong e7c2bc468d13 from 172.31.23.187:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:17.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: send, starting discovery for [0UbXg] (d:ecee900b5554b13b)
2022-08-30 14:45:17.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], derp-999) sent call-me-maybe
2022-08-30 14:45:17.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], 172.31.23.187:44809) sent ping tx=ff3c81b956bf
2022-08-30 14:45:19.362 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: starting endpoint update (periodic)
2022-08-30 14:45:19.380 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    netcheck: [v1] report: udp=true v6=true mapvarydest=false hair=true portmap= v4a=45.77.76.151:54499 v6a=[2001:19f0:5:278d:5400:3ff:feec:cd12]:54878 derp=2 derpdist=2v4:2ms,2v6:4ms,999v4:13ms
2022-08-30 14:45:19.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong d4889e9806c3 from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:20.006 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:4e92f14833ab99d7<-d:1911ac45e861ab0f ([XIv/+], derp-999)  got call-me-maybe, 2 endpoints
2022-08-30 14:45:20.007 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:4e92f14833ab99d7->d:1911ac45e861ab0f ([XIv/+], 172.31.31.123:50234) sent ping tx=d898f99f3b8f
2022-08-30 14:45:20.007 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:4e92f14833ab99d7->d:1911ac45e861ab0f ([XIv/+], 34.209.81.24:50234) sent ping tx=780aa0477bd9
2022-08-30 14:45:20.077 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:4e92f14833ab99d7<-d:1911ac45e861ab0f ([XIv/+], 34.209.81.24:50234)  got pong tx=780aa0477bd9 latency=70ms pong.src=45.77.76.151:54499
2022-08-30 14:45:22.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong ff3c81b956bf from 172.31.23.187:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:22.419 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong ffd1a7a6e35a from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:23.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: send, starting discovery for [0UbXg] (d:ecee900b5554b13b)
2022-08-30 14:45:23.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: want call-me-maybe but endpoints stale; restunning
2022-08-30 14:45:23.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: starting endpoint update (refresh-for-peering)
2022-08-30 14:45:23.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], 172.31.23.187:44809) sent ping tx=63556f8eb5b6
2022-08-30 14:45:23.439 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    netcheck: [v1] report: udp=true v6=true mapvarydest=false hair=false portmap= v4a=45.77.76.151:45444 v6a=[2001:19f0:5:278d:5400:3ff:feec:cd12]:57772 derp=2 derpdist=2v4:4ms,2v6:4ms,999v4:13ms
2022-08-30 14:45:23.446 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: STUN done; sending call-me-maybe to d:ecee900b5554b13b [0UbXg]
2022-08-30 14:45:23.447 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], derp-999) sent call-me-maybe
2022-08-30 14:45:25.269 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v2] [0UbXg] - Sending handshake initiation
2022-08-30 14:45:25.420 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong 502e159d8b35 from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:26.662 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v2] [XIv/+] - Sending keepalive packet
2022-08-30 14:45:28.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong 1b16dee1c80e from 54.202.2.137:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:28.421 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: timeout waiting for pong 63556f8eb5b6 from 172.31.23.187:44809 ([0UbXg], d:ecee900b5554b13b)
2022-08-30 14:45:28.779 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    magicsock: closing connection to derp-999 (idle), age 1m15s
2022-08-30 14:45:28.779 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    magicsock: 1 active derp conns: derp-2=cr2m0s,wr2m0s
2022-08-30 14:45:29.422 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: send, starting discovery for [0UbXg] (d:ecee900b5554b13b)
2022-08-30 14:45:29.422 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], derp-999) sent call-me-maybe
2022-08-30 14:45:29.422 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v1] magicsock: disco: d:9100959705000a6e->d:ecee900b5554b13b ([0UbXg], 172.31.23.187:44809) sent ping tx=0efa516a0a64
2022-08-30 14:45:30.562 [DEBUG] (coderd.tailnet.wgengine)       <./tailnet/conn.go:515> Logger.func1    [v2] [0UbXg] - Handshake did not complete after 5 seconds, retrying (try 2)

kylecarbs and others added 18 commits June 27, 2022 18:47

fix: Add coder user to docker group on installation

789e878

This makes for a simpler setup, and reduces the likelihood a user runs into a strange issue.

Add wgnet

c5f91a8

Add ping

844071d

Add listening

36655f0

Finish refactor to make this work

38f4004

Add interface for swapping

5ba96b5

Fix conncache with interface

7fe91a2

Merge branch 'main' into tailnet

66f33ad

Merge branch 'main' into tailnet

ea55740

Merge branch 'main' into tailnet

c3a6cd8

Merge branch 'main' into tailnet

e0e5f8f

chore: update gvisor

4348b88

fix tailscale types

47af2b7

linting

4f34245

more linting

d36f61d

Add coordinator

01c52c7

Add coordinator tests

94e90a3

Merge branch 'main' into tailnet

5cf709c

kylecarbs self-assigned this Aug 14, 2022

kylecarbs added 3 commits August 15, 2022 17:54

Fix coordination

cd88dca

It compiles!

f4e5887

Move all connection negotiation in-memory

93f965a

kylecarbs force-pushed the tailnet branch from e79d69b to 93f965a Compare August 21, 2022 02:22

kylecarbs added 3 commits August 21, 2022 02:48

Migrate coordinator to use net.conn

d03849e

Add closed func

5e7197c

Fix close listener func

a117d75

kylecarbs force-pushed the tailnet branch from 969607f to a117d75 Compare August 21, 2022 06:04

kylecarbs added 3 commits August 21, 2022 06:42

Make reconnecting PTY work

cb89378

Fix reconnecting PTY

85be897

Update CI to Go 1.19

4356b00

ammario mentioned this pull request Aug 24, 2022

Surprising terminal typing latency #3657

Closed

Merge branch 'main' into tailnet

2f899f2

kylecarbs force-pushed the tailnet branch from 6c91111 to 2f899f2 Compare August 30, 2022 03:15

kylecarbs added 2 commits August 30, 2022 03:27

Fix connection RBAC

9458070

Fix migration numbers

2382dc2

kylecarbs added 3 commits August 30, 2022 21:55

Fix forwarding TCP to a local port

04367d9

Implement ping for tailnet

db65752

Rename to ForceHTTP

3f48e18

kylecarbs force-pushed the tailnet branch from b8a206c to 3f48e18 Compare August 31, 2022 01:26

kylecarbs added 7 commits August 31, 2022 02:51

Add external derpmapping

94658f5

Expose DERP region names to the API

fd29e7c

Add global option to enable Tailscale networking for web

77dda4a

Mark DERP flags hidden while testing

b9f2b77

Merge branch 'main' into tailnet

5046398

Update DERP map on reconnect

b18b823

Add close func to workspace agents

7b774c9

kylecarbs marked this pull request as ready for review August 31, 2022 20:59

kylecarbs requested a review from a team as a code owner August 31, 2022 20:59

kylecarbs requested review from code-asher and removed request for a team August 31, 2022 20:59

kylecarbs added 3 commits August 31, 2022 23:39

Fix race condition in upstream dependency

b62b46c

Fix feature columns race condition

0606551

Merge branch 'main' into tailnet

3cd8485

coadler approved these changes Sep 1, 2022

View reviewed changes

kylecarbs merged commit 9bd83e5 into main Sep 1, 2022

kylecarbs deleted the tailnet branch September 1, 2022 01:09

bpmct mentioned this pull request Sep 27, 2022

docs: add health checks to docs & examples #4223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add Tailscale networking #3505

feat: Add Tailscale networking #3505

Uh oh!

kylecarbs commented Aug 14, 2022 •

edited by matifali

Loading

Uh oh!

coadler commented Aug 24, 2022

Uh oh!

kylecarbs commented Aug 30, 2022

Uh oh!

bpmct commented Aug 30, 2022 •

edited

Loading

Uh oh!

Uh oh!

feat: Add Tailscale networking #3505

feat: Add Tailscale networking #3505

Uh oh!

Conversation

kylecarbs commented Aug 14, 2022 • edited by matifali Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Todo

Might do

Uh oh!

coadler commented Aug 24, 2022

Uh oh!

kylecarbs commented Aug 30, 2022

Uh oh!

bpmct commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kylecarbs commented Aug 14, 2022 •

edited by matifali

Loading

bpmct commented Aug 30, 2022 •

edited

Loading