Description
A followup from #14462 where @ethanndickson made some great headway for me on getting Coder agents to be able to provide an additional header.
My configuration is that I have an AWS ALB in front of Coder, and it is configured to allow requests from privileged IPs, and requests that provide an additional header "bypass token". All other connections are forwarded for OIDC authentication to protect Coder from the internet.
This is working partially right now with the CODER_AGENT_HEADER variable set. With the changes in the previous ticket, the Coder agent's primary HTTP connections are working and I have a live agent that is able to phone home to the Coder server. However, it doesn't look like Wireguard proxied connections are working:
Note the spinner next to Open Ports - and clicking on a Terminal or anything else just brings up a dead window.
In the logs on the Agent side, I see this when trying to launch Terminal:
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.428 [debu] net.tailnet.net.wgengine: derphttp.Client.Recv: connecting to derp-999 (coder)
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derphttp.Client.Recv: TLS version 0x304
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derpclient: got cert coder.tools.plotly.host
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derpclient: got cert Amazon RSA 2048 M03
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derpclient: got cert Amazon Root CA 1
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derpclient: got cert Starfield Services Root Certificate Authority - G2
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.465 [debu] net.tailnet.net.wgengine: derphttp.Client.Recv: not using fast start
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.482 [debu] net.tailnet.net.wgengine: derphttp.Client.Recv: DERP server returned status 302
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.482 [debu] net.tailnet.net.wgengine: ...
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: msg= magicsock: [0xc000704140] derp.Recv(derp-999): derphttp.Client.Recv connect to region 999 (coder): GET failed: <nil>: <html>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: <head><title>302 Found</title></head>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: <body>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: <center><h1>302 Found</h1></center>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: </body>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: </html>
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.482 [debu] net.tailnet.net.wgengine: derp-999: [v1] backoff: 842 msec
which indicates to me that this DERP connection is not respecting the additional header, and instead is getting 302'd to the OIDC IDP, and so it needs to be threaded through for the DERP part of the stack as well.
Another noteworthy log is:
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.534 [debu] net.tailnet.net.wgengine: netcheck: [v1] measuring ICMP latency of coder (999): no address for node 999b
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.598 [debu] net.tailnet.net.wgengine: netcheck: [v1] netcheck: measuring HTTP(S) latency of coder (999): Get "https://LONG REDACTED URL THAT GOES TO OUR IDENTITY PROVIDER SHOWING THAT IT MUST BE FOLLOWING THE 302 REDIRECT/": only one conn expected
Oct 17 23:59:21 mikeheadergcp2-services.gcp.plotly.host coder[9285]: 2024-10-17 23:59:21.598 [debu] net.tailnet.net.wgengine: netcheck: [v1] report: udp=true v6=false v6os=true mapvarydest=false hair=false portmap= v4a=34.150.228.252:39918 derp=0
which shoes me that this latency check is also getting bounced off the auth redirect on the ALB.