At a really high level, the performance issues come down to packet reordering. The receiving TCP stack gets segments slightly out of order (1-4 packets affected is common), which generates dup ACKs, and the sending TCP stack lowers the congestion window (cwnd).

I did some hacking in wireguard-go to dump crypto keys to file when establishing tunnels and to tailscale to capture and dump packets as they are pushed into the magicsock (in addition to the hooks already in place to capture packets as they enter the tailnet via the TUN).

This allowed me to use Wireshark to decrypt the wireguard packets and verify they are in order as they go into the magicsock. They are.

Next, I used tcpdump to capture packets as they are passed to the NIC on the sender and as they are received from the NIC at the receiver. Since wireguard packets have a cleartext counter, I was able to verify that they are passed to the sending NIC in order, but then come back out of order from the receiving NIC.

So, the reordering is happening very deep in the networking stack. However, I ran raw iperf3 in UDP mode between the same VMs, and iperf3 doesn't report any UDP packet reordering.

I also set up a kernel wireguard tunnel between the VMs and then ran iperf3 in TCP mode, tunneled over wireguard. I captured the encrypted packets and checked the counters --- kernel wireguard doesn't exhibit reordering. I'm not sure why this is the case.

I have recreated schmoder workspaces with machine_type=c3d-standard-8 and gVNIC (following the example VMs from Colin's previous tests), and in these workspaces I see performance similar to Colin's original tests, with no significant reordering. If I had to guess, I think that something tailscale is doing on the UDP socket is interacting with the older VIRTIO network interfaces, and that gVNIC doesn't exhibit the problem.

I plan to re-do some tests with the other machine_type to confirm that gVNIC makes the difference. If confirmed we should find some way to at least document the interaction for customers, if not understand it & fix it.

Investigate degraded GCP networking performance #13042

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions