The golang idtoken library first calls generateAccessToken on the impersonated service account as the source user, and then uses that access token to call generateIdToken on the service account.
This requires the service account to have the permission of iam.serviceAccounts.getOpenIdToken access on itself.

The issue is that the idtoken library [in Go lang] does not use the source_credentials subfield in the JSON struct when constructing the inner client, and instead uses the entire credential json. The other clients (like JS and PHP clients) do not operate in this way.