Skip to content

Translate SDK - Using External Account Credentials (WIF) Results in Permission Error #13855

New issue
New issue
Open
Bug
Open
Translate SDK - Using External Account Credentials (WIF) Results in Permission Error#13855
Bug
Assignees
parthea
Labels
priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
@pablofgaeta

Description

@pablofgaeta
pablofgaeta
opened on Apr 29, 2025

Determine this is the right repository

  • I determined this is the correct repository in which to report this bug.

Summary of the issue

Context
I am deploying building a CICD pipeline in Gitlab that authenticates to Google Cloud using Workload Identity Federation. The workload identity pool and the service account it impersonates are in a separate project from the main application. Some of the pipeline tests require usage of the google-cloud-translate client using quota in the application project. Because the Translation API is client-based, I have to specify the quota project ID of the application project to authenticate.

The error indicates that the Service Usage Consumer role is missing on the application project, but I've added this role to both the WIF principal as well as the service account it's impersonating. I've also tested directly calling the Translation API with curl on the same pipeline step with no issues, so it seems to be a client-related issue.

Expected Behavior:
Providing the quota project ID to translation client methods (e.g. translate.TranslationServiceAsyncClient.translate_text), successfully returns a translated response.

Actual Behavior:
The following exception trace with application code and project information redacted.

self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7f5c0eaf3290>
    def __await__(self) -> Iterator[P]:
        try:
>           response = yield from self._call.__await__()
.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:85: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = Getting metadata from plugin failed with error: ('Unable to acquire i...\\n        ]\\n      }\\n    ]\\n  }\\n}\\n\')", grpc_status:14, created_time:"2025-04-29T21:47:08.572523819+00:00"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "Getting metadata from plugin failed with error: ('Unable to acquire impersonated credentials', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "containerInfo": "{{APPLICATION_PROJECT}}",\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/{{APPLICATION_PROJECT}}"\n        }\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",\n        "locale": "en-US",\n        "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes."\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}"\n          }\n        ]\n      }\n    ]\n  }\n}\n')"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Getting metadata from plugin failed with error: (\'Unable to acquire impersonated credentials\', \'{\\n  \"error\": {\\n    \"code\": 403,\\n    \"message\": \"Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.\",\\n    \"status\": \"PERMISSION_DENIED\",\\n    \"details\": [\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\\n        \"reason\": \"USER_PROJECT_DENIED\",\\n        \"domain\": \"googleapis.com\",\\n        \"metadata\": {\\n          \"containerInfo\": \"{{APPLICATION_PROJECT}}\",\\n          \"service\": \"iamcredentials.googleapis.com\",\\n          \"consumer\": \"projects/{{APPLICATION_PROJECT}}\"\\n        }\\n      },\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\\n        \"locale\": \"en-US\",\\n        \"message\": \"Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.\"\\n      },\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.Help\",\\n        \"links\": [\\n          {\\n            \"description\": \"Google developer console IAM admin\",\\n            \"url\": \"https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}\"\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n\')", grpc_status:14, created_time:"2025-04-29T21:47:08.572523819+00:00"}"
E               >
.venv/lib/python3.12/site-packages/grpc/aio/_call.py:327: AioRpcError
...
    translation_response = await translation_client.translate_text(
.venv/lib/python3.12/site-packages/google/cloud/translate_v3/services/translation_service/async_client.py:482: in translate_text
    response = await rpc(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7f5c0eaf3290>
    def __await__(self) -> Iterator[P]:
        try:
            response = yield from self._call.__await__()
            return response
        except grpc.RpcError as rpc_error:
>           raise exceptions.from_grpc_error(rpc_error) from rpc_error
E           google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: ('Unable to acquire impersonated credentials', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "containerInfo": "{{APPLICATION_PROJECT}}",\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/{{APPLICATION_PROJECT}}"\n        }\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",\n        "locale": "en-US",\n        "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes."\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}"\n          }\n        ]\n      }\n    ]\n  }\n}\n')
.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:88: ServiceUnavailable

API client name and version

google-cloud-translate==3.15.5

Reproduction steps: code

file: main.py

# /// script
# dependencies = [
#   "google-cloud-translate==3.15.5",
#   "google-auth==2.38.0",
# ]
# ///

from google import auth
from google.cloud import translate

APPLICATION_PROJECT = "..."

credentials, _ = auth.default()
translation_client = translate.TranslationServiceAsyncClient(
    credentials=credentials,
    client_options=client_options.ClientOptions(
        quota_project_id=APPLICATION_PROJECT
    ),
)

translation_response = await translation_client.translate_text(
    request={
        'parent': f'projects/{APPLICATION_PROJECT}/locations/global',
        'contents': ["Antigonish pharmacies"],
        'source_language_code': "gd",
        'target_language_code': "en",
        'mime_type': 'text/plain',
    }
)

print(translation_response)

Reproduction steps: supporting files

file: .gitlab-ci.yaml

variables:
  APPLICATION_PROJECT: "..."

  # WIF configuration
  GCP_PROJECT_NUMBER: "..."
  GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID: "..."
  GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID: "..."
  GCP_SERVICE_ACCOUNT: "..."
  GCP_WORKLOAD_IDENTITY_PROVIDER: "projects/${GCP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID}/providers/${GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID}"

stages:
- test

test:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  image: google/cloud-sdk:latest

  script:
    # Create ADC and login
    - echo "${GITLAB_OIDC_TOKEN}" > .ci_job_jwt_file
    - gcloud iam workload-identity-pools create-cred-config ${GCP_WORKLOAD_IDENTITY_PROVIDER}
      --service-account="${GCP_SERVICE_ACCOUNT}"
      --output-file=.gcp_temp_cred.json
      --credential-source-file=`pwd`/.ci_job_jwt_file
    - gcloud auth login --cred-file=`pwd`/.gcp_temp_cred.json --update-adc --quiet

    # Successful translation
    - |
      curl -X POST https://translate.googleapis.com/v3/projects/${APPLICATION_PROJECT}:translateText \
          -H "Content-Type: application/json" \
          -H "Authorization: Bearer $(gcloud auth print-access-token)" \
          -H "x-goog-user-project: ${APPLICATION_PROJECT}" \
          -d '{
        "contents": [
          "Antigonish pharmacies"
        ],
        "mimeType": "text/plain",
        "sourceLanguageCode": "gd",
        "targetLanguageCode": "en"
      }'

    # Run failing test script
    - curl -LsSf https://astral.sh/uv/0.6.17/install.sh | sh
    - source $HOME/.local/bin/env
    - uv run main.py

  stage: test

  rules:
    - when: always

Reproduction steps: actual results

CURL-based translation

file: stdout

{
  "translations": [
    {
      "translatedText": "Antigonish pharmacies"
    }
  ]
}

Python test file output

file: stdout

self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7f5c0eaf3290>
    def __await__(self) -> Iterator[P]:
        try:
>           response = yield from self._call.__await__()
.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:85: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <_AioCall of RPC that terminated with:
	status = Getting metadata from plugin failed with error: ('Unable to acquire i...\\n        ]\\n      }\\n    ]\\n  }\\n}\\n\')", grpc_status:14, created_time:"2025-04-29T21:47:08.572523819+00:00"}"
>
    def __await__(self) -> Generator[Any, None, ResponseType]:
        """Wait till the ongoing RPC request finishes."""
        try:
            response = yield from self._call_response
        except asyncio.CancelledError:
            # Even if we caught all other CancelledError, there is still
            # this corner case. If the application cancels immediately after
            # the Call object is created, we will observe this
            # `CancelledError`.
            if not self.cancelled():
                self.cancel()
            raise
    
        # NOTE(lidiz) If we raise RpcError in the task, and users doesn't
        # 'await' on it. AsyncIO will log 'Task exception was never retrieved'.
        # Instead, if we move the exception raising here, the spam stops.
        # Unfortunately, there can only be one 'yield from' in '__await__'. So,
        # we need to access the private instance variable.
        if response is cygrpc.EOF:
            if self._cython_call.is_locally_cancelled():
                raise asyncio.CancelledError()
            else:
>               raise _create_rpc_error(
                    self._cython_call._initial_metadata,
                    self._cython_call._status,
                )
E               grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
E               	status = StatusCode.UNAVAILABLE
E               	details = "Getting metadata from plugin failed with error: ('Unable to acquire impersonated credentials', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "containerInfo": "{{APPLICATION_PROJECT}}",\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/{{APPLICATION_PROJECT}}"\n        }\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",\n        "locale": "en-US",\n        "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes."\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}"\n          }\n        ]\n      }\n    ]\n  }\n}\n')"
E               	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Getting metadata from plugin failed with error: (\'Unable to acquire impersonated credentials\', \'{\\n  \"error\": {\\n    \"code\": 403,\\n    \"message\": \"Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.\",\\n    \"status\": \"PERMISSION_DENIED\",\\n    \"details\": [\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\\n        \"reason\": \"USER_PROJECT_DENIED\",\\n        \"domain\": \"googleapis.com\",\\n        \"metadata\": {\\n          \"containerInfo\": \"{{APPLICATION_PROJECT}}\",\\n          \"service\": \"iamcredentials.googleapis.com\",\\n          \"consumer\": \"projects/{{APPLICATION_PROJECT}}\"\\n        }\\n      },\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\\n        \"locale\": \"en-US\",\\n        \"message\": \"Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.\"\\n      },\\n      {\\n        \"@type\": \"type.googleapis.com/google.rpc.Help\",\\n        \"links\": [\\n          {\\n            \"description\": \"Google developer console IAM admin\",\\n            \"url\": \"https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}\"\\n          }\\n        ]\\n      }\\n    ]\\n  }\\n}\\n\')", grpc_status:14, created_time:"2025-04-29T21:47:08.572523819+00:00"}"
E               >
.venv/lib/python3.12/site-packages/grpc/aio/_call.py:327: AioRpcError
...
    translation_response = await translation_client.translate_text(
.venv/lib/python3.12/site-packages/google/cloud/translate_v3/services/translation_service/async_client.py:482: in translate_text
    response = await rpc(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <google.api_core.grpc_helpers_async._WrappedUnaryUnaryCall object at 0x7f5c0eaf3290>
    def __await__(self) -> Iterator[P]:
        try:
            response = yield from self._call.__await__()
            return response
        except grpc.RpcError as rpc_error:
>           raise exceptions.from_grpc_error(rpc_error) from rpc_error
E           google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: ('Unable to acquire impersonated credentials', '{\n  "error": {\n    "code": 403,\n    "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.ErrorInfo",\n        "reason": "USER_PROJECT_DENIED",\n        "domain": "googleapis.com",\n        "metadata": {\n          "containerInfo": "{{APPLICATION_PROJECT}}",\n          "service": "iamcredentials.googleapis.com",\n          "consumer": "projects/{{APPLICATION_PROJECT}}"\n        }\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.LocalizedMessage",\n        "locale": "en-US",\n        "message": "Caller does not have required permission to use project {{APPLICATION_PROJECT}}. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}} and then retry. Propagation of the new permission may take a few minutes."\n      },\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developer console IAM admin",\n            "url": "https://console.developers.google.com/iam-admin/iam/project?project={{APPLICATION_PROJECT}}"\n          }\n        ]\n      }\n    ]\n  }\n}\n')
.venv/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py:88: ServiceUnavailable

Reproduction steps: expected results

CURL-based translation

file: stdout

{
  "translations": [
    {
      "translatedText": "Antigonish pharmacies"
    }
  ]
}

Python test file output

file: stdout

{
  "translations": [
    {
      "translatedText": "Antigonish pharmacies"
    }
  ]
}

OS & version + platform

Debian Bookworm (via google/cloud-sdk:latest image) on Gitlab Runner 17.10.1

Python environment

Python 3.12.10

Python dependencies

google-cloud-translate==3.15.5
google-auth==2.38.0

Additional context

No response

Metadata

Metadata

Assignees

Labels

priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions