Description
The following issues appear to have reappeared.
Example failure as follows.
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 553, in _compiler return dialect.statement_compiler(dialect, self, **kw) File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/pybigquery/sqlalchemy_bigquery.py", line 137, in __init__ super(BigQueryCompiler, self).__init__(dialect, statement, column_keys, inline, **kwargs) TypeError: SQLCompiler.__init__() got multiple values for argument 'cache_key'
I am not in a position to downgrade to an older sqlalchemy version below 1.4.22. I have attempted using newer versions of 1.4.x series and 2.x series of sqlalchemy and latest versions of this library.
All efforts (after making required code changes to manage library changes) inevitably result in this error.
What is most peculiar is up until 2025-05-24 00:00:00 UTC there was no sign of this error. Then throughout the day of 2025-05-24 00:00:00UTC it was intermittent. Then from 2025-05-25 00:00:00 UTC onwards the error has is consistently raised. The local and container environments have not changed over this period (except for attempting to debug).
Maybe there has been some upstream change that could be causing this error as a red herring?
Environment details
- OS type and version: Ubuntu 22.04 WSL and 24.04 Container Executions.
- Python version:
Python 3.10.12 - pip version:
pip 22.0.2 sqlalchemy-bigqueryversion:Version: 1.7.0through to most recent1.14.1
Steps to reproduce
- Run our code as normal...
- See examples below that all trigger the same result.
Code example
# engine creation:
kwargs = {
"location": location
}
if service_account_credentials_json:
kwargs["credentials_path"] = local_credentials_path
engine = create_engine(f'bigquery://{project}', **kwargs)result_set: Result = engine.execute(orm)result_set: Result = engine.execute(sql)with engine.connect() as engine_connection:
result_set: Result = engine_connection.execute(orm)with engine.connect() as engine_connection:
result_set: Result = engine_connection.execute(text(sql)) with Session(engine) as session:
result_set: Result = session.execute(orm)...
with Session(engine) as session:
result_set: Result = session.execute(text(sql))
...Stack trace
Traceback (most recent call last):
File "/home/ben/scti_repos/azure-data-shuttle/main.py", line 444, in <module>
main()
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/ben/scti_repos/azure-data-shuttle/main.py", line 416, in main
process_changes(
File "/home/ben/scti_repos/azure-data-shuttle/main.py", line 66, in process_changes
generator_df: Iterator[pd.DataFrame] = source.fetch(
File "/home/ben/scti_repos/azure-data-shuttle/source.py", line 105, in fetch
select_fields=[f.name for f in entity.fetch_fields()],
File "/home/ben/scti_repos/azure-data-shuttle/entity.py", line 79, in fetch_fields
self._fields = self.source.fetch_fields(self)
File "/home/ben/scti_repos/azure-data-shuttle/sources/database/bigquery/source.py", line 20, in fetch_fields
self.client.fetch_dict(
File "/home/ben/scti_repos/azure-data-shuttle/database.py", line 211, in fetch_dict
df = self.fetch_df(sql=sql, path=path, orm=orm, **sql_variables)
File "/home/ben/scti_repos/azure-data-shuttle/database.py", line 180, in fetch_df
dfs = list(self.fetch_generator(sql=sql, path=path, orm=orm, **sql_variables))
File "/home/ben/scti_repos/azure-data-shuttle/database.py", line 132, in fetch_generator
result_set: Result = session.execute(text(sql))
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1583, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1444, in _execute_clauseelement
compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 531, in _compile_w_cache
compiled_sql = self._compiler(
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 553, in _compiler
return dialect.statement_compiler(dialect, self, **kw)
File "/home/ben/scti_repos/azure-data-shuttle/venv/lib/python3.10/site-packages/pybigquery/sqlalchemy_bigquery.py", line 137, in __init__
super(BigQueryCompiler, self).__init__(dialect, statement, column_keys, inline, **kwargs)
TypeError: SQLCompiler.__init__() got multiple values for argument 'cache_key'