Skip to content

chore: add tx metrics and logs for serialization errors #15215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 25, 2024
Merged

chore: add tx metrics and logs for serialization errors #15215

merged 12 commits into from
Oct 25, 2024

Conversation

Emyrk
Copy link
Member

@Emyrk Emyrk commented Oct 24, 2024 •
edited
Loading

Before db_metrics were all or nothing. Now InTx metrics are always recorded, and query metrics are opt in.

What this does

Adds instrumentation & logging around serialization failures in the database. When the database hits a serialization error, we retry the transaction up to 3 times. This mostly works, however in production we have seen all 3 retries fail.

We have 0 information on the frequency of failures if they retry once or twice. Or where they are located. Making debugging challenging.

The goal of this PR is not to solve the issue, but begin to investigate the frequency and location of the failures. This is assuming sometimes retries are successful.

Example metric of a failure.

Log line:

[erro]  database transaction hit serialization error and had to retry  success=false  executions=2  id=foobar_factory

Prometheus

# HELP coderd_db_tx_duration_seconds Duration of transactions in seconds.
# TYPE coderd_db_tx_duration_seconds histogram
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.005"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.01"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.025"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.05"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.1"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.25"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="0.5"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="1"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="2.5"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="5"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="10"} 1
coderd_db_tx_duration_seconds_bucket{id="foobar_factory",success="false",le="+Inf"} 1
coderd_db_tx_duration_seconds_sum{id="foobar_factory",success="false"} 9.135e-06
coderd_db_tx_duration_seconds_count{id="foobar_factory",success="false"} 1
# HELP coderd_db_tx_executions_count Total count of transactions executed. 'retries' is expected to be 0 for a successful transaction.
# TYPE coderd_db_tx_executions_count counter
coderd_db_tx_executions_count{id="foobar_factory",retries="1",success="false"} 1

Context

For investigating https://github.com/coder/customers/issues/683

@Emyrk Emyrk marked this pull request as ready for review October 24, 2024 17:07
@Emyrk Emyrk requested a review from coadler October 25, 2024 13:56
@Emyrk Emyrk merged commit ccfffc6 into main Oct 25, 2024
27 checks passed
@Emyrk Emyrk deleted the stevenmasley/sql_metrics branch October 25, 2024 16:14
@github-actions github-actions bot locked and limited conversation to collaborators Oct 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@coadler coadler coadler approved these changes

Assignees

@Emyrk Emyrk

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Emyrk @coadler