Repository navigation
fix(stt): retry refused Soniox transcription deletes so they cannot fill the account - #25432
Merged
Merged
Conversation
…ill the account Soniox batch recovery deletes its temporary transcription after each call. When that DELETE was refused (429 under request-rate limiting, or the transcription was still processing at the 120s deadline), the transcription leaked. Ids are never persisted, so leaked transcriptions are orphans, and they count toward Soniox's 2,000 stored-transcription cap. Prod hit the cap at 2026-11-13 11:10Z; since then every POST /transcriptions has returned 429 and Soniox failover/second-chance units sit in vendor_held rate_limited. A refused transcription DELETE is now queued and retried by the next calls (3 per call, bounded queue), and the cleanup warning carries the HTTP status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Soniox batch recovery (
utils/stt/soniox_batch.py) deletes its temporary transcription after each call. If that DELETE is refused, the transcription leaks. That happens on a 429 under Soniox request-rate limiting, or when the transcription is still processing at the 120s deadline. Transcription ids are never persisted, so leaked ones are orphans, and they count toward Soniox's 2,000 stored-transcription cap.Prod evidence (2026-11-15):
recovery.wavtranscriptions: 100 on Nov 8, 1,081 on Nov 10, 264 on Nov 12, 501 on Nov 13. The newest is 2026-11-13 11:10Z.event=soniox_batch_cleanup outcome=failedper day matches those counts exactly.create429s: 0/hr through Nov 11, then ~420–510/hr from Nov 13 12:00Z on.vendor_held{reason="rate_limited"}, cycling held → redriven → 429.The 2,000 orphans were deleted operationally (2,000 × 204). This PR stops the leak from coming back.
Change
http_status.Tests
Failure-Class: none
🤖 Generated with Claude Code