ENGINEERING VERSION LOG · 2026.08

From tail latency to a real-time transcription path

Long recordings exposed stop-time tail latency: faster isolated GPU inference does not automatically create a real-time experience. This report defines the target SLO and staged delivery path. It is a production design and experiment log, not a claim that the full release gate has already passed.

InkTyper Engineering2026.08.02

250msTarget p50 for the first partial

500msTarget p95 for final ASR after stop

700msTarget p95 for Fast Polish

1.5sTarget p95 from stop to delivered text

ScopeProduction-path design with staged implementationTargets are not production promises

EvidenceTarget SLOs plus an implemented segmented pathObserved results, targets, and estimates are kept separate.

StatusPartially deployedThis log records a dated engineering state, not an evergreen promise.

01 · THE SYMPTOM

The batch path leaves the important work until after stop

When the complete recording uploads only after stop, ASR, final correction, and AI editing stack serially. That path remains useful for old clients, long files, and an explicit streaming fallback, but it cannot remain the primary experience.

Electron microphone→Streaming gateway→Warm ASR→Incremental polish

02 · LATENCY BUDGET

Define perceived latency before optimizing isolated benchmarks

The targets are a 250ms p50 first partial, 200ms p50 partial cadence, 500ms p95 final ASR after stop, 700ms p95 Fast Polish, and 1.5s p95 stop-to-delivered text.

These are release budgets. Real-request percentiles and a soak test must prove them; one fast GPU invocation cannot.

03 · PROGRESSIVE ASR

Segmented recognition stays timely and stop corrects only the tail

The current production path uses pause-aware segmented SenseVoice throughout. It produces partial and stable segments during recording, then closes only the unstable tail at stop instead of decoding the complete recording again. Paraformer has left production routing because of unacceptable English and Chinese-English behavior.

Stable clauses enter Fast Polish during recording. The final request contains only the unstable tail and a short immutable context.

04 · FALLBACK

Fallback must be explicit, idempotent, and bounded

Old clients and failed streaming sessions may fall back to POST /api/transcribe.
The fallback reason enters telemetry and full inference is retried no more than once.
Suspicious short output is preserved and flagged, not recursively re-inferred.
When AI exceeds its 700ms deadline, reliable ASR text is delivered immediately.

05 · RELEASE GATE

The production control plane and 1,000-request soak test remain

Release requires an always-on gateway, Redis/Postgres, at least two ASR replicas, and a 1,000-request soak test. Acceptance covers latency plus zero duplicated text, language flips, or lost final segments.

EVIDENCE · DIAGNOSIS

What the evidence establishes—and what it does not

The root cause was not one slow model. The old path serialized upload, ASR, AI, and paste after stop. The new path separates work done during recording from work left at stop: stable segments are produced early, only the tail is finalized, and reliable ASR is delivered when AI fails.

Reading ruleProduction observations describe the observed path. Controlled benchmarks isolate a component. Targets remain targets until a sustained distribution proves them.

OPERATIONS · OPEN WORK

Failure behavior and the next verification gate

Queue time under concurrency, gateway backpressure, session recovery, and replica removal still require proof. A single node can validate the interaction but cannot establish multi-replica availability. The 250 ms, 500 ms, and 1.5 s figures are budgets until sustained telemetry demonstrates them.

Failure visibility: Every fallback needs a reason code and stage timing. A successful final transcript must not erase evidence that real-time, AI, or paste failed first.
Release evidence: Keep request ID, actual model, node, queue, upload, ASR, AI, and delivery timing together so a regression can be reconstructed end to end.