ENGINEERING VERSION LOG · 2026.07

SenseVoice and Whisper inference benchmark

Fast GPU inference does not guarantee a fast public-network experience. This benchmark separates model compute, service time, and full end-to-end latency.

InkTyper Engineering2026.07.29

204msCUDA for 7.22s Chinese audio

218msCUDA for 7.07s English audio

272msCUDA for 64.40s Chinese audio

1,246MiBResident SenseVoice GPU memory

ScopeControlled hardware benchmarkIt does not directly establish concurrent capacity

EvidenceSame-node CUDA inference plus an older public-path comparisonObserved results, targets, and estimates are kept separate.

StatusReproducible but sample-limitedThis log records a dated engineering state, not an evergreen promise.

01 · TEST BED

One machine and one audio format

The test node used an RTX 3080 20GB with PyTorch 2.6, CUDA 12.4, FunASR, and faster-whisper. Samples were normalized to 16kHz mono WAV and sent through the production container local endpoint.

02 · SENSEVOICE

Long-form Chinese improved in both speed and completeness

SenseVoice Small processed 7.22 seconds of Chinese audio in 204ms CUDA time and 64.40 seconds in 272ms. The three samples did not reproduce the looping fragments seen in the earlier Whisper streaming assembler.

03 · WHISPER BASELINE

Model compute and public end-to-end time diverged

Whisper Small used 449ms CUDA time for 7.22 seconds of Chinese audio, while the old public route took 3.15 seconds. A 64.40-second clip used 2.00 seconds of CUDA but 5.65 seconds end to end. Network and scheduling cannot hide behind model benchmarks.

04 · SCHEDULING

A single GPU slot favored stability first

Only one model remained resident, old weights were released on switching, and models unloaded after 30 idle minutes. Before increasing concurrency, queue, GPU, and end-to-end latency must be measured separately.

EVIDENCE · DIAGNOSIS

What the evidence establishes—and what it does not

The benchmark separates model compute from product latency. On the RTX 3080 20 GB node, local CUDA time for 7.22 s and 64.40 s Chinese samples was about 204 ms and 272 ms. That describes a compute ceiling, not queueing, upload, gateway, AI, or paste.

Reading ruleProduction observations describe the observed path. Controlled benchmarks isolate a component. Targets remain targets until a sustained distribution proves them.

OPERATIONS · OPEN WORK

Failure behavior and the next verification gate

Comfortable single-card concurrency depends on segment arrival distribution and the shared inference lock, not only on how many models fit in VRAM. Release evidence still needs cold/warm single-user runs, 2/4/8 concurrency, queue time, peak VRAM, error rate, and end-to-end p95 tied to exact model versions and an audio manifest.

Failure visibility: Every fallback needs a reason code and stage timing. A successful final transcript must not erase evidence that real-time, AI, or paste failed first.
Release evidence: Keep request ID, actual model, node, queue, upload, ASR, AI, and delivery timing together so a regression can be reconstructed end to end.