Subly In closed testing

A video in a language you do not speak, a lecture or a stream with no subtitles. Cloud services can transcribe and translate it, but that means sending everything you listen to off the phone.

Subly does it on the phone itself. It captures the audio another app is playing, transcribes it, translates it, and draws the captions in a floating overlay on top of that app.

The engine is a standalone Android SDK, subly-android-sdk, with a Flow-based API: a session reports its state, and captions arrive as a stream. The app is a thin consumer of it, with onboarding, language selection and the overlay. Developing them as a Gradle composite build means an SDK change shows up in the app immediately, without publishing.

Model files are downloaded on first use rather than bundled, pinned to exact upstream revisions and checked against SHA-256 hashes where they come from mirrors. A first run downloads only the recogniser for the source language.

No single speech recogniser is good at all five languages on a phone. I benchmarked more than twenty candidates on the Galaxy S10, the low-end baseline, against FLEURS clips with human references, and picked one per language with a fixed rule:

1. 1Lowest error among models that decode at least twice as fast as real time on the S10.
2. 2Prefer a streaming model within about 2 points of the best offline one, because streaming shows words as they are spoken.
3. 3Among near-equals, prefer the smaller download.

The measurements ruled out some obvious choices:

• The most accurate model was too slow. Qwen3-ASR led on several languages, but ran at 0.6 to 0.85 times real time on the S10 and is a 987 MB download.
• A streaming model that cannot stream. Parakeet's streaming variant took 12.7 seconds to decode each second of audio on the S10.
• Segmentation matters as much as accuracy. Offline models caption one speech segment at a time. With a short pause rule, every breath in Vietnamese became its own caption and was translated as a fragment; a 0.8 s pause made captions whole sentences and cut word error from 12.6% to 9.7%. ReazonSpeech showed the opposite: a 9.4 s segment decoded as a single word, so Japanese keeps short segments.

Dropping the two older general-purpose engines from the app also shrank the release APK from 110.8 MB to 95.4 MB. They stay in the SDK for other consumers.

ML Kit translates a line in about 100 ms with a 30 MB model per language pair, and it runs anywhere. A small on-device LLM translates noticeably better: it knew that "elements" on the periodic table are chemical elements, where ML Kit chose the word for "factors". But the LLM is a 2.6 GB download and needs a phone with plenty of memory.

So the translator depends on the device. On phones with 10.5 GB of RAM or more, Subly uses Gemma 4 E2B through LiteRT-LM; elsewhere it uses ML Kit. A handover translator prepares both at once and starts with whichever is ready first, then switches to the LLM once it has loaded. That turned out to matter for more than speed: ML Kit's model downloads sometimes hung for over 20 minutes with no error, and the handover kept captions working regardless.

Gemma 4 E2B matched or beat the previous model in 6 of 8 directions and was 3.2 times faster per sentence. Turning on its multi-token prediction drafter made sentences another 19% faster at the same quality, well short of the "up to 2.2x" headline, which was measured on a newer phone.

The first full pipeline was accurate but late. On fast Mandarin dialogue translated to Vietnamese, the first translated words appeared 11.5 seconds after a sentence started. A per-caption latency tracker showed three causes, in order of size:

1. 01The translator was slow

   Lines arrived faster than the old model could translate them, so they queued. Moving to Gemma 4 E2B fixed most of it.

2. 02Sentences were committed late

   Nobody in a fast dialogue pauses for 0.8 s, so the recogniser only finished a sentence at its 10 s cap. An early committer now takes a sentence from the live hypothesis once it is complete, followed by more speech, and unchanged for about 0.4 s.

3. 03Nothing was translated while someone spoke

   A live preview now translates the sentence still being spoken whenever the translator is idle, and is cancelled the moment committed work needs it.

Every model choice and default in Subly comes from a measurement, and the numbers live next to the code that depends on them. A dedicated benchmark module, never shipped, runs on physical phones: a 2019 Galaxy S10 with 8 GB as the low-end baseline and a Galaxy S24 Ultra with 12 GB.

• A real test set. FLEURS clips in all five languages with human references, plus a real recording of fast Mandarin dialogue, scored by word or character error.
• Tested the way users run it. The final numbers come from the release app with a clip playing in a video player and captured through MediaProjection, not from feeding audio files straight into the engine.
• Honest comparisons. The S24 throttles after about an hour of GPU work, so A/B runs alternate on and off back to back. A control run caught a speech-gate change that had quietly moved more than a third of caption starts, and it was fixed.

Audio is held only in short-lived memory buffers while it is processed, and captions are cleared shortly after they stop updating. Nothing is written to storage and no audio or text is sent anywhere. Even the benchmark logs only print transcripts when verbose logging is switched on explicitly, because transcripts are the user's speech.

Android asks for consent each time capture starts and shows a notification for as long as it runs. The network is used to download models and to send diagnostics, which the privacy policy spells out.

• In closed testing on Google Play, captioning and translating between English, Chinese, Japanese, Spanish and Vietnamese.
• Japanese and Vietnamese use offline recognisers, so their source text appears about 2.5 s into a sentence, against about 0.5 s for the streaming languages. That is the next latency gap to close.
• Gemma 4 E2B has not yet been measured on 8 GB phones. If it fits, it could replace ML Kit on the S10 class of device too.
• Recognisers are chosen per language today; choosing them per device class, as translation already is, would let 12 GB phones use the most accurate model.

Numbers measured in October 2026.