— Field Note · From the Forge —
Voice, on the device.
A real-time speech loop with no backend, no lock-in, and nothing leaving the phone.
We wanted an AI you talk to out loud. Not a chat window with a microphone icon — a voice that answers in character, in Polish or English, fast enough that nobody reaches for their phone while they wait.
We built it for a tabletop gaming table. The domain is play. The constraints are not. Three of them shaped every decision, and all three come back the moment anyone asks for a voice interface in a warehouse, a truck cab, or a service desk.
- Push to talk One utterance per turn. Tap again to cancel.
- Speech recognition On the device. The raw audio never leaves the phone.
- Model seam One interface, three vendors. Key read from the keychain at send time, never cached.
- Sentence splitter Streams the reply out sentence by sentence, so speech starts before the answer is finished.
- Voice seam Device voice by default, cloud voice if a key is present. Same seam either way.
- Speaker + face Mouth driven by the amplitude of the audio — no phonemes, so any language works.
No server appears anywhere in this diagram. That is the point of it.
I
No backend.
There is no Járnhaus server anywhere in this system. None.
The user brings their own model key. It goes into hardware-backed storage — iOS Keychain, Android Keystore — marked device-only, never synced, never written to a plain file, never in the bundle. Requests go straight from the phone to whichever vendor the user picked. Speech recognition runs on the device, so the raw audio becomes text before anything is sent anywhere at all.
This is an architecture choice with a legal shadow. No server means no logs to breach, no processor to name in a DPA, no cross-border transfer to justify. For a game that is merely tidy. For a company in Munich weighing a voice interface for its staff, it is the entire conversation.
II
No lock-in.
Anthropic, OpenAI and Google sit behind one interface. Switch vendor in settings; nothing downstream notices.
That is easy to claim, so here is the rule that makes it true: no call site in the app imports a vendor SDK. Call sites import the seam. Vendor SDKs live only inside their own adapter file — and a test reads the registry bytes to catch anyone who quietly imports one there. Adding a fourth vendor is one new file and one line, not a refactor.
The seam also forces honesty about failure. Every adapter must classify its own errors into the same four causes — auth, quota, network, unknown — because something went wrong is not a message anyone can act on. A vendor cannot be registered at all without bringing that classifier with it.
Ask what happens when your model provider triples its price, goes down for a day, or ships a version that is worse at your task. If the answer is a rewrite, the architecture was wrong.
III
Latency you can hear.
The naive loop is: record, transcribe, wait for the whole reply, synthesise it, play it. Every one of those steps is dead air, and dead air is where people give up on voice.
So the reply is streamed and spoken sentence by sentence. Which sounds
simple until you have to decide where a sentence ends inside a token
stream. np. is not a sentence. 3.14 is not
two. A full stop at the end of a growing snapshot might be a full stop,
or it might be the first dot of an ellipsis that has not arrived yet.
The rule we settled on: when unsure, hold. Never speak a fragment, never drop prose. A boundary only counts once whitespace proves the model moved on. The test suite sweeps a no-content-loss invariant across the whole table, so the rule survives edits by people who were not there when it was made.
The mouth on the animated face is driven by the raw amplitude of the audio, not by phonemes. It cost less to build and it works in any language — including the two we shipped and the ones we did not.
Time-to-first-audio is the number that matters. Total time is what you optimise second.
IV
The part that is not about voice.
Every decision above has an identifier, and it is written into the file that implements it — beside the code, with the reason and the price paid.
One example. A model vendor is offered to the user as usable only after that vendor's own reply has been recorded on a real device. Not after its tests pass. A green suite has never flipped that flag and is not permitted to. All three vendors earned it separately, each off its own capture, on a real phone.
Fifty-seven of the hundred and twenty-five source files are tests. The pure arithmetic — lip-sync amplitude, sentence boundaries, locale rules — is written as pure functions with no SDK imports, so it tests with zero mocks and no audio hardware.
This is what we mean by spec-driven. Not documents nobody reads. Decisions written down, priced, and made hard to unpick by accident.
V
What it looks like away from the table.
Strip the fantasy persona off and a pattern is left standing:
- a hands-free voice loop for people whose hands are already busy — warehouse picking, driver check-ins, field service;
- running on the device, so it survives bad signal and keeps recordings out of anyone's cloud;
- over an interchangeable model layer, so the vendor decision stays reversible;
- in two languages, because the crew and the client rarely share one.
We did not build a warehouse tool. We built the loop, on our own time, against tighter constraints than a warehouse would have set. The code is the argument.
The receipts
313 commits. TypeScript, React Native, Expo. MIT licensed and complete — screens, adapters, voice loop, tests, the lot. Read it before you talk to us.