KineticLLama is a drop-in, OpenAI-compatible proxy that guarantees deterministic structured output and reliable tool calls from small local models — on Ollama, vLLM and llama.cpp. No retries. No meaningful latency.
{
"items": [{"type": "Keyboard"}], // wrong shape
"status": "in_progress", // not in enum
"zip_code": "NY-10001" // fails ^\d{5}$
}
# Ollama native + pattern → HTTP 400
{
"items": ["2x Mechanical Keyboards"],
"status": "processing",
"zip_code": "10001"
}
# x-kineticllama-valid: true
Running 3B–14B models locally is cheap and private. But under real schemas, they drift — and prompt-and-pray costs you retries, latency and silent bad data.
Conversational filler and Markdown fences wrap the JSON and break your parser before logic even runs.
Arrays of strings become arrays of objects, enums invent new values, and regex constraints like ZIP codes are ignored.
Retrying invalid responses 3–4 times erases the entire cost and latency advantage of running local.
Point your OpenAI or Ollama client at KineticLLama instead of the backend. It compiles your JSON Schema into a grammar the model must obey — and repairs anything a black-box backend gets wrong.
Your schema is compiled to a grammar (GBNF) and enforced during sampling. Invalid tokens are masked — the model physically cannot produce malformed structure.
When the backend can't constrain, a deterministic repair engine cleans the stream: strips filler, closes truncated JSON, normalizes quotes — then validates against your schema.
Constrained decoding is commodity. Reliability across a whole agent run is not.
OpenAI-compatible endpoint. Change one base URL — your existing client, LangChain or agent just works.
Objects, enums, arrays, $ref, anyOf and regex patterns compile to GBNF — including constructs backends reject outright.
The same broken input always yields the same valid output. No LLM in the repair loop, no randomness.
Offered tools compile into the grammar, so a model cannot invent a tool that was never offered or send another tool's arguments. Output is normalized to real tool_calls objects.
A grammar cannot express value ranges. Instead of silently dropping minimum or maxLength, we report them per-field and let the validator rule.
Every response reports the path taken, whether it was repaired, and whether it passed your schema — via x-kineticllama-* headers.
A single static Rust binary with a tiny footprint. Runs alongside your model on a Jetson or gateway without hogging VRAM.
The "hard" extraction task with an array-of-strings, an enum and a ^\d{5}$ ZIP pattern. Same prompt, same 3B model, five approaches.
| Approach | Structure & enum | ZIP ^\d{5}$ | Result |
|---|---|---|---|
| Prompt-only (Ollama) | valid | NY-10001 | invalid, silent |
| Ollama native, no pattern | valid | NY-10001 | invalid, silent |
| Ollama native, with pattern | — | — | HTTP 400 crash |
| KineticLLama repair (Path B) | valid | NY-10001 | flagged invalid |
| KineticLLama grammar (Path A) | valid | 10001 | guaranteed valid |
// The prompt fed the model "NY-10001". The grammar made any non-5-digit ZIP physically ungeneratable → "10001". Same schema Ollama returns a 400 on.
Structured output has two failure classes. We are precise about which one we solve, because a confidently-wrong value is worse than a loud failure.
A grammar guarantees the shape of output, never its truth. Constraining a field can push a model to invent a plausible value rather than admit uncertainty.
minimum/maxLength) — reported, then validatedallOf// Today: native grammar enforcement on llama.cpp (JSON schemas and tool calling), repair + validation on any OpenAI-compatible backend, non-streaming. Streaming repair and agent state graphs are on the roadmap.
The proxy and JSON enforcement are free forever. Pay for agent-grade constraints, observability and enterprise controls.
MIT/Apache proxy + core engine. For builders and self-hosters.
tool_calls normalizationFor teams building local agentic workflows in production.
On-prem, high-availability middleware for regulated teams.
Single-JSON constrained decoding is now commodity — most runtimes ship it. KineticLLama sits above that primitive: it compiles schema constructs backends reject (like regex patterns), repairs black-box output, normalizes tool calls across backends, and gives you observability across a whole agent run.
On the native path, no meaningful overhead — grammars are compiled once and cached, and enforcement happens inside the backend's own sampler. Repair runs as a fast streaming pass, still far cheaper than retry loops.
Any OpenAI-compatible backend for repair and validation. Native grammar enforcement works today with llama.cpp's server; Ollama, vLLM and TGI adapters follow the same interface.
KineticLLama is on-prem first. Nothing leaves your machine by default, and telemetry — when enabled — is content-free metrics only, never prompts or outputs.
Rust. A single static binary with a tiny memory footprint, designed to run alongside a model on edge hardware without competing for VRAM.
Guarantee valid structured output from the small models you already run. Free to start, one base URL to change.