Stop paying for easy prompts

Architecting hybrid local-to-cloud AI systems

Abed Matini · written companion to a 45-minute talk for Escape 2026 on 10 September.

https://www.youtube.com/watch?v=e9QDEYfsxJ4

Many AI applications begin with one model endpoint. A standup summary, invoice extraction, and security architecture review all take the same cloud path.

Those requests do not require the same capability, but they inherit the same network hop, pricing model, and data boundary.

At small volume, that simplicity is sensible. At production volume, it becomes a decision worth measuring. A hybrid architecture gives predictable, low-risk work a local execution path and reserves cloud models for tasks that need more capability.

The goal is not “local first” at any cost. It is the least expensive path that still meets your quality, reliability, and policy requirements.

The one-endpoint problem

A cloud-only architecture is easy to build:

application → cloud model → response

The problem appears when every request takes that route by default.

• Cost: high-volume routine work accumulates API spend.

• Latency: every request pays for a network round trip, even when local execution could satisfy it.

• Data exposure: text crosses a network and vendor boundary before the application has decided whether that is necessary.

Cloud is not automatically slow or expensive, and local execution is not free. Local hardware consumes power, memory, and engineering time. The useful question is whether each workload needs the capability and operational profile of the cloud path.

This article avoids hard-coding a provider price into the architecture. Prices change. Check the current Gemini API pricing when calculating savings, and include local operating cost in the comparison.

“Easy” is a workload class

An easy request is not simply a short prompt.

Suitable for local evaluation:

• predictable and repeatable;

• low-risk;

• constrained output;

• easy to validate automatically;

• supported by the selected local model.

Examples include classification, constrained JSON extraction, short summaries, and simple rewrites.

Better suited to a stronger cloud model:

• ambiguous or multi-step;

• context-heavy;

• high-consequence;

• difficult to validate;

• beyond the measured capability of the local model.

A short medical or legal question may be high-risk. A long, structured extraction may be routine. Route by required capability and risk, not by token count alone.

When the router is unsure, choose the safer path and record why.

The conference demo makes that decision deliberately transparent:

def classify(prompt):
    if has_hard_markers(prompt) or word_count(prompt) >= 120:
        return "cloud"
    if has_easy_markers(prompt) and word_count(prompt) <= 80:
        return "local"
    if word_count(prompt) <= 40:
        return "local"
    return "cloud"  # ambiguous requests default safely

Forced routes and sensitive-data handling are applied separately. These thresholds are teaching rules, not universal definitions of intelligence. A production router should derive its task classes and thresholds from labelled requests, evaluation results, and the consequences of a wrong decision.

One application, two execution paths

Application
    │
    ▼
Policy router ──► local SLM ──┐
    │                         │
    └──────────► cloud LLM ───┴─► validated response

Before cloud: detect and minimize sensitive data
After local:  validate and fall back once if needed

The application talks to one routing layer. That layer inspects the task, sensitivity, size, and risk before choosing an execution path.

Use stable names such as local and cloud. Let the router own auto. Application code should not repeat provider-specific model IDs across every service.

The router should choose the cheapest path that meets the quality and policy bar. That policy—not a particular model name—is the architecture.

Local is not automatically better

A hybrid design needs an explicit quality guardrail:

• small models have lower capability ceilings;

• local inference can be slower on weak hardware;

• local infrastructure still needs authentication, updates, observability, and capacity planning;

• keeping data on one device does not fix poor access control;

• sensitive tasks may require stronger controls regardless of model location.

The companion demo uses qwen2.5:0.5b because it is small enough for a conference laptop and fails visibly on work beyond its capability. It demonstrates routing. It is not a production model recommendation.

Quality is the gate. Cost optimization begins only after the output passes the task evaluation.

Routing is a request lifecycle

Model selection is one step in a larger lifecycle:

1. Inspect the request: task, sensitivity, size, output contract, and risk.

2. Choose the local or cloud path.

3. Execute using a stable backend alias.

4. Validate schema, safety rules, task-specific quality, and timeout.

5. Fall back once when a local result fails the bar.

6. Respond once through the application’s normal contract.

7. Record the decision, reason, model, latency, fallback, and estimated cost.

A model returning text does not mean the request succeeded.

For extraction, validate required fields and types. For grounded answers, check that requested evidence is present. For every route, enforce a timeout. If a local result fails validation, perform one controlled cloud fallback. If that also fails, expose the failure instead of hiding it behind an infinite retry loop.

Different execution paths can still produce one application response. Telemetry explains how that response was produced.

The films accompanying the talk use the same single-path lifecycle as the runnable demo: select local or cloud, validate the candidate result, fall back to cloud once when a local attempt fails, and return one response. The demo’s checks are intentionally simple transport and weak-answer tests; a production system needs concrete schema, safety, task-quality, and timeout validators.

Give each tool one job

The demo separates five responsibilities:

Ollama provides a convenient local model server. llama.cpp is a widely used local inference runtime. A deployment may use either directly or through higher-level tooling; they are not the same architectural layer.

LiteLLM provides the gateway seam. The application calls stable aliases while configuration maps them to the current providers.

The demo uses qwen2.5:0.5b locally and gemini/gemini-3.6-flash in the cloud. If the cloud model changes, update CLOUD_MODEL rather than every caller.

How LiteLLM works in this demo

application → policy chooses an alias → LiteLLM → provider API
                                      ├─ local → Ollama
                                      └─ cloud → Gemini

The application uses one call shape:

complete("local", messages)
complete("cloud", messages)

LiteLLM translates the request for the selected provider and normalizes the response and token-usage shape. The default CLI path uses the LiteLLM Python SDK directly. The optional LiteLLM proxy exposes the same aliases through an OpenAI-compatible endpoint and can add deployment load balancing, retries, health checks, and controlled fallback.

LiteLLM does not decide what “easy” means in this project. The application policy chooses local or cloud first; LiteLLM delivers the request to the configured backend. Keeping those responsibilities separate makes the routing policy easier to test and change.

Multi-turn memory is an application responsibility

Hybrid routing introduces an important conversational challenge.

Suppose the first request runs locally, the second runs in cloud, and the third depends on both. Neither model automatically remembers the earlier calls. Model APIs are effectively stateless request interfaces, and Ollama and Gemini do not share hidden state or a transferable KV cache.

A conversation displayed in a browser is not necessarily model context. In the current demo, the React interface retains messages for display, but /api/chat sends only the newest prompt. The selected model therefore receives no previous turns.

A production design needs memory above the router:

session → context builder → policy router → local or cloud
   ▲                                         │
   └──────────── append response ─────────────┘

The application should:

1. Store a canonical session containing user turns, model responses, tool results, and route/model provenance.

2. Build context from the recent window, relevant older turns, retrieved facts, and a running summary.

3. Route the current task using that explicit context.

4. Append the response regardless of which model produced it.

Do not resend the entire transcript forever. That increases latency and token cost, and it may expose information that should stay local. For sensitive conversations, maintain two representations:

• a complete local/private history;

• a redacted or summarized cloud-safe view containing only what the cloud task requires.

The portable memory contract is explicit conversation text, summaries, retrieved facts, and tool state—not provider-specific hidden state.

Start with an explainable router

The first router can be a small set of visible rules:

if sensitive data is found:
    redact locally, then route the minimized text
if task is high-risk, ambiguous, or requires heavy reasoning:
    cloud
if task is constrained, measured, and easy to validate:
    local
otherwise:
    cloud
if local fails validation:
    cloud fallback once

Rules can use task type, sensitivity, context size, required output shape, and risk. Log every decision and evaluate routing mistakes.

Once labeled traces exist, the design can mature:

1. Rule-based routing: explicit thresholds and task classes.

2. Semantic routing: compare prompt embeddings with known routes.

3. Learned routing: predict whether the cheaper model will satisfy the request.

4. Continuous evaluation: update thresholds using production traces.

RouteLLM and Semantic Router are useful later steps. They are not prerequisites for testing the architecture.

Minimize data before cloud

Edge preprocessing should reduce the data that crosses the cloud boundary:

1. Detect known sensitive fields locally.

2. Replace them with stable, typed placeholders.

3. Keep the placeholder-to-original mapping inside the local boundary.

4. Remove irrelevant text or summarize locally when the summary preserves what the task requires.

5. Route only the minimum necessary redacted context.

6. Restore authorized values locally only if the final response requires them.

For example:

Email abed@example.com and use sk-demo-secret
         ↓ detect and replace locally
Email [EMAIL_1] and use [API_KEY_1]

Using the same placeholder whenever a value repeats preserves relationships without disclosing the original identifier. The demo uses regular expressions for email addresses, phone numbers, account numbers, API-key patterns, payment-card patterns, and South African IDs. Card candidates must also pass a Luhn check.

The demo creates and retains the replacement mapping, but it does not implement response restoration. That is an application workflow decision: many cloud tasks never need the original value reinserted.

Regex makes the processing order visible, but it cannot understand every context-dependent identifier or relationship.

Redaction reduces exposure. It does not by itself guarantee POPIA or GDPR compliance. Production systems need tested DLP, access controls, retention policies, vendor governance, and appropriate legal review.

Use synthetic data when demonstrating this flow.

Try the four scenarios

The CLI runs four requests through the same policy:

python -m demo.router --scenario A   # constrained summary → local
python -m demo.router --scenario B   # architecture reasoning → cloud
python -m demo.router --scenario C   # PII → redact locally → cloud
python -m demo.router --scenario D   # forced local failure → cloud fallback

Watch the telemetry rather than judging the generated prose:

• route

• reason

• model

• latency_ms

• est_cost_usd

• fallback

Use --dry-run to inspect routing decisions without calling a model.

The project also includes a chat interface backed by the same router:

./scripts/start-chat.sh

Open http://127.0.0.1:43173. Load scenarios A–D or enter your own prompt. Route only performs the same classification as --dry-run, and Force local demonstrates the demo’s simple weak-answer and fallback path.

Never commit .env, expose a real API key on stage, or paste customer data into a demo.

Measure the whole system

A smaller API invoice is not enough to prove success.

If quality drops outside the target, the route failed even if it saved money. If fallback is frequent, local execution may add latency without producing useful savings.

Start with one narrow production task

Do not begin with a platform rewrite.

1. Label one week of requests by task, risk, and sensitivity.

2. Choose one high-volume task with a clear automated evaluation.

3. Add a local path behind a feature flag.

4. Compare it with the cloud baseline.

5. Add output validation, one fallback, and route telemetry.

6. Expand only when quality, reliability, latency, cost, and privacy remain inside the targets.

A gateway is useful immediately. A learned router can wait until there is evidence to train and evaluate it.

Optional: more local capacity

The same local API contract can later point to more than one approved machine. Multiple known deployments can share a LiteLLM model name so independent requests are distributed across healthy workers:

application → policy router → local alias ─┬─ worker A
                                           ├─ worker B
                                           └─ worker C
                           ↘ cloud             policy or fallback

This is a worker pool: each request goes to one worker running the complete model. It improves throughput and resilience; it does not make the small model more capable.

True model sharding is a different design. It divides one model execution across machines and requires a distributed inference runtime plus fast, reliable networking. LiteLLM brokers requests across configured endpoints; it does not discover laptops, enrol devices, or split model weights.

That does not make a colleague’s laptop a free server. Ownership, authentication, software updates, capacity, availability, and data boundaries must all be explicit. Peer execution is an optional scale-out pattern, not a requirement of the hybrid architecture.

Takeaways

1. Route by required capability and risk, not prompt length alone.

2. Use local models only for workload classes they pass in evaluation.

3. Keep local and cloud providers behind stable gateway aliases.

4. Keep conversational memory in an application-owned session above the router.

5. Build a separate minimized, cloud-safe context when sensitive history is involved.

6. Validate every result and cap fallback.

7. Measure quality and reliability before claiming cost savings.

8. Treat local worker pools and distributed model sharding as different architectures.

9. Start with explainable rules; adopt semantic or learned routing when traces justify it.

Predictable work can run locally. Work that needs more capability can use the cloud. The router makes that trade-off explicit, measurable, and replaceable.