Local-First Hermes Runtime Architecture

Source Excerpt

Summary

The local-first Hermes runtime architecture is the emerging design direction for running Clark Farming Company’s day-to-day agent work primarily on local models while using frontier models only as reviewers, evaluators, curriculum designers, and occasional planners. The core principle is simple: local models do the work; frontier models improve the harness outside the hot path.

The discussion began with a broad idea: use frontier Large Language Models to maximize the effectiveness of weaker local inference by optimizing the Hermes Agent harness around it. The early architecture proposed lightweight launch files, deterministic skills, context tiering, ;;eval capture, weekly frontier review, and possible progressive tool injection. Later governance review narrowed the implementation path: avoid deep Hermes core patches on a machine that updates nightly; build update-safe plugin/telemetry/eval infrastructure first; attack tool-output bloat before building complex runtime tier machinery.

This page records the durable synthesis and the current implementation boundary.

Runtime Lane vs. Improvement Lane

The architecture separates two lanes.

The runtime lane is local-first. Routine work should run through local Qwen3.6-27B-class inference via Hermes, with local tools, local artifacts, and local session state. This is the hot path: reading files, using tools, writing drafts, managing wiki work, inspecting code, and performing bounded workflows.

The improvement lane may use frontier models, but not as live workers. Frontier models are useful for weekly review of eval artifacts, failure analysis, skill rewrites, proposed eval cases, launch-file critique, and harness architecture review. Their output should generally be reports, recommendations, and proposed patches. The system should not automatically apply frontier suggestions without review.

This distinction matters because routing normal work to frontier models defeats the purpose of local-first infrastructure. The goal is not to make local models pretend to be frontier models. The goal is to build rails that let local models perform bounded work reliably.

Launch File Philosophy

SOUL.md and other always-loaded files should stay lightweight. Julius and the other agents should retain identity and moral/operational posture, but procedural clutter should move out of launch context and into skills, references, or deterministic scripts.

The durable split is:

• SOUL.md: identity, nonnegotiables, evidence-before-claims, safety/approval stance, operating temperament.
• AGENTS.md / TOOLS.md / COMMUNICATION.md: practical behavior rules, role boundaries, tool-use discipline, channel norms.
• Skills: repeatable workflows, exact steps, verification checklists, expected artifacts, failure recovery.
• Model adapters: optional and only justified by repeated model-specific failure patterns.

This reduces time to first token, limits instruction collision, and makes local models less likely to drown in broad doctrine.

Context Tiering

The initial runtime design favored a two-tier context model:

• Tier 1: 64K context, Qwen3.6-27B no-reasoning, default for new sessions.
• Tier 2: 262K context, same local model class, hot idle for long-session continuation.

The intent was to promote automatically near roughly 95% of Tier 1 usage, without warning, without asking, and without compaction first. Promotion would not move the llama.cpp KV cache; it would transfer useful structured session state into the higher-context server. The goal was to replace lossy compaction with local continuation.

Empirical review supported 64K as a practical starting tier. Recent non-cron sessions had approximate raw transcript percentiles around p50 22K, p75 40K, p90 64K, and p95 79K tokens. However, the same review showed that long sessions were often dominated by tool output, not human dialogue. That finding changed the implementation priority.

;;eval Capture

The ;;eval control is a capture command, not an evaluation command. It should save a deterministic artifact with the user’s note, session reference, recent turns, recent tool calls, current model/tier metadata where available, and pointers to raw logs or session state.

It must not:

• call a frontier model immediately;
• begin judging the session;
• trigger a second model call unless explicitly added later;
• change runtime tier by itself.

The weekly frontier review should read eval artifacts first and follow full-session references only when the artifact lacks enough context. This makes frontier review targeted rather than broad and expensive.

Lazy Tool Loading and Prior Art

The research pass confirmed that lazy tool loading is real prior art, not just a local invention. The paper "Tool Attention Is All You Need" proposes compact tool summaries, intent routing, full-schema promotion only for selected tools, and deterministic rejection of hidden tool calls. Claude Code / Agent Skills embody a similar principle for procedural content: metadata is visible, but full skill bodies load only when needed. Model Context Protocol supports dynamic tool lists through protocol concepts such as tools/list and tools.listChanged, though clients can still waste context by eagerly dumping schemas into prompts.

The lesson for Hermes is not to immediately build a clever router. The lesson is to keep discovery surfaces small, load full capability detail only when needed, preserve deterministic guards, and gather usage telemetry before optimizing.

Tool Output Stewardship

The strongest near-term insight is that tool output pressure may matter more than tool schema pressure. Large terminal logs, web extracts, browser snapshots, session search windows, dependency install logs, build logs, long JSON, and repeated stack traces can bury the signal and consume context on every later turn.

The safe principle is context stewardship, not blind truncation:

• Raw tool output is never lost.
• Large raw results are saved verbatim as artifacts.
• Active model context receives success/failure, metadata, head/tail excerpts, detected errors/warnings, and the artifact path.
• The agent can retrieve exact raw content by range when needed.

This avoids the bad version of summarization where details disappear. It turns a giant blob into an indexed artifact while preserving auditability.

Rollout should be staged:

1. Telemetry only: log tool name, result size, duration, session ID, and rough token estimate.
2. Advisory mode: save references but do not alter model-visible output.
3. Conservative reference mode: for oversized results only, store raw output and return metadata/excerpts/reference.
4. Tool-specific handlers: customize behavior for terminal, pytest, web extracts, session search, browser snapshots, and other high-bloat tools.

Scope Decision

After critique, the implementation scope narrowed. Because the Mac Studio updates Hermes nightly, deep core edits would create a maintenance tax. The current decision is:

• keep Hermes core stock for now;
• build update-safe user-space plugin work under ~/.hermes/plugins/cfc-runtime/ where possible;
• prioritize tool telemetry, large-output artifact/reference handling, and capture-only eval artifacts;
• run weekly frontier review over artifacts;
• audit skills and launch files for local-friendly determinism;
• defer automatic context tiering, dynamic schema injection, runtime profile state machines, and core ;; parser patches until evidence justifies upstreamable seams.

Future work can still include 64K→262K promotion, 32/64/140/262 context ladders, ;;tier, ;;long, ;;reason, Tool Attention-style lazy schema loading, and usage-history-driven tool profiles. They are deferred, not rejected.

Operating Rule

Build evidence first. Improve the rails before modifying the engine. If Hermes core needs new hooks, they should be generic, tested, and upstreamable rather than CFC-only hacks.