Hermes Agent Configuration Lessons

Source Excerpt

Overview

Lessons learned from configuring and operating Hermes Agent across the retired internal project Quad infrastructure.

Key Lessons

Cron Job Timeouts

Problem: Local inference models (Qwen3.6-27b via LM Studio) are slow. Default 600-second timeout causes cron jobs to fail mid-execution.

Solution: Increase timeout to 1200 seconds for jobs running local inference.

Affected Jobs:

• Wiki lint cron (local)
• Industrial knowledge curation (local)

Model Selection

Problem: Different tasks require different model capabilities. Default model isn't optimal for all tasks.

Solution: Use per-job model overrides:

• julius-3.6-27b-nothink for fleet updates (fast, no reasoning needed)
• Default model for complex reasoning tasks

Tool Restriction

Problem: Loading all toolsets for simple cron jobs wastes input tokens.

Solution: Use enabled_toolsets to restrict jobs to only the tools they need:

• ["terminal", "delegation"] for fleet updates
• ["terminal", "delegation", "file"] for wiki lint

SSH Key Management

Problem: SSH connections to remote machines fail with "too many authentication failures" when multiple keys are offered.

Solution: Use IdentitiesOnly yes in SSH config and specify the exact key file.

Wiki Path Consistency

Problem: Different agents have different wiki paths:

• Julius: ~/ClarkFarmingCompany/wiki/wiki/
• Reacher: ~/ClarkFarmingCompany/wiki/ (no double wiki)
• Octavius: /home/jared596/wiki/wiki/

Solution: Always verify wiki path before dispatching tasks. Document in agent memory.

Session File Format

Problem: Session files are JSONL format with nested JSON. Agents need to parse them correctly.

Solution: Use Python's json.loads() for each line. Skip empty lines. Extract role and content fields.

Session Search Timeout with Local Models

Problem: session_search tool fires up to 3 concurrent summarization requests (one per matching session), each sending ~100K characters of conversation transcript through the Large Language Model for summarization. When running a local llama-server with --parallel 1 (single request queue), these requests queue sequentially instead of running concurrently. With a 27B model generating 10K+ output tokens (including 8K reasoning budget), each summary takes 5-15+ minutes. Three sequential summaries easily exceed the cron job's 1800-second inactivity limit, causing the entire cron job to die with:

Job 'Quad Daily Standup' idle for 1802s (inactivity limit 1800s) | last_activity=executing tool: session_search

Root Cause Chain:

1. llama-server --parallel 1 — single request queue, no concurrency
2. session_search default max_concurrency: 3 — fires 3 concurrent summarization requests
3. Each summary: ~100K chars input, 10K output tokens, 8K reasoning budget
4. At ~5-10 tokens/sec for a 27B local model, each summary takes 5-15+ minutes
5. Sequential on --parallel 1: 3 × 5-15 min = 15-45 minutes total
6. Cron inactivity limit (1800s) kills the job before session_search completes

Solution Applied (2026-05-14):

auxiliary:
  session_search:
    max_concurrency: 1          # Serialize requests to match --parallel 1 server
    summarization_timeout: 300  # 5-minute ceiling per summary (code clamp max in older versions)

Important Notes:

• The summarization_timeout config option was removed upstream in later versions of Hermes Agent. The new code has no batch-level timeout — only per-call HTTP timeout from auxiliary.session_search.timeout: 300.
• With max_concurrency: 1 and no batch timeout, worst case is 3 sessions × 3 retries × 300s = 45 minutes, but realistically 5-15 minutes as failed summaries fall back to raw 500-char previews.
• The --parallel 1 limitation is the fundamental constraint. A faster local model or increasing --parallel on llama-server would be the long-term fix.
• Session search fallback: when summarization fails/times out, returns raw 500-char string slice of the formatted transcript (no Large Language Model involved) — useful enough to identify the session topic.

Long-term Options:

1. Use a dedicated faster local model for auxiliary tasks (session_search, compression, vision)
2. Increase --parallel on llama-server if hardware allows
3. Route auxiliary tasks to a cloud provider instead of local model

Fleet Update Does Not Update Local Machine

Problem: The hermes-fleet-update cron job runs hermes update on remote machines (Cypher, Reacher, Octavius) via SSH but does NOT update the local machine (Julius) that runs the cron. This caused Julius to fall 233 commits behind while the remote machines stayed current.

Root Cause: The fleet update skill is designed to update the remote fleet. Julius (the local machine) is assumed to stay current separately — but there is no automated process for this.

Solution: Include Julius in the update procedure or create a separate local update mechanism. The fleet update skill should document that it does not update the local machine.

Detection: Compare git log --oneline -1 across all machines. Julius was at 3c23b15f8 while remotes were at ddb8d8fa8 (same tag v2026.5.7, different commits).

Additional Blocker: Uncommitted local modifications on Julius prevented hermes update from pulling new commits. The update process uses git stash internally but local changes in core files (like our session_search patch) need careful handling.

Auxiliary Config Requires Gateway Restart

Problem: Changes to auxiliary.* configuration in config.yaml are read once at gateway startup. Modifying the config file does not take effect until the gateway is restarted.

Solution: After changing auxiliary config, run hermes gateway restart to reload the configuration.

Evidence: After patching max_concurrency: 1 and summarization_timeout: 300 in config.yaml, the changes only took effect after hermes gateway restart.

Cypher's Gemini Config Lost — Two Hermes Directories Out of Sync

Date Discovered: 2026-05-15

Problem: Cypher's Gemini provider config disappeared from ~/.hermes/config.yaml, leaving only LM Studio as the primary provider. The Gemini config was intact in ~/hermes/config.yaml but Hermes reads from ~/.hermes/config.yaml.

Root Cause: Cypher's machine has TWO separate Hermes directories:

• ~/hermes/config.yaml — has Gemini config, intact, NOT used by Hermes
• ~/.hermes/config.yaml — minimal LM Studio only, THIS is what Hermes loads

The hermes update process creates pre-update snapshots of ~/.hermes/config.yaml (it's in STATE_FILES), so the Gemini config was captured correctly through May 15 at 9:01 AM. Something after that (likely config migration or fleet update) recreated the file with minimal content.

Timeline:

• May 4: provider: google-gemini-cli with base_url: cloudcode-pa://google
• May 5: provider: gemini with gemini-3.1-flash-lite-preview
• May 12: Still had Gemini config in pre-update snapshot
• May 15: Config overwritten — only LM Studio remains

Resolution: Restored Gemini config from pre-update snapshot (state-snapshots/20260515-090135-pre-update/config.yaml). Gateway restarted to pick up changes.

Prevention:

1. Eliminate the dual-directory situation — ensure ~/hermes/ and ~/.hermes/ point to the same location
2. Consider tracking config.yaml in git to prevent silent overwrites
3. Verify provider config after each fleet update

Source: Session 20260515_104954_671cbc35.jsonl (Discord, May 15, 2026)

Disabled Toolsets — Interactive Session Context Optimization

Date: 2026-05-26

Problem: Tool definitions in the system prompt are the heaviest per-turn context burden. Loading toolsets that are never used (Feishu, Spotify, Discord admin, video tools) wastes input tokens on every turn.

Solution: Use disabled_toolsets in ~/.hermes/config.yaml to exclude unused toolsets from interactive sessions. Changes take effect on next session restart — not mid-conversation.

Current disabled toolsets (Julius, 2026-05-26):

disabled_toolsets:
  - clarify
  - homeassistant
  - computer_use
  - pibrain
  - tts
  - image_gen
  - vision
  - spotify
  - feishu_doc
  - feishu_drive
  - discord_admin
  - yuanbao
  - video
  - video_gen
  - browser

Notes:

• enabled_toolsets in cron jobs.json controls per-job tool access (narrowing). disabled_toolsets in config.yaml controls global interactive session defaults (subtracting).
• Feishu (ByteDance enterprise suite: docs + drive) is not used by CFC.
• hermes gateway restart is required for changes to take effect.
• Startup token count includes all non-disabled tool definitions — reducing enabled tools directly reduces per-turn context overhead.

Source: Session 20260526_000310_213fb1b7 (Discord, 2026-05-26)

Session Storage Format — JSONL Files, Not SQLite

Date Clarified: 2026-05-15

Current State: Sessions are stored as JSONL files organized per agent and per profile:

• ~/.hermes/sessions/ — main profile sessions
• ~/.hermes/profiles/<agent>/sessions/ — per-profile sessions
• Each file: <date>_<time>_<hash>.jsonl with one JSON object per line

Planned State: SQLite SessionDB with FTS5 search is in development but not yet deployed. Do not document SQLite as the current session storage mechanism.

Impact: Previous documentation incorrectly stated sessions were stored in SQLite. System overview document was corrected to reflect JSONL storage.

Source: Session 20260515_104954_671cbc35.jsonl (Discord, May 15, 2026)

Wiki Canonical Location — Local on Julius, Not Octavius

Date Clarified: 2026-05-15

Correct Location: The canonical wiki repository lives locally on Julius at ~/ClarkFarmingCompany/wiki/wiki/, synced to GitHub (jared596/ClarkFarmingCompany-wiki). Octavius maintains a cloned copy at /home/jared596/wiki/wiki/ for local wiki operations.

Previous Error: Documentation incorrectly stated the wiki lived on Octavius and was mirrored locally. The subagent searching for the wiki ran search_files looking for files named "wiki" instead of checking ~/ClarkFarmingCompany/.

Source: Session 20260515_104954_671cbc35.jsonl (Discord, May 15, 2026)

Fleet Update Findings — 2026-05-15

Date: 2026-05-15

Summary: Fleet update successfully pulled 100 commits (ddb8d8fa8 → 13c72fb48) on Cypher, Reacher, and Octavius.

Issues Found:

1. Julius behind: This machine was 45 commits behind upstream (at 4695d2716, needs update to 13c72fb48). The fleet update skill does not update the local machine.
2. Reacher xAI API key commented out: # XAI_API_KEY= in Reacher's .env file. Reacher uses xAI Grok-4.3 as primary model — this needs restoration.
3. AGENTS.md conflicts: All three machines had AGENTS.md conflicts during update — resolved by clean reset and re-copy from CFC source folders.

Prevention: Include Julius in the update procedure or create a separate local update mechanism.

Source: Session cron_b0c0796c3cf5_20260515_050034 (cron fleet update, May 15, 2026)

Status

Lessons documented and applied to current configuration.