Firecrawl Web Search Evaluation

Source Excerpt

Background

In May 2026, Firecrawl was installed and configured on the Hermes Agent instances as an advanced web search and scraping capability. The question: does it actually improve search quality over the existing web_search tool?

Existing Search Tools

At the time of evaluation, Hermes Agent provided:

1. web_search — General-purpose web search engine. Good for broad queries, news, current events.
2. web_extract — Extract content from URLs into markdown format.
3. Browser tools — Full browser automation for complex interactions.

Firecrawl Capabilities

Firecrawl offers:

• Advanced scraping — Handle JavaScript-rendered pages, pagination, dynamic content
• Structured extraction — Extract specific data patterns from web pages
• Crawling — Follow links and build site maps
• Document parsing — Convert PDFs, images, and complex documents to text

Evaluation Results

What Worked

• Document parsing — PDF extraction quality exceeded web_extract for complex documents
• JavaScript rendering — Successfully scraped SPAs and dynamic content that web_extract missed
• Local deployment — Self-hosted Firecrawl instance provides privacy and no rate limits

What Didn't Justify the Overhead

• Search quality — Firecrawl's search didn't meaningfully outperform existing web_search for general queries
• Complexity — Additional infrastructure (Firecrawl server, API configuration) for marginal gains
• Use case mismatch — Firecrawl excels at scraping known sites, not discovering unknown information

Decision

Firecrawl was retained for document parsing and complex scraping tasks but not as a replacement for general web search. The existing web_search tool remains the primary search mechanism.

Configuration Notes

• Firecrawl instances running on local infrastructure
• v1 scrape API changed to v2 — requires configuration update
• Environment variables: FIRECRAWL_API_KEY, FIRECRAWL_BASE_URL

SearXNG Integration Decision (May 30, 2026)

Decision: ADD SearXNG to the web search stack (not yet deployed as of May 30, 2026).

What is SearXNG?

SearXNG is an open-source, self-hosted metasearch engine (~31k GitHub stars, actively maintained). It doesn't maintain its own index — it queries 70–243 upstream search providers (Google, Bing, DuckDuckGo, Brave, Wikipedia, academic sources, etc.) and aggregates results. No tracking, no profiling, no ads. Provides a JSON API for programmatic access.

Why Add SearXNG?

• Search breadth — Aggregates across 70+ engines, providing wider coverage than any single search backend
• Privacy — Self-hosted, no tracking or profiling of queries
• Complements Firecrawl — SearXNG handles discovery/search; Firecrawl handles extraction/parsing. The combination covers both halves of the web research pipeline.
• No Hermes code changes needed — Agents can use SearXNG via curl to the JSON API, then Firecrawl for extraction
• Independent validation — Codex (OpenAI's coding agent) independently recommended the same SearXNG + Firecrawl combination

Current Configuration

• Firecrawl running locally at localhost:3002
• Web search config has empty backend fields (web.backend: '', search_backend: '')
• SearXNG not yet deployed

Deploy Plan

1. Deploy SearXNG as a Docker container (one container)
2. Configure JSON API endpoint
3. Agents use SearXNG via curl for search discovery
4. Continue using Firecrawl for page extraction and document parsing
5. No changes to Hermes Agent core code required

Status

Firecrawl: installed and configured. Used for document parsing and complex scraping. Not a replacement for general web search.

SearXNG: approved for addition to stack, not yet deployed. Planned deployment: Docker container, JSON API, paired with Firecrawl for extraction.