WebReaper

Scrape any site.
Feed your AI.

WebReaper is an AI-native web scraper for .NET. One ~12 MB binary turns any site, even bot-protected ones, into clean Markdown or structured data. Build the pipeline below and the command and the code write themselves.

MIT license~12 MB single binary.NET libraryMCP serversbring any LLM: or none
REAPING LINE: configurator every option is a real flag or builder call
CLI one binary, no runtime 

            

          

          

            
C#: same line, as a library
              

            

            
Output shape for this config: 

          

        

      

    


    

      
Every station is a seam.

      
The builder is sugar over public interfaces. Don't like a station?
      Implement one interface and clip your own part into the line: the other stations never notice.
      Fifteen NuGet packages of ready parts ship in lockstep.

      

        

          
Transport

          IPageLoadTransport
          
How a URL becomes a document. HTTP lives in core; browsers are satellite parts.

          
  • HTTP (core)
  • WebReaper.Playwright
  • WebReaper.Cdp
  • Stealth.CloakBrowser

        

        

          
Extraction

          IContentExtractor
          
How a document becomes data. Deterministic fold by default; LLM adapters when you ask.

          
  • Markdown (core)
  • Schema fold (core)
  • WebReaper.AI inferrer
  • LLM fallback / self-heal

        

        

          
Destination

          IScraperSink
          
Where records land. Fan-out is concurrent; add as many sinks as you like.

          
  • Console / CSV / JSONL
  • WebReaper.Mongo
  • WebReaper.Redis
  • WebReaper.Cosmos

        

        

          
Crawl state

          IScheduler · IVisitedLinkTracker
          
Queue and dedup live behind seams too: swap in Redis or Service Bus and
          multiple workers share one crawl. Same code, distributed.

          
  • in-memory / file (core)
  • WebReaper.Sqlite
  • WebReaper.Redis
  • AzureServiceBus

        

      

    


    

      
The line defends itself.

      
Most scrapers make you guess when a site is blocking you. WebReaper detects the
      block and escalates on its own: per page, paying the climb once per host.

      

        

          
1 · HTTPfast fetch, zero overhead403 challenge

          
2 · Browserheadless Chromium renderstill blocked

          
3 · StealthCloakBrowser fork200 OK

        

        
    
          
  • Climbs only when a page actually looks blocked: status, header, or body marker.
    • 
          
  • First confirmed block lifts that host's floor: the rest of the crawl starts at the working tier.
    • 
          
  • Still blocked at the top? The page is dropped and the run exits non-zero. Challenge pages never masquerade as data.
    • 
          
  • Self-hosted. No cloud round-trip. Stealth (~220 MB) downloads only if you opt in.
    • 
        

      

    


    

      
Clips straight into your agent.

      
Three interop surfaces, one engine underneath.

      

        

          
Claude Code skill

          
One command writes the skill; the next session routes "scrape X" intents to the CLI automatically.

          
$ webreaper init
Wrote WebReaper Agent Skill to
  .claude/skills/webreaper/SKILL.md

        

        

          
MCP servers

          
Six tools: scrape, map, extract, extract_with_prompt, extract_inferred, crawl: over stdio (Cursor, Claude Desktop) or Streamable HTTP (n8n).

          
$ docker run -p 8080:8080 \
  -e WEBREAPER_MCP_TOKEN=secret \
  ghcr.io/alex-on-ai/webreaper-mcp-http

        

        

          
Autonomous agent (library)

          
A goal, a URL, any IChatClient. Decide → persist → execute, with durable resume across restarts.

          
var r = await LlmAgent.RunAsync(
  "https://example.com",
  goal: "find the support email",
  chatClient);

        

      

    


    

      

        
Bolt it to your bench.

        

          
macOS / Linuxbrew install alex-on-ai/webreaper/webreaper

          
any POSIX shcurl -fsSL https://raw.githubusercontent.com/alex-on-ai/WebReaper/master/scripts/install.sh | sh

          
.NET librarydotnet add package WebReaper

        

        
Windows binaries on GitHub Releases ·
        six platforms · macOS builds are Apple-notarized · no Docker, no signup, no metering.