Lists · Linux
Web scraping and crawling tools for Linux
Open-source scrapers, crawlers and browser automation tools that run on Linux, with a setup manual for each.
15 repos, most stars first
- firecrawl/★ 200kfirecrawlSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥 Involved30+ minTypeScript
- browser-use/★ 120kbrowser-useOpen-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API. Easy~5 minPython
- D4Vinci/★ 100kScrapling🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev Moderate~15 minPython
- heygen-com/★ 81khyperframesWrite HTML. Render video. Built for agents. Moderate~15 minTypeScript
- scrapy/★ 65kscrapyScrapy, a fast high-level web crawling & scraping framework for Python. Easy~5 minPython
- huginn/★ 50khuginnCreate agents that monitor and act on your behalf. Your agents are standing by! Involved30+ minRuby
- fastapi/★ 46kfull-stack-fastapi-templateFull-stack web application template with FastAPI, React, SQLModel, PostgreSQL, Vite, Tailwind CSS, shadcn/ui, FastAPI Cloud, and Docker Compose. Moderate~15 minTypeScript
- lightpanda-io/★ 42kbrowserLightpanda: the headless browser designed for AI and automation Easy~5 minZig
- microsoft/★ 39kplaywright-mcpPlaywright MCP server Easy~5 minTypeScript
- h4ckf0r0day/★ 31kobscuraThe headless browser for AI agents and web scraping Easy~5 minRust
- MODSetter/★ 17kSurfSenseAir-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9 Moderate~15 minPython
- pinchtab/★ 11kpinchtabHigh-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard. Moderate~15 minGo
- EchterAlsFake/★ 59unofficial-api-for-xnxxUnofficial Python client for xnxx.com: search, metadata extraction, and video download (curl-cffi + BeautifulSoup) Moderate~15 minPython
- hemanta212/★ 11Nepali-news-portal-kbdOnline webapp that scrapes news from different new portals of Nepal and worldwide. Hosted at heroku. Moderate~15 minPython
- ariyankhalid97-creator/★ 1lead-intelligence-workbenchLocal-first Windows lead research and outreach workspace with multi-source discovery, evidence-backed verification, optional Ollama AI, and approval-gated Zoho campaigns. Easy~5 minPython
More for Linux
- Self-hosted apps that run on Linux60+ repos →
- Open-source AI tools that run on Linux60+ repos →
- Run LLMs locally on Linux22 repos →
- AI coding agents and tools for Linux60+ repos →
- AI image generators you can run on Linux15 repos →
- Speech-to-text and voice tools for Linux25 repos →
- Command-line tools for Linux56 repos →
- MCP servers you can run on Linux53 repos →
- Open-source developer tools for Linux36 repos →
- Databases and database tools for Linux13 repos →
- Open-source note-taking apps for Linux13 repos →
- Open-source video and audio tools for Linux36 repos →
- Privacy and security tools that run on Linux34 repos →
- Static site generators and blog engines for Linux7 repos →
- Open-source apps with a ready-made Linux download60+ repos →