Is Scrapegraph-ai still maintained?

Yes, active. Last change 1 months ago. Latest release v2.3.1, 1 months ago.

How widely used is Scrapegraph-ai?

Widely used. 33k stars, 2 years old, by an organization. As with any download, get it from the official repo.

How to install on Linux

ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

Python#ai-crawler#ai-scraping#ai-search#crawler

Before you start

Is it free?: YesOpen source (MIT).
Still maintained?: Yes, activeLast change 1 months ago. Latest release v2.3.1, 1 months ago.
How widely used?: Widely used33k stars, 2 years old, by an organization. As with any download, get it from the official repo.

Difficulty: Easy
Time: about 5 min
Stars: ★ 32,765
Activity: Updated 1 months ago

Saved setup from 5 h ago

Will it work on Linux?

Yes, it installs on your Linux

The README gives a pip install for Linux.

Your system

macOS Windows Linux

Start here

Website ↗scrapegraphai.comThe project's own site.

Also: Docs ↗

Heads-upBuilding it yourself? It also needs 24 settings in .env (1 secret).

Install with a package manager

First: a virtual environment

Python's own advice: install into a fresh environment, in its own folder (~/scrapegraph-ai), so it can't clash with other tools.

mkdir -p ~/scrapegraph-ai
cd ~/scrapegraph-ai
python3 -m venv .venv
source .venv/bin/activate

pipNeeds Python >=3.12,<4.0

pip install scrapegraphai

From the README, exactly as the maintainers wrote it.

Did you get it running?

No reports for Linux yet.

Still stuck? Get a written guide

GitManual AI Agent reads the 15 key files and writes a step-by-step guide for Linux. Once written, it's saved for everyone.

Parts list

1×Python >=3.12,<4.0runtime
1×uv package manager

24× settings in .env

OPENAI_API_KEYsecret
MAX_TOKENSsecretoptional
MODEL_NAMEoptional
TEMPERATUREoptional
CUSTOM_NODE_TIMEOUT
MAX_NODES
DEBUG_MODE
LOG_LEVEL
OCR_ENABLED
EXTRACT_METADATA
MAX_FILE_SIZE
SUPPORTED_FORMATS
MAX_DEPTH
CRAWL_DELAY
RESPECT_ROBOTS_TXT
MAX_PAGES_PER_DOMAIN
USER_AGENT
DEFAULT_LANGUAGE
GENERATE_TESTS
ADD_DOCUMENTATION
CODE_STYLE
TYPE_CHECKING
CSV_DELIMITER
MAX_ROWS

Share this manual

Is this your repo?

Add a badge to your README so newcomers find these steps.

[![Setup manual](https://gitmanual.com/ScrapeGraphAI/Scrapegraph-ai/badge.svg)](https://gitmanual.com/ScrapeGraphAI/Scrapegraph-ai)

Good to know

Dockerfile available (base image: python:3.11-slim).
Makefile has a make install target — it may automate most of these steps.
Has AGENTS.md — instructions for coding agents. You can open Claude Code (or another agent) in the cloned folder and ask it to do the setup for you.

Details

License: MIT
Open issues: 32
Branch: main

Files we read (15)

.github/workflows/codeql.yml

.github/workflows/dependency-review.yml

.github/workflows/release.yml

.github/workflows/test-suite.yml

CONTRIBUTING.md

Dockerfile

Makefile

README.md

docker-compose.yml

examples/code_generator_graph/.env.example

examples/csv_scraper_graph/.env.example

examples/custom_graph/.env.example

examples/depth_search_graph/.env.example

examples/document_scraper_graph/.env.example

pyproject.toml

393 files in the repo

Contact

ScrapeGraphAI/Scrapegraph-ai

Before you start

Your setup manual for Scrapegraph-ai

What's inside

Handy for

Yes, it installs on your Linux

Start here

Install with a package manager

Did you get it running?

Still stuck? Get a written guide

Yes, it installs on your Linux

Start here

Install with a package manager

Did you get it running?

Still stuck? Get a written guide

More like Scrapegraph-ai that run on Linux