Is crawlee still maintained?

Yes, active. Last change today. Latest release v3.18.2, 2 months ago.

How widely used is crawlee?

Widely used. 26k stars, 10 years old, by an organization. As with any download, get it from the official repo.

gitmanual

Reading the repo

Opening the box…

How to install crawlee on Linux (apify/crawlee) · gitmanual

How to install on Linux

apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Node.js#apify#automation#crawler#crawling

Before you start

Is it free?: YesOpen source (Apache-2.0).
Still maintained?: Yes, activeLast change today. Latest release v3.18.2, 2 months ago.
How widely used?: Widely used26k stars, 10 years old, by an organization. As with any download, get it from the official repo.

Difficulty: Easy
Time: about 5 min
Stars: ★ 26,437
Activity: Updated this month

Saved setup from 6 h ago

Will it work on Linux?

Yes, as a library on Linux

It's a library: you add it to your own project (npm install crawlee playwright), there's no app to open.

Your system

macOS Windows Linux

Start here

Website ↗crawlee.devThe project's own site.

Use it in your own project

This is a library: you add it to your code rather than opening an app.

For your own code: JSNeeds Node.js

npm install crawlee playwright

For your own code: JSNeeds Node.js

npm install crawlee@next

From the README, exactly as the maintainers wrote it.

Did you get it running?

No reports for Linux yet.

Still stuck? Get a written guide

GitManual AI Agent reads the 10 key files and writes a step-by-step guide for Linux. Once written, it's saved for everyone.

Parts list

1×Node.js 24.22.0runtime
1×pnpm pnpm@12.18.0package manager

Share this manual

Is this your repo?

Add a badge to your README so newcomers find these steps.

[![Setup manual](https://gitmanual.com/apify/crawlee/badge.svg)](https://gitmanual.com/apify/crawlee)

Good to know

Monorepo (workspaces) — install from the repo root.
Nested projects: docs/package.json, packages/basic-crawler/package.json, packages/browser-crawler/package.json, packages/browser-pool/package.json and 11 more

Details

License: Apache-2.0
Open issues: 141
Branch: master

package.json scripts (25)

prepublishOnlyturbo run copy --filter=./packages/*
cleanturbo run clean --filter=./packages/* && rimraf .turbo packages/*/.turbo packages/*/*.tsbuildinfo
buildturbo run build --filter=./packages/* && node ./scripts/typescript-fixes.mjs
ci:buildturbo run build --filter=./packages/* --cache-dir=".turbo" && node ./scripts/typescript-fixes.mjs
testvitest run --silent
test:e2enode test/e2e/run.mjs
test:integrationcross-env CRAWLEE_DIFFICULT_TESTS=1 vitest run --silent=true test/integration
test:integration:services:updocker network create crawlee-it 2>/dev/null; docker run -d --rm --name crawlee-it-browserless --network crawlee-it -p 3000:3000 -e CONCURRENT=4 ghcr.io/browserless/chromium && docker run -d --rm --name crawlee-it-httpbin --network crawlee-it --network-alias httpbin -p 8080:80 kennethreitz/httpbin
test:integration:services:downdocker stop crawlee-it-browserless crawlee-it-httpbin; docker network rm crawlee-it 2>/dev/null; true
test:fullcross-env CRAWLEE_DIFFICULT_TESTS=1 CRAWLEE_TEST_BROWSERS=chromium,firefox,webkit vitest run --silent
tsc-check-teststsc --noEmit --project test/tsconfig.json
coveragevitest --coverage
publish:nextlerna publish from-package --contents dist --dist-tag v4 --force-publish --yes
release:nextpnpm build && pnpm publish:next
publish:rclerna publish from-package --contents dist --dist-tag rc --force-publish --yes
publish:prodlerna publish from-package --contents dist --force-publish --yes
release:prodpnpm build && pnpm publish:prod
release:pin-versionsturbo run copy -- -- --pin-versions
api:extractpnpm build && pnpm dlx github:apify/api-extractor-report --exclude=@crawlee/cli,@crawlee/templates --extract-command='pnpm api:extract'
api:checkpnpm dlx github:apify/api-extractor-report --exclude=@crawlee/cli,@crawlee/templates --extract-command='pnpm api:extract' --verify
lintoxlint packages test docs --tsconfig=tsconfig.json --type-aware
lint:fixoxlint packages test docs --tsconfig=tsconfig.json --type-aware --fix
formatoxfmt packages test docs --write
format:checkoxfmt packages test docs --check
preparehusky

Files we read (10)

.github/workflows/check-pr-title.yml

.github/workflows/deploy-nginx.yml

.github/workflows/docs.yml

.github/workflows/test-ci.yml

.github/workflows/test-e2e.yml

.github/workflows/test-integration.yml

CONTRIBUTING.md

README.md

package.json

pnpm-workspace.yaml

3,023 files in the repo

Contact

apify/crawlee

Before you start

Your setup manual for crawlee

What's inside

Handy for

Yes, as a library on Linux

Start here

Use it in your own project

Did you get it running?

Still stuck? Get a written guide

Yes, as a library on Linux

Start here

Use it in your own project

Did you get it running?

Still stuck? Get a written guide

More like crawlee that run on Linux