Reading the repo
Reading the repo
Assembly instructions
Pipeline to find company websites, detect contact forms and load results into MySQL
Saved setup from 5 h ago
Will it work on macOS?
pip install selenium webdriver-manager requests beautifulsoup4 pydoll-python mysql-connector-python
Google Chrome is required for the Selenium and pydoll stages.
db_loader.py reads its connection settings from environment variables:
export DB_HOST=... DB_USER=... DB_PASSWORD=... DB_NAME=... DB_PORT=3306
Batch sizes, crawl depth and delays are set in the CONFIG block at the top of each script.
Then run bash setup.sh. It stops at the first error and is safe to run again. Servers and tests aren't started; it lists them at the end. Read it first.
Needs Git: xcode-select --install
git clone https://github.com/Hassan-as-DA/contact-form-submission-pipeline.git cd contact-form-submission-pipeline
Python 3
brew install python
In the maintainers' order. Do the README instructions above first — they cover steps outside the terminal.
pip install selenium webdriver-manager requests beautifulsoup4 pydoll-python mysql-connector-python export DB_HOST=... DB_USER=... DB_PASSWORD=... DB_NAME=... DB_PORT=3306
On macOS: 0 got it running, 1 got stuck.
Opens a chat with this manual already filled in, for macOS. It uses your own account; gitmanual sends nothing.