🕷️ ScraperPro

A powerful, terminal-based web scraper built in pure Python 3. Search multiple engines, extract rich data, crawl websites, and export results — all from your terminal with no browser needed.

---

✨ Features

• 🔍 Multi-engine search — DuckDuckGo, Bing, Wikipedia (no API keys needed)
• 📧 Extracts emails, phones, images, social links, tables
• 🌐 Crawl mode — follows internal links across a whole website
• 🔄 Rotating user-agents + optional proxy support
• 💾 Export to JSON, CSV, and TXT
• ⚡ Multi-threaded — parallel scraping for speed
• 🖥️ Pure terminal UI — works inside nano, bash, any shell

---

📁 Project Structure

scraperpro/
├── scraper_pro.py   ← main script (run this)
├── README.md        ← you are here
└── requirements.txt ← optional dependencies

---

🚀 Quick Start

1. Clone the repo

git clone https://github.com/aymanaghi/scaperpro.git
cd scaperpro

2. Install dependencies (optional but recommended)

pip install requests beautifulsoup4 lxml

The scraper works with zero dependencies using Python's built-in urllib. Installing the above just makes it faster and more reliable.

3. Run

python3 scraper_pro.py

---

🧭 Menu Options

#	Option	Description
1	Search & scrape by keyword	Searches DDG + Bing + Wikipedia, scrapes all results in parallel
2	Scrape a single URL	Deep extraction from one specific page
3	Crawl a website	Follows internal links up to N levels deep
4	Configure proxies	Add rotating proxies (host:port or user:pass@host:port)

---

📤 Output

Every run auto-exports 3 files:

File	Contents
.json	Full structured data per page
.csv	Flat table — one row per page, all fields
.txt	Human-readable report

---

📦 What Gets Extracted

• Emails — regex-filtered, deduplicated
• Phone numbers — validated by digit count
• Images & media — src URLs for images, video, audio
• Social links — Twitter/X, Instagram, LinkedIn, GitHub, TikTok, Discord, Telegram, and 15+ more
• HTML tables — parsed into rows and cells
• Keyword text blocks — paragraphs/headings matching your search term
• Internal & external links — up to 100 of each

---

⚙️ Requirements

• Python 3.8+
• No mandatory external libraries (pure stdlib works)

Optional (for speed + reliability):

requests
beautifulsoup4
lxml

---

🛡️ Ethical Use

This tool is intended for legal, ethical web scraping only:

• Always respect a website's robots.txt
• Do not overload servers — use the built-in politeness delays
• Only scrape publicly available data
• Do not use for spam or unsolicited contact

---

👤 Author

Ayman Naghi
GitHub: @aymanaghi

---

📄 License

MIT License — free to use, modify, and distribute.