跳到主要内容
Supermarket
返回能力市场
MCP Server
data
BSD-3-Clause

Scrapling

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

D4VinciD4Vinci
91/ 100

公开评测 · 综合采用结论

证据充分,整体质量与安全表现优秀

查看评测依据 查找我项目的报告基于公开项目证据,非安全认证或安装推荐
103.7kstars
10.8kforks
最近更新 2天前
评测生成时间(北京时间)
本报告引擎
v3.9.0
当前引擎
v3.17.3

本报告与当前引擎使用不同规则;原分数不会自动更新,不同版本的分数不宜直接对比。

重新评测此项目

进入后确认来源与额度,提交才会创建任务。

代码版本未记录或记录无效,无法核对本报告的 commit 与送评文本;不会用当前项目代码回填历史记录。

Evaluation report

综合采用结论

91
A
满分 100 · 静态采用参考
值得推荐低风险
决策摘要

证据充分,整体质量与安全表现优秀

100%
高置信度
88
文档
100
安全
88
质量
100
活跃
70
采用
  • 基础评测完成+25/25确定性评分与静态安全扫描已完成
  • README 有效证据+25/2521,674 个去重后的有效字符
  • 独立证据来源+20/205 类非重复证据,重复文件不叠加
  • 仓库元数据+10/10已取得仓库状态与采用数据
  • 活跃记录+5/5已取得最近提交时间
  • AI 复核+15/15已完成结构化 AI 证据复核

本报告评了什么

综合分用于静态采用筛选,不是任务成功率,也不代表使用 Skill 后的能力提升。

静态检查范围
记录 6 个文件 · 65,697 字符
只检查采集到的受限文本与已知风险模式;未读取的脚本、依赖和运行环境不在结论内。
AI 内容复核
已使用
复核送评文本的说明和设计,不是安装测试或实际任务执行。记录缺失时不推断已使用或未使用。
真实任务效果
本报告未提供实测证据
没有使用/不使用 Skill 的任务对照、重复运行或耗时与 token 成本结果。采用前仍需在隔离环境验证自己的任务。

脚本引用覆盖

脚本引用覆盖记录不可得;不推断没有引用或已经检查。

How it works · 流程图

Scrapling 抓取流程

README 展示了从安装到执行抓取的连续步骤,包括 fetcher 和 spider 的使用。

AI 提取 · 证据约束

左右滑动查看完整图示

Scrapling 抓取流程README 展示了从安装到执行抓取的连续步骤,包括 fetcher 和 spider 的使用。使用 fetcher返回页面提取数据安装依赖用户获取页面fetcher解析数据parser输出结果spider
图示依据
  • • 安装部分:pip install scrapling
  • • Basic Usage 示例:Fetcher.get 和 page.css
  • • Spiders 示例:MySpider().start()
五维表现
Scrapling 提供从单请求到全站爬取的完整方案,示例丰富,安装步骤清晰,但 README 未明确列出限制与错误处理细节,需查阅文档。
质量证据
  • 安装命令:pip install scrapling 及可选依赖
  • 示例代码:StealthyFetcher.fetch 和 Spider 类
  • 功能列表:暂停恢复、代理轮换、AutoThrottle
  • CLI 示例:scrapling extract get 'https://example.com' content.md
采用建议
优势
  • 问题与用途描述
  • 有效 README
  • 安装或接入步骤
  • 可执行示例
  • 未发现已知高风险模式
关注点
  • 缺少限制、权限或边界
  • README 未列出明确的限制或边界条件
  • 错误处理与排障信息缺失
  • MCP server 的具体配置步骤未在 README 中展示
  • 依赖安装后需额外运行 scrapling install,但未说明失败处理
适合

需要绕过反爬的网页抓取、构建大规模并发爬虫、需要自适应元素定位的长期爬取、AI 代理集成抓取

不建议直接用于

需要严格遵守 robots.txt 的合规场景(需自行配置)、对错误处理有严格要求的自动化流程

也有自己的公开项目?先免登录查已有报告;没有报告时,可保留地址继续登录评测。

查找我项目的报告 →
文档证据
88/100
问题与用途描述10 分
有效 README12 分
安装或接入步骤14 分
可执行示例16 分
输入、参数或工具说明11 分
输出或结果说明9 分
限制、权限或边界12 分
错误处理或排障8 分
许可证信息5 分
结构化章节3 分
安全证据
低风险
未发现已知高风险模式

静态扫描不是安全保证,生产接入前仍应人工复核权限和数据边界。

优先改进清单
  1. 01补充限制、权限或边界
方法、证据与局限展开
数据来源

GitHub Repository API

扫描范围

6 个文件 · 65,697 字符

评测引擎

v3.9.0 · AI 内容复核:已使用(deepseek-chat)

局限
  • 静态评测不会安装或执行项目代码
  • 安全扫描基于高信号文件与已知模式,不能替代人工审计
  • 流行度只反映采用程度,不代表安全或工程质量

30 天热度趋势

README

Scrapling Poster
Effortless Web Scraping for the Modern Web

D4Vinci%2FScrapling | Trendshift Star History Rank
README بالعربية README en Español README em Português (Brasil) README en Français README auf Deutsch 简体中文版自述文件 日本語のREADME Русская версия README 한국어 README
Tests PyPI version Docker Pulls PyPI package downloads Static Badge OpenClaw Skill
Discord X (formerly Twitter) Follow
Supported Python versions

Selection methods · Fetchers · Spiders · Proxy Rotation · CLI · MCP

Scrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.

Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports.

Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.

What you can do with Scrapling

  • Fetch & interact: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
  • Parse & extract: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
  • Crawl & export: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
  • Scrape with AI or from your terminal: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.

Start with a single page:

from scrapling.fetchers import Fetcher

page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)

Or scale up with a spider:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }

result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")

This prints 10 and saves the quotes and authors to quotes.json.

At DataImpulse, we specialize in developing custom proxy services for your business. Make requests from anywhere, collect data, and enjoy fast connections with our premium proxies.

Platinum Sponsors

README 图片 NodeMaven - The most efficient proxy provider for Web Scraping and Automation with the Highest Quality IP on the market. Use code SCRAPLING35 for 35% discount.
README 图片 Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, Hyper Solutions provides API endpoints that generate valid antibot tokens for Akamai, DataDome, Kasada, and Incapsula. Simple API calls, no browser automation required.
README 图片 Evomi : residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass.
Scraper API for hassle-free results. MCP and N8N integrations are available.
README 图片 TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets.
Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off.
README 图片 Read a full review of Scrapling on The Web Scraping Club (Nov 2025), the #1 newsletter dedicated to Web Scraping.
README 图片 Swiftproxy provides scalable residential proxies with 80M+ IPs across 195+ countries, delivering fast, reliable connections, automatic rotation, and strong anti-block performance. Free trial available.
README 图片 Byteful supplies ethical residential, mobile, ISP, and datacenter proxies, plus scraping and AI automation tools. Use code SCRAPLING10: 10% off residential proxies.
README 图片 ProxyLane: 90% fewer CAPTCHAs on your Scrapling runs. 28M real P2P residential devices, 98.7% clean IPs, 195 countries. City/ISP/ZIP targeting, sticky/rotating sessions, traffic never expires. Try 350 MB for $1.95. Code SCRAPEANDTAKE: 25% off.
README 图片 LundProxy: Stop paying per GB. Unlimited residential traffic, 100M+ IPs, 190+ countries. Pay by bandwidth. Request a free 50 Mbps residential proxies trial.
README 图片 Resolvr: Scraping at scale means CAPTCHAs at scale. Resolvr solves reCAPTCHA and Cloudflare challenges on real devices through one API. Free beta access is available now.
README 图片 hCaptcha in your Scrapling runs? NoneCap returns valid hCaptcha tokens, Enterprise sitekeys included, through one API. 1,300 free credits on signup.
README 图片 $1/GB at DataImpulse. Ethical first-party proxies. 90M+ IPs, 195+ countries. Residential, datacenter, mobile. Pay-as-you-go = no subscription. Traffic never expires.

Do you want to show your ad here? Click here

Sponsors

README 图片


README 图片 README 图片 README 图片 README 图片 README 图片 README 图片

Do you want to show your ad here? Click here and choose the tier that suits you!


Key Features

Spiders - A Full Crawling Framework

  • 🕷️ Scrapy-like Spider API: Define spiders with start_urls, async parse callbacks, and Request/Response objects.
  • ⚡ Concurrent Crawling: Configurable concurrency limits, per-domain throttling, and download delays.
  • 🔄 Multi-Session Support: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
  • 💾 Pause & Resume: Save pending requests and seen URLs with crawldir. Press Ctrl+C to stop gracefully and restart with the same directory.
  • 📡 Streaming Mode: Stream scraped items as they arrive via async for item in spider.stream() with real-time stats - ideal for UI, pipelines, and long-running crawls.
  • 🛡️ Blocked Request Detection: Automatic detection and retry of blocked requests with customizable logic.
  • 🚦 AutoThrottle: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what Retry-After asks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops.
  • 🤖 Robots.txt Compliance: Optional robots_txt_obey flag that respects Disallow, Crawl-delay, and Request-rate directives with per-domain caching.
  • 🧪 Development Mode: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your parse() logic without re-hitting the target servers.
  • 🧩 Ready-made Spider Templates: Skip the boilerplate with CrawlSpider for rule-based link following, SitemapSpider for sitemap/robots.txt-driven crawls, XMLFeedSpider/CSVFeedSpider for iterating XML/RSS and CSV feeds, and ShopifySpider to collect products from public Shopify stores, one item per variant.
  • 🔗 Link Extraction: A standalone LinkExtractor primitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own.
  • 📦 Built-in Export: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with result.items.to_json(), to_jsonl(), to_csv(), and to_xml().

Advanced Websites Fetching with Session Support

  • HTTP Requests: Fast and stealthy HTTP requests with the Fetcher class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
  • Dynamic Loading: Fetch dynamic websites with full browser automation through the StealthyFetcher class supporting Playwright's Chromium and Google's Chrome.
  • Anti-bot Bypass: Advanced stealth capabilities with StealthyFetcher and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
  • Session Management: Persistent session support with FetcherSession and StealthySession classes for cookie and state management across requests.
  • Proxy Rotation: Built-in ProxyRotator with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
  • Domain & Ad Blocking: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
  • DNS Leak Prevention: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
  • Remote Browsers: Instead of launching a browser locally, connect to one that's already running through CDP with cdp_url, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build with executable_path.
  • Background API Capture: Enable record_requests=True on a browser session, then search its saved responses with session.network.search(limit=None). Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies.
  • Async Support: Complete async support across all fetchers and dedicated async session classes.

Parsing and Data Extraction

  • 🔄 Smart Element Tracking: Relocate elements after website changes using intelligent similarity algorithms.
  • 🎯 Smart Flexible Selection: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
  • 🔍 Find Similar Elements: Automatically locate elements similar to found elements.

AI Features

  • 🤖 MCP Server: Let AI chatbots and agents (Claude/Cursor/etc) scrape through Scrapling with HTTP requests, browser actions, screenshots, saved login state, and background API responses. Uses our stealthy browser with the right tools to enable suitable AI models to bypass any CAPTCHA is faces while working (not to mention our Cloudflare solver). It uses a lot of methods to reduce the content sent to the model, block ads, and protect it against Prompt-Injection. Remote browsers over CDP and HTTP transport with authentication options are included. (demo video)
  • 🧠 Agent Skill: A ready-to-install Agent Skill that teaches coding agents the whole library, so the code they write with Scrapling matches the current API instead of guessing.
  • 📚 RAG-ready Markdown: Turn any page into clean, sanitized, LLM-ready Markdown with one line (page.markdown()), or crawl a whole website into a Markdown corpus with the ContentSpider template, all without an LLM in the loop. (docs)

High-Performance & battle-tested Architecture

  • 🚀 Lightning Fast: Optimized performance outperforming most Python scraping libraries.
  • 🔋 Memory Efficient: Optimized data structures and lazy loading for a minimal memory footprint.
  • ⚡ Fast JSON Serialization: 10x faster than the standard library.
  • 🏗️ Battle tested: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.

Developer/Web Scraper Friendly Experience

  • 🎯 Interactive Web Scraping Shell: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
  • 🚀 Use it directly from the Terminal: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
  • 🛠️ Rich Navigation API: Advanced DOM traversal with parent, sibling, and child navigation methods.
  • 🧬 Enhanced Text Processing: Built-in regex, cleaning methods, and optimized string operations.
  • 📝 Auto Selector Generation: Generate robust CSS/XPath selectors for any element.
  • 🔌 Familiar API: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
  • 🤝 Drop-in Scrapy Integration: Already invested in Scrapy? Decorate any callback with scrapling_response to parse the responses you already fetch with Scrapling's parser, no rewrite needed.
  • 📘 Complete Type Coverage: Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with PyRight and MyPy with each change.
  • 🔋 Ready Docker image: With each release, a Docker image containing all browsers is automatically built and pushed.

Getting Started

Let's give you a quick glimpse of what Scrapling can do without deep diving.

Basic Usage

HTTP requests with session support

from scrapling.fetchers import Fetcher, FetcherSession

with FetcherSession(impersonate='chrome') as session:  # Use latest version of Chrome's TLS fingerprint
    page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
    quotes = page.css('.quote .text::text').getall()

# Or use one-off requests
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()

Advanced stealth mode

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, solve_cloudflare=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
    data = page.css('#padded_content a').getall()

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
data = page.css('#padded_content a').getall()

Full browser automation

from scrapling.fetchers import StealthyFetcher, StealthySession

with StealthySession(headless=True, disable_resources=False, network_idle=True) as session:  # Keep the browser open until you finish
    page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
    data = page.xpath('//span[@class="text"]/text()').getall()  # XPath selector if you prefer it

# Or use one-off request style, it opens the browser for this request, then closes it after finishing
page = StealthyFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()

Spiders

Build full crawlers with concurrent requests, multiple session types, and pause/resume:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10
    
    async def parse(self, response: Response):
        for quote in response.css('.quote'):
            yield {
                "text": quote.css('.text::text').get(),
                "author": quote.css('.author::text').get(),
            }
            
        next_page = response.css('.next a')
        if next_page:
            yield response.follow(next_page[0].attrib['href'])

result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")

Use multiple session types in a single spider:

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession

class MultiSessionSpider(Spider):
    name = "multi"
    start_urls = ["https://example.com/"]
    
    def configure_sessions(self, manager):
        manager.add("fast", FetcherSession(impersonate="chrome"))
        manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
    
    async def parse(self, response: Response):
        for link in response.css('a::attr(href)').getall():
            # Route protected pages through the stealth session
            if "protected" in link:
                yield Request(link, sid="stealth")
            else:
                yield Request(link, sid="fast", callback=self.parse)  # explicit callback

Pause and resume long crawls with checkpoints by running the spider like this:

QuotesSpider(crawldir="./crawl_data").start()

With crawldir set, press Ctrl+C to save pending requests and seen URLs. Restart with the same crawldir to resume. Save scraped items separately.

Or skip writing the crawling logic altogether with the ready-made templates, like pulling an entire Shopify store's catalog:

from scrapling.spiders import ShopifySpider

class MyStore(ShopifySpider):
    target_website = "example.com"

result = MyStore().start()  # Every product in the store, one item per variant

The template reads products exposed through public collection endpoints, one item per variant.

Advanced Parsing & Navigation

from scrapling.fetchers import Fetcher

# Rich element selection and navigation
page = Fetcher.get('https://quotes.toscrape.com/')

# Get quotes with multiple selection methods
quotes = page.css('.quote')  # CSS selector
quotes = page.xpath('//div[@class="quote"]')  # XPath
quotes = page.find_all('div', {'class': 'quote'})  # BeautifulSoup-style
# Same as
quotes = page.find_all('div', class_='quote')
quotes = page.find_all(['div'], class_='quote')
quotes = page.find_all(class_='quote')  # and so on...
# Find element by text content
quotes = page.find_by_text('Albert Einstein', first_match=False)

# Advanced navigation
quote_text = page.css('.quote')[0].css('.text::text').get()
quote_texts = page.css('.quote').css('.text::text').getall()  # Chained selectors
first_quote = page.css('.quote')[0]
author = first_quote.css('.author::text').get()
next_quote = first_quote.next
parent_container = first_quote.parent

# Element relationships and similarity
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements

You can use the parser right away if you don't want to fetch websites like below:

from scrapling.parser import Selector

page = Selector("<html>...</html>")

And it works precisely the same way!

Async Session Management Examples

import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession

async def main():
    async with FetcherSession(http3=True) as http_session:  # `FetcherSession` is context-aware and can work in both sync/async patterns
        page1 = await http_session.get('https://quotes.toscrape.com/')
        page2 = await http_session.get('https://quotes.toscrape.com/page/2/', impersonate='chrome')
        print(len(page1.css('.quote')), len(page2.css('.quote')))

    async with AsyncStealthySession(max_pages=2) as browser_session:
        urls = ['https://quotes.toscrape.com/js/', 'https://quotes.toscrape.com/js/page/2/']
        pages = await asyncio.gather(*(browser_session.fetch(url, wait_selector='.quote') for url in urls))
        print([len(page.css('.quote')) for page in pages])

asyncio.run(main())

CLI & Interactive Shell

Scrapling includes a powerful command-line interface:

asciicast

Launch the interactive Web Scraping shell

pip install "scrapling[shell]"
scrapling shell

Extract pages to a file directly without programming. Use --css-selector to pick the content, or --ai-targeted to select and sanitize the body content. Use .txt for text, .md for Markdown, or .html for HTML.

scrapling extract get 'https://quotes.toscrape.com/' quotes.md
scrapling extract get 'https://quotes.toscrape.com/' quotes.txt --css-selector '.quote' --impersonate 'chrome'
scrapling extract fetch 'https://quotes.toscrape.com/js/' quotes.md --css-selector '.quote' --wait-selector '.quote' --no-headless
scrapling extract fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare

[!NOTE] There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation here

Performance Benchmarks

Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.

Text Extraction Speed Test (5000 nested elements)

#LibraryTime (ms)vs Scrapling
1Scrapling1.991.0x
2Parsel/Scrapy2.061.035
3Raw Lxml2.561.286
4PyQuery23.98~12x
5Selectolax197.02~99x
6MechanicalSoup1545.15~776.5x
7BS4 with Lxml1562.1~785.0x
8BS4 with html5lib3412.73~1714.9x

Element Similarity & Text Search Performance

Scrapling's adaptive element finding capabilities significantly outperform alternatives:

LibraryTime (ms)vs Scrapling
Scrapling2.31.0x
AutoScraper12.585.47x

All benchmarks represent averages of 100+ runs. See benchmarks.py for methodology.

Installation

Scrapling requires Python 3.10 or higher:

pip install scrapling

[!IMPORTANT] This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from scrapling.fetchers or scrapling.spiders, like in the examples above, will raise ModuleNotFoundError with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.

Optional Dependencies

  1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:

    pip install "scrapling[fetchers]"
    
    scrapling install           # normal install
    scrapling install  --force  # force reinstall
    

    This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.

    Or you can install them from the code instead of running a command like this:

    from scrapling.cli import install
    
    install([], standalone_mode=False)          # normal install
    install(["--force"], standalone_mode=False) # force reinstall
    
  2. Extra features:

    • Install the MCP server feature:
      pip install "scrapling[ai]"
      
    • Install dependencies for (building RAG systems):
      pip install "scrapling[rag]"
      
    • Install shell features (Web Scraping shell and the extract command):
      pip install "scrapling[shell]"
      
    • Install everything:
      pip install "scrapling[all]"
      

    Run scrapling install when you need browser fetchers. HTTP requests do not need a browser installation.

Docker

You can also install a Docker image with all extras and browsers with the following command from DockerHub:

docker pull pyd4vinci/scrapling

Or download it from the GitHub registry:

docker pull ghcr.io/d4vinci/scrapling:latest

This image is automatically built and pushed using GitHub Actions and the repository's main branch.

Contributing

We welcome contributions! Please read our contributing guidelines before getting started.

Disclaimer

[!CAUTION] This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.

🎓 Citations

If you have used our library for research purposes please quote us with the following reference:

  @misc{scrapling,
    author = {Karim Shoair},
    title = {Scrapling},
    year = {2024},
    url = {https://github.com/D4Vinci/Scrapling},
    note = {A Python Web Scraping framework for everything from a single request to a full-scale crawl!}
  }

License

This work is licensed under the BSD-3-Clause License.

Acknowledgments

This project includes code adapted from:

  • Parsel (BSD License)-Used for translator submodule

Designed & crafted with ❤️ by Karim Shoair.