Python Libraries Used in Web Scraping

framework

scrapy
46,263 2.8.0 (22 days ago) Jul 26 2019
scrapyd
2,598 1.4.1 (14 days ago) Sep 04 2013
autoscraper
4,911 1.1.14 (7 months ago) Jul 26 2019
gracy
156 1.11.1 (9 hours ago) Feb 05
scrapydweb
2,671 1.4.0 (3 years ago) Sep 30 2018
ruia
1,642 0.8.5 (5 months ago) Oct 17 2018
photon
9,417 1.1.9 (4 years ago) Aug 24 2018
gerapy
2,914 0.9.12 (a month ago) Jul 04 2017
dude
341 0.1.2 (2 years ago) Feb 20 2022

scrapy

scrapy
46,263 2.8.0 (22 days ago) Jul 26 2019
scrapyd
2,598 1.4.1 (14 days ago) Sep 04 2013
scrapydweb
2,671 1.4.0 (3 years ago) Sep 30 2018
gerapy
2,914 0.9.12 (a month ago) Jul 04 2017
splash
3,778 3.5 (2 years ago) Apr 25 2014

html-parser

lxml
2,289 4.9.2 (2 months ago) Dec 13
beautifulsoup
- 4.11.2 (23 days ago) Jul 26 2019
xmltodict
5,059 0.13.0 (9 months ago) Jul 30 2007
html5lib
1,010 1.1 (2 years ago) Jul 30 2007
cssselect
268 1.2.0 (3 months ago) Apr 14 2012
feedparser
1,550 6.0.10 (9 months ago) Jun 15 2007
pyquery
2,183 2.0.0 (a month ago) Dec 05 2008
parsel
884 1.7.0 (3 months ago) Jul 26 2019
requests-html
12,968 0.10.0 (4 years ago) Feb 25 2018
untangle
579 1.2.1 (7 months ago) Jun 09 2011
html5-parser
642 0.4.10 (1 year, 5 months ago) Jun 03 2007
selectolax
750 0.3.12 (a month ago) Mar 01 2018
gazpacho
698 1.1 (2 years ago) Dec 28 2012
chopper
22 0.5.0 (7 months ago) Jul 24 2014

http-client

requests
49,097 2.28.2 (a month ago) Feb 14 2011
aiohttp
13,292 3.8.4 (11 days ago) Jul 26 2019
httpx
10,106 0.23.3 (a month ago) Jul 26 2019
pycurl
949 7.45.2 (2 months ago) Feb 25 2003
treq
552 22.2.0 (1 year, 15 days ago) Dec 28 2012

html-extractor

html2text
1,357 2020.1.16 (3 years ago) Dec 14 2008
trafilatura
783 1.4.1 (a month ago) Jul 17 2019
readability
2,222 0.8.1 (2 years ago) Jun 30 2011
newspaper
12,488 0.2.8 (4 years ago) Dec 28 2012
extruct
733 0.14.0 (4 months ago) Oct 27 2015
sumy
3,063 0.11.0 (4 months ago) Oct 20 2013
photon
9,417 1.1.9 (4 years ago) Aug 24 2018
extractnet
74 2.0.7 (3 months ago) Dec 11 2020

gui

scrapydweb
2,671 1.4.0 (3 years ago) Sep 30 2018
gerapy
2,914 0.9.12 (a month ago) Jul 04 2017

http-utils

httptools
1,088 0.5.0 (5 months ago) Dec 28 2012
requests-cache
1,077 0.9.8 (a month ago) Feb 14 2011

educational

youtube-dl
117,964 2021.12.17 (1 year, 2 months ago) Feb 22 2012
cloudscraper
2,748 1.2.68 (a month ago) Dec 28 2012
you-get
46,721 0.4.1650 (2 months ago) Sep 01 2012

scraper

youtube-dl
117,964 2021.12.17 (1 year, 2 months ago) Feb 22 2012
trafilatura
783 1.4.1 (a month ago) Jul 17 2019
you-get
46,721 0.4.1650 (2 months ago) Sep 01 2012

cli-tool

youtube-dl
117,964 2021.12.17 (1 year, 2 months ago) Feb 22 2012
you-get
46,721 0.4.1650 (2 months ago) Sep 01 2012

captcha

captcha_solver
205 0.1.5 (3 years ago) Feb 17 2015
unicaps
124 1.2.0 (4 months ago) Jul 27 2020

data

jsonpath-ng
382 1.5.3 (1 year, 7 months ago) Feb 09 2022
nested-lookup
209 0.2.25 (7 months ago) Feb 09 2022
xhtml2pdf
1,983 0.2.9 (24 days ago) May 19 2011

data-output

xhtml2pdf
1,983 0.2.9 (24 days ago) May 19 2011

html-utils

bleach
2,482 6.0.0 (a month ago) Feb 21 2010

microformats

extruct
733 0.14.0 (4 months ago) Oct 27 2015

blocking

cloudscraper
2,748 1.2.68 (a month ago) Dec 28 2012

playwright

playwright
8,079 1.31.0 (2 days ago) Feb 24 2021

browser-automation

selenium
25,889 4.8.2 (6 days ago) Apr 25 2008
playwright
8,079 1.31.0 (2 days ago) Feb 24 2021
requestium
1,716 0.1.15 (2 months ago) Dec 28 2012
splash
3,778 3.5 (2 years ago) Apr 25 2014

selenium

selenium
25,889 4.8.2 (6 days ago) Apr 25 2008
requestium
1,716 0.1.15 (2 months ago) Dec 28 2012

text-parsing

sumy
3,063 0.11.0 (4 months ago) Oct 20 2013

css

cssutils
49 2.6.0 (6 months ago) Aug 04 2007

seo

advertools
770 0.13.2 (4 months ago) Dec 28 2012

json-parser

jsonpath-ng
382 1.5.3 (1 year, 7 months ago) Feb 09 2022
nested-lookup
209 0.2.25 (7 months ago) Feb 09 2022

jsonpath

jsonpath-ng
382 1.5.3 (1 year, 7 months ago) Feb 09 2022

scrapers

crosslinked
709 0.2.0 (5 months ago) Feb 14 2011