Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Distributed Web Scraping in Python

PyCon US via YouTube

Start learning Write review
DataCamp Ad

PowerBI Data Analyst - Create visualizations and dashboards from scratch

Learn More → Boot.dev Ad

The Most Addictive Python and SQL Courses

Learn More →

Overview

Google, IBM & Meta Certificates – 40% Off
One Coursera Plus subscription covers most Professional Certificates on Coursera.
Unlock All Certificates
This talk shows how to build a distributed web scraper in Python for large batch jobs, focusing on reducing run times and improving code durability. It covers the system’s components, queues, message brokers, and lessons learned.

Syllabus

Intro
Outline
Introduction
Data Science Project Stages
What is Distributed Web Scraping
Setting the Stage
Iteration - A Single Request
Looping Requests
Iteration 1 - Issues
Intermediate improvements
Iteration 2 - Issues
Distributed - Mental Model
Distributed - Controller
Distributed - Scraping Node
Distributed - Advantages
Distributed - Disadvantages
Distributed - Queues
Distributed - Message Brokers
Code Management
Useful Python Packages
Considerations
Conclusion

Taught by

PyCon US

Reviews

Start your review of Distributed Web Scraping in Python

Start learning

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Sign up for free
Someone learning on their laptop while sitting on the floor.