🕷️
STUDENT NOTES · DAY 21 OF 24
The Friendly Spider 🕷️
Every web page is data wearing a costume called HTML. Today you learn to unwrap it — politely.
1HTML is a tree of tags
🐍
Py: Tags inside tags inside tags — a family tree. BeautifulSoup climbs it for you.
2Soup time 🍜
🤖
Bittu: requests DOWNLOADS the page (Day 15 skill). BeautifulSoup PARSES it. Two tools, one heist — a legal one.
3find_all — harvest everything
4Scrape POLITELY 🙏
🦉
Professor Hoot: Check robots.txt (the site's rules for bots), add delays with time.sleep, and never hammer a server. Good spiders are welcome; rude ones get blocked.
🐞
Bugly: Scrape a site 1000 times a second and they'll block your IP. Then who's laughing? …me. Always me.
5Quick quiz! 🧠
The right order for scraping is…
💡 Download first, parse second. Soup can't parse what you haven't fetched.
robots.txt tells you…
💡 It's the site's rulebook for crawlers — read it, respect it.
6Your mission 🚀
Mission 1: Paste the class demo HTML into a string and extract the h1 with soup.find.
Mission 2: Print every link's text AND its href from the demo page.
Mission 3: Visit any site's /robots.txt in your browser and read what it allows. Detective work!
Day 21 done — friendly spider certified! 🕷️
Next: automation — code that does your chores while you sleep.
Go to Day 22: Automation →