Day 2 — Web Scraping & APIs
Functions · Lists · Libraries · HTTP Requests · JSON
⬇ Today's slides (PDF)Schedule
| Time | Activity |
|---|---|
| 0:00 – 0:45 | Lecture: what is the web? functions, lists, importing libraries |
| 0:45 – 1:30 | Guided: fetch data from your chosen track, print results |
| 1:30 – 1:40 | Break |
| 1:40 – 3:00 | Open project: explore and extend |
Quick Reference — New Things Today
Functions
# def defines a new function — give it a name and list the info it needs (parameters)
def greet(name): # "name" is a parameter — a variable the caller fills in
print(f"Hello, {name}!") # use the parameter inside the function body
greet("Alice") # call the function — "Alice" gets assigned to name
greet("Bob") # call it again with different input — the function runs again
Lists
# A list holds multiple values in order, separated by commas
fruits = ["apple", "banana", "cherry"]
print(fruits[0]) # apple — positions start at 0, not 1!
print(fruits[1]) # banana
print(len(fruits)) # 3 — len() counts how many items are in the list
for fruit in fruits: # loop through every item one at a time
print(fruit)
fruits.append("mango") # add "mango" to the end of the list
Installing and Importing Libraries
# Run this in the terminal FIRST — installs the library once:
# pip install requests
import requests # now the library is available in your script
# .get() sends a request to a URL, like typing it into your browser
response = requests.get("https://example.com")
print(response.status_code) # 200 means OK, 404 means not found, 500 means server error
print(response.text) # the HTML or JSON content that came back
JSON (data format used by APIs)
import requests
response = requests.get("https://some-api.com/data")
# APIs send data as JSON text — .json() converts it into a Python dictionary
data = response.json()
# Access values by their key name, just like a dictionary
print(data["name"]) # get the "name" field
print(data["score"]) # get the "score" field
Pick Your Track
Choose one of the three options below. They all teach the same concepts — pick what sounds most interesting to you.
Books to Scrape — HTML Scraping
Scrape book titles and prices from a fake bookstore site built for practicing. You'll use BeautifulSoup to read the HTML.
Install: pip install requests beautifulsoup4
import requests
from bs4 import BeautifulSoup # BeautifulSoup reads and navigates HTML
url = "https://books.toscrape.com"
response = requests.get(url) # download the webpage (same as your browser would)
# Parse the HTML into a structure we can search — "html.parser" is built into Python
soup = BeautifulSoup(response.text, "html.parser")
# find_all() returns a list of every HTML element matching that tag and CSS class
# We use class_= (with underscore) because "class" is a reserved word in Python
books = soup.find_all("article", class_="product_pod")
for book in books:
title = book.h3.a["title"] # navigate the HTML tree to get the title attribute
price = book.find("p", class_="price_color").text # .text strips the HTML tags, leaving just the text
print(f"{title}: {price}")
PokeAPI — Pokémon Data
Fetch Pokémon stats directly from the official Pokémon API. No login, no key, completely free.
Install: pip install requests
import requests
# .lower() converts the input to lowercase — the API requires lowercase names
name = input("Enter a Pokémon name (e.g. pikachu): ").lower()
# Build the URL with the Pokémon name plugged in using an f-string
url = f"https://pokeapi.co/api/v2/pokemon/{name}"
response = requests.get(url)
# status_code 200 means success; 404 means the Pokémon wasn't found
if response.status_code == 200:
data = response.json() # convert the JSON response to a Python dictionary
print(f"\n{data['name'].capitalize()}") # .capitalize() makes first letter uppercase
print(f" Height: {data['height'] / 10} m") # API gives height in decimeters; divide by 10 for meters
print(f" Weight: {data['weight'] / 10} kg") # API gives weight in hectograms; divide by 10 for kg
print("\n Stats:")
for stat in data["stats"]: # data["stats"] is a list of stat objects
stat_name = stat["stat"]["name"] # each stat object has a nested dict with the name
stat_value = stat["base_stat"] # and a "base_stat" number
print(f" {stat_name}: {stat_value}")
else:
print("Pokémon not found! Check the spelling.")
for name in ["pikachu", "charizard", "mewtwo"]:
NASA — Astronomy Picture of the Day
Fetch today's NASA Astronomy Picture of the Day. Get a free API key instantly at api.nasa.gov (no credit card needed).
Install: pip install requests
import requests
# APIs use keys to track who is making requests — get a free one at api.nasa.gov
# DEMO_KEY works for testing but has a low daily limit
API_KEY = "DEMO_KEY"
# The ? in the URL starts the "query string" — extra parameters sent with the request
url = f"https://api.nasa.gov/planetary/apod?api_key={API_KEY}"
response = requests.get(url)
data = response.json() # converts the JSON response into a Python dictionary
# Pull values out of the dictionary using their key names
print(f"Title: {data['title']}") # the name of today's image
print(f"Date: {data['date']}") # date in YYYY-MM-DD format
print(f"URL: {data['url']}") # direct link to the image
print()
print(data['explanation']) # paragraph from NASA explaining the image
DEMO_KEY to start without signing up. It has a low rate limit but works for testing. Get a real key to do more.
Stretch Goals (all tracks)
- Save your results to a text file using
with open("results.txt", "w") as f:— thewithform closes the file for you automatically - Fetch multiple items — scrape page 2 of the bookstore, loop through 10 Pokémon, fetch yesterday's NASA APOD
- Find the most expensive book, the Pokémon with the highest Attack stat, or the oldest NASA APOD you can find
- Let the user type what they want to search for, and keep asking until they type "quit"
- Store all results in a list, then sort it and print the top 5
- Boss challenge: Save your data to a CSV file (use Python's built-in
csvmodule). You'll be able to open it in Excel!
More Project Options — Pick Any
The three tracks above aren't the only choices. Any project here uses the same skills — requests, a URL, and reading fields out of the response with obj[key]. All the APIs below are free and need no key. The Boss ones combine or save data.
Starter — one request, print the result
Weather Lookup (Open-Meteo)
Type a latitude/longitude and get the current temperature. No key needed.
import requests
url = "https://api.open-meteo.com/v1/forecast?latitude=42.3&longitude=-122.9¤t_weather=true"
data = requests.get(url).json()
print(f"Temperature: {data['current_weather']['temperature']} °C")
Dad Joke Fetcher
Ask icanhazdadjoke.com for a random joke.
import requests
headers = {"Accept": "application/json"} # ask the API for JSON, not a web page
data = requests.get("https://icanhazdadjoke.com/", headers=headers).json()
print(data["joke"])
Bored? Activity Suggester
Ask the Bored API for something to do when you can't think of anything.
import requests
data = requests.get("https://bored-api.appbrewery.com/random").json()
print("You could:", data["activity"])
Standard — a loop, a list, or a bit more logic
Quotes Scraper
Sister site to the bookstore: quotes.toscrape.com. Scrape each quote and its author with BeautifulSoup (same pattern as Track A).
Trivia Quiz with Score
Pull questions from the Open Trivia DB, ask the user each one, and keep score with a loop and an if — ties right back to Day 1.
Crypto Price Checker
Let the user pick a coin and print its live price from the free CoinGecko API.
import requests
coin = input("Coin id (e.g. bitcoin, ethereum, dogecoin): ")
url = f"https://api.coingecko.com/api/v3/simple/price?ids={coin}&vs_currencies=usd"
data = requests.get(url).json()
print(f"{coin} = ${data[coin]['usd']}")
Boss Challenge — combine, save, or automate
Two-API Mashup
Combine two sources into one program — e.g. look up a city's weather and a random activity, then suggest an indoor or outdoor plan based on the temperature.
Scrape Every Page to a CSV
Loop through all pages of the bookstore or quotes site, collect every item into a list, and save it to a CSV with Python's built-in csv module. You'll open it in Day 3.