CS Logo

Python Coding Camp

Day 2 — Web Scraping & APIs

Functions · Lists · Libraries · HTTP Requests · JSON

⬇ Today's slides (PDF)

Schedule

TimeActivity
0:00 – 0:45Lecture: what is the web? functions, lists, importing libraries
0:45 – 1:30Guided: fetch data from your chosen track, print results
1:30 – 1:40Break
1:40 – 3:00Open project: explore and extend

Quick Reference — New Things Today

Functions

# def defines a new function — give it a name and list the info it needs (parameters)
def greet(name):              # "name" is a parameter — a variable the caller fills in
    print(f"Hello, {name}!")  # use the parameter inside the function body

greet("Alice")   # call the function — "Alice" gets assigned to name
greet("Bob")     # call it again with different input — the function runs again

Lists

# A list holds multiple values in order, separated by commas
fruits = ["apple", "banana", "cherry"]

print(fruits[0])       # apple — positions start at 0, not 1!
print(fruits[1])       # banana
print(len(fruits))     # 3 — len() counts how many items are in the list

for fruit in fruits:   # loop through every item one at a time
    print(fruit)

fruits.append("mango")  # add "mango" to the end of the list

Installing and Importing Libraries

# Run this in the terminal FIRST — installs the library once:
# pip install requests

import requests   # now the library is available in your script

# .get() sends a request to a URL, like typing it into your browser
response = requests.get("https://example.com")
print(response.status_code)  # 200 means OK, 404 means not found, 500 means server error
print(response.text)         # the HTML or JSON content that came back

JSON (data format used by APIs)

import requests

response = requests.get("https://some-api.com/data")

# APIs send data as JSON text — .json() converts it into a Python dictionary
data = response.json()

# Access values by their key name, just like a dictionary
print(data["name"])    # get the "name" field
print(data["score"])   # get the "score" field

Pick Your Track

Choose one of the three options below. They all teach the same concepts — pick what sounds most interesting to you.

Track A

Books to Scrape — HTML Scraping

Scrape book titles and prices from a fake bookstore site built for practicing. You'll use BeautifulSoup to read the HTML.

Install: pip install requests beautifulsoup4

import requests
from bs4 import BeautifulSoup   # BeautifulSoup reads and navigates HTML

url = "https://books.toscrape.com"
response = requests.get(url)   # download the webpage (same as your browser would)

# Parse the HTML into a structure we can search — "html.parser" is built into Python
soup = BeautifulSoup(response.text, "html.parser")

# find_all() returns a list of every HTML element matching that tag and CSS class
# We use class_= (with underscore) because "class" is a reserved word in Python
books = soup.find_all("article", class_="product_pod")

for book in books:
    title = book.h3.a["title"]                        # navigate the HTML tree to get the title attribute
    price = book.find("p", class_="price_color").text  # .text strips the HTML tags, leaving just the text
    print(f"{title}: {price}")
Tip: Right-click the bookstore website in your browser and choose "Inspect" to see the HTML structure. That's what BeautifulSoup is reading.
Track B

PokeAPI — Pokémon Data

Fetch Pokémon stats directly from the official Pokémon API. No login, no key, completely free.

Install: pip install requests

import requests

# .lower() converts the input to lowercase — the API requires lowercase names
name = input("Enter a Pokémon name (e.g. pikachu): ").lower()

# Build the URL with the Pokémon name plugged in using an f-string
url = f"https://pokeapi.co/api/v2/pokemon/{name}"
response = requests.get(url)

# status_code 200 means success; 404 means the Pokémon wasn't found
if response.status_code == 200:
    data = response.json()   # convert the JSON response to a Python dictionary

    print(f"\n{data['name'].capitalize()}")        # .capitalize() makes first letter uppercase
    print(f"  Height: {data['height'] / 10} m")   # API gives height in decimeters; divide by 10 for meters
    print(f"  Weight: {data['weight'] / 10} kg")  # API gives weight in hectograms; divide by 10 for kg

    print("\n  Stats:")
    for stat in data["stats"]:             # data["stats"] is a list of stat objects
        stat_name  = stat["stat"]["name"]  # each stat object has a nested dict with the name
        stat_value = stat["base_stat"]     # and a "base_stat" number
        print(f"    {stat_name}: {stat_value}")
else:
    print("Pokémon not found! Check the spelling.")
Tip: Try fetching multiple Pokémon in a loop using for name in ["pikachu", "charizard", "mewtwo"]:
Track C

NASA — Astronomy Picture of the Day

Fetch today's NASA Astronomy Picture of the Day. Get a free API key instantly at api.nasa.gov (no credit card needed).

Install: pip install requests

import requests

# APIs use keys to track who is making requests — get a free one at api.nasa.gov
# DEMO_KEY works for testing but has a low daily limit
API_KEY = "DEMO_KEY"

# The ? in the URL starts the "query string" — extra parameters sent with the request
url = f"https://api.nasa.gov/planetary/apod?api_key={API_KEY}"
response = requests.get(url)
data = response.json()   # converts the JSON response into a Python dictionary

# Pull values out of the dictionary using their key names
print(f"Title: {data['title']}")        # the name of today's image
print(f"Date:  {data['date']}")         # date in YYYY-MM-DD format
print(f"URL:   {data['url']}")          # direct link to the image
print()
print(data['explanation'])              # paragraph from NASA explaining the image
Tip: You can use DEMO_KEY to start without signing up. It has a low rate limit but works for testing. Get a real key to do more.

Stretch Goals (all tracks)

  1. Save your results to a text file using with open("results.txt", "w") as f: — the with form closes the file for you automatically
  2. Fetch multiple items — scrape page 2 of the bookstore, loop through 10 Pokémon, fetch yesterday's NASA APOD
  3. Find the most expensive book, the Pokémon with the highest Attack stat, or the oldest NASA APOD you can find
  4. Let the user type what they want to search for, and keep asking until they type "quit"
  5. Store all results in a list, then sort it and print the top 5
  6. Boss challenge: Save your data to a CSV file (use Python's built-in csv module). You'll be able to open it in Excel!

More Project Options — Pick Any

The three tracks above aren't the only choices. Any project here uses the same skills — requests, a URL, and reading fields out of the response with obj[key]. All the APIs below are free and need no key. The Boss ones combine or save data.

Starter — one request, print the result

Starter

Weather Lookup (Open-Meteo)

Type a latitude/longitude and get the current temperature. No key needed.

import requests

url = "https://api.open-meteo.com/v1/forecast?latitude=42.3&longitude=-122.9¤t_weather=true"
data = requests.get(url).json()
print(f"Temperature: {data['current_weather']['temperature']} °C")
Starter

Dad Joke Fetcher

Ask icanhazdadjoke.com for a random joke.

import requests

headers = {"Accept": "application/json"}   # ask the API for JSON, not a web page
data = requests.get("https://icanhazdadjoke.com/", headers=headers).json()
print(data["joke"])
Starter

Bored? Activity Suggester

Ask the Bored API for something to do when you can't think of anything.

import requests

data = requests.get("https://bored-api.appbrewery.com/random").json()
print("You could:", data["activity"])

Standard — a loop, a list, or a bit more logic

Standard

Quotes Scraper

Sister site to the bookstore: quotes.toscrape.com. Scrape each quote and its author with BeautifulSoup (same pattern as Track A).

Standard

Trivia Quiz with Score

Pull questions from the Open Trivia DB, ask the user each one, and keep score with a loop and an if — ties right back to Day 1.

Standard

Crypto Price Checker

Let the user pick a coin and print its live price from the free CoinGecko API.

import requests

coin = input("Coin id (e.g. bitcoin, ethereum, dogecoin): ")
url  = f"https://api.coingecko.com/api/v3/simple/price?ids={coin}&vs_currencies=usd"
data = requests.get(url).json()
print(f"{coin} = ${data[coin]['usd']}")

Boss Challenge — combine, save, or automate

Boss

Two-API Mashup

Combine two sources into one program — e.g. look up a city's weather and a random activity, then suggest an indoor or outdoor plan based on the temperature.

Boss

Scrape Every Page to a CSV

Loop through all pages of the bookstore or quotes site, collect every item into a list, and save it to a CSV with Python's built-in csv module. You'll open it in Day 3.