# Chapter 8 Labs: AI and Machine Learning for Security Practitioners

This chapter's labs are Python/ML exercises. Rather than attacking a network target,
you train classifiers, visualise neural-network internals, and drive a local language
model through a tool-calling loop. All three labs reinforce the chapter's central
argument: AI literacy is about owning the numbers, not just consuming the output.

The compose environment provides a Jupyter server (scikit-learn + transformers +
bertviz) for Labs 08a and 08b, plus a tiny nginx endpoint for Lab 08c's tool-calling
exercise. Lab 08c (Ollama) runs on the host because Ollama requires direct GPU/CPU
access and manages model weights outside a container.

---

## Prerequisites

- Docker and Docker Compose (v2+)
- Free disk space: approx 4 GB for the ml-tools image build (PyTorch CPU wheel is large)
- Internet access during `docker compose up` (first build only - pulls base image and Python packages)
- For Lab 08c (Ollama - host setup): approx 8-16 GB disk for model weights

---

## Quick Start

```bash
cd /path/to/labs/ch08
docker compose up -d
```

- Jupyter (Labs 08a, 08b): http://localhost:8888
- Lab data endpoint (Lab 08c): http://localhost:8080/lab-data.json

Copy the token from `docker compose logs ml-tools` if Jupyter prompts for one.

---

## Lab 08a: Train a Classifier and Break the Accuracy Myth

**Objective:** Train four classical ML classifiers on the UCI Phishing Websites
dataset, score them properly with per-class precision/recall/F1, then reproduce the
"accuracy trap" by down-sampling phishing to a realistic 1% base rate and watching a
do-nothing DummyClassifier score 99% accuracy while catching zero phishing sites.

**Setup:** The `ml-tools` container provides the full environment. The notebook is
pre-loaded at `/lab/notebooks/08a_phishing_classifier.py`. Toolchain pins:
scikit-learn 1.8.0, ucimlrepo, pandas.

**Walkthrough:**

1. Open Jupyter at http://localhost:8888 and open `08a_phishing_classifier.py`
   (or run `docker compose exec ml-tools python notebooks/08a_phishing_classifier.py`).
2. The script fetches the dataset from UCI directly (CC BY 4.0, 11,055 rows x 30
   features): `from ucimlrepo import fetch_ucirepo; ds = fetch_ucirepo(id=327)`.
3. It performs a stratified train/validation/test split (approx 7,700 / 1,650 / 1,650)
   - the test set is touched exactly once, at the end.
4. Four classifiers are fitted: `DecisionTreeClassifier`, `RandomForestClassifier`,
   `GradientBoostingClassifier`, `SVC`. Each trains in under two seconds on this data.
5. Scores are printed as per-class precision, recall, and F1 (never bare accuracy).
   Expected: the lone decision tree near 0.96 F1, ensembles edging ahead to 0.97-0.98.
6. Phishing is down-sampled to approx 1% to mirror real-world prevalence. A
   `DummyClassifier(strategy='most_frequent')` is then scored: watch it report ~99%
   accuracy with 0.0% recall on phishing. A RandomForest with `class_weight='balanced'`
   recovers real recall on the same imbalanced data.

**Expected output:**

```
-- RandomForest --
              precision    recall  f1-score
  legitimate      0.98      0.97      0.97
    phishing      0.97      0.98      0.98

DummyClassifier (always predict 'legitimate'):
    phishing      0.00      0.00      0.00   <-- accuracy ~99%, recall zero
```

**Verify completion:** The script prints "Real recall recovered - this is the lesson
of Lab 08a." at the end. The RandomForest on imbalanced data should show phishing
recall > 0.50 (class_weight='balanced' lifts it substantially).

**Cleanup:** `docker compose down` - the container is stateless.

---

## Lab 08b: See Attention for Yourself

**Objective:** Load a small BERT model with `output_attentions=True`, run a single
forward pass, and visualise the Q * K^T attention weights with bertviz. Find the
coreference: "the server refused the request because it was overloaded" - watch
which token "it" attends to most in each layer.

**Setup:** The `ml-tools` container includes transformers 5.8.0 and bertviz 1.4.1
(pinned to 11 May 2026 access date). The notebook is at `08b_attention_viz.py`.
First run downloads bert-base-uncased weights (~440 MB) into the `ml-data-cache`
volume - they are cached on subsequent runs.

**Walkthrough:**

1. Open Jupyter at http://localhost:8888 and open `08b_attention_viz.py`.
2. Step 1 loads `bert-base-uncased` with `output_attentions=True`.
3. Step 2 tokenises the coreference sentence and runs a single forward pass
   (no training, no gradient computation).
4. Step 3 calls `bertviz.head_view(attention, tokens)` and
   `bertviz.model_view(attention, tokens)`. In Jupyter these render as interactive
   HTML widgets: each cell in the heat-map is one weight from the softmax over the
   key-query dot products (the Q * K^T grid made visible).
5. Step 4 inspects the coreference numerically: for each layer it finds the token
   that "it" attends to most strongly (averaged across heads). In deeper layers
   this should bind "it" to "server."

**Expected output (script mode):**

```
Tokens: ['[CLS]', 'the', 'server', 'refused', 'the', 'request', 'because', 'it', 'was', 'overloaded', '.', '[SEP]']
Layer  0: 'it' -> '[CLS]' (weight=0.xxx)
...
Layer 11: 'it' -> 'server' (weight=0.xxx)
```

The exact layer where "it" -> "server" dominates varies with random initialisation
and BERT version, but by the middle-to-late layers coreference resolution is visible.
The interactive `head_view` in Jupyter makes this immediately legible.

**Verify completion:** The script prints layer-by-layer attention targets without
errors, and at least one layer shows "it" attending to "server" or a near-synonym.

**Cleanup:** `docker compose down` - bert weights remain in the `ml-data-cache`
volume; run `docker compose down -v` to remove them.

---

## Lab 08c (Part 1): Run a Model Locally with Ollama

Host setup (not containerised here). Ollama requires direct CPU/GPU access and
manages model weights as local files outside Docker.

**Objective:** Install Ollama, pull three small open models (llama3.2, mistral,
qwen2.5), run a security-analyst prompt against all three from the CLI, then call
the local API from Python. The privacy case: the sensitive prompt is processed
locally and leaves the host zero times.

**Upstream project:** https://ollama.com

**Setup and walkthrough:**

1. Install Ollama v0.23.2 from https://ollama.com/download (pin the version to
   reproduce book results; later releases may behave differently).
   ```bash
   ollama --version   # should print 0.23.2
   ```

2. Pull three small open models (each is approx 2-4 GB):
   ```bash
   ollama pull llama3.2
   ollama pull mistral
   ollama pull qwen2.5
   ```

3. Run a security-analyst prompt from the CLI (data never leaves the host):
   ```bash
   ollama run llama3.2 "Summarise the key risks in this nginx config: server { listen 80; ... }"
   ```

4. Call the local API from Python:
   ```python
   import requests
   resp = requests.post("http://localhost:11434/api/chat", json={
       "model": "llama3.2",
       "messages": [{"role": "user", "content": "Name three common web app vulnerabilities."}],
       "stream": False,
   })
   print(resp.json()["message"]["content"])
   ```

5. Compare all three models on the same prompt. Rate the answers. Because all three
   run locally, the sensitive prompt is processed three times and leaves the host
   zero times - the concrete privacy contrast with a cloud API.

**Verify completion:** All three models produce a coherent answer to the security
prompt. `ollama list` shows all three models downloaded.

---

## Lab 08c (Part 2): Give a Model Two Tools, Then Watch It (ReAct Loop)

**Objective:** Define a `calculator` tool and a `fetch_url` tool, ask the model a
question that requires both (fetch a number, then compute from it), and observe the
ReAct loop - thought, action, observation, repeat. Note where the model hesitates,
loops, or calls a tool it did not need.

**Setup:** Ollama must be running from Part 1. The `lab-data` service in compose
provides the local JSON endpoint the `fetch_url` tool targets.

```bash
docker compose up -d lab-data
```

Then run:

```bash
docker compose exec ml-tools python notebooks/08c_tool_calling.py
```

Or on the host (with `pip install requests`):

```bash
python notebooks/08c_tool_calling.py
```

**Walkthrough:**

1. The script defines two tools for the Ollama model: `calculator` (evaluates a
   safe arithmetic expression) and `fetch_url` (fetches a URL from an allow-list).
2. The prompt asks the model to fetch scan data from
   `http://localhost:8080/lab-data.json`, then compute the hit rate as hits per second.
3. The model issues tool calls; the script prints each call and observation.
4. Observe the ReAct pattern: the model thinks, calls a tool, receives an observation,
   and thinks again. Note where it hesitates or makes an unexpected call.

**Authorisation guard rail (mandatory):** The `fetch_url` tool is restricted to
`http://localhost:8080/lab-data.json` and `http://127.0.0.1:8080/lab-data.json` only.
Do NOT expand the allow-list to arbitrary internet addresses. Only test against the
provided lab target or a system you own and are authorised to test. An agent with
unrestricted network reach can be driven by injected text in a tool result into
real-world action.

**Expected output:**

```
[Turn 1] TOOL CALL: fetch_url({'url': 'http://localhost:8080/lab-data.json'})
[Turn 1] OBSERVATION: {"port_scan_hits": 47, ...}
[Turn 2] TOOL CALL: calculator({'expression': '47 / 30'})
[Turn 2] OBSERVATION: 1.5666...
[Turn 3] FINAL ANSWER: The scan hit rate is approximately 1.57 hits per second.
```

**Verify completion:** The model reaches a FINAL ANSWER and the hit rate calculation
is correct (47 / 30 = 1.57 hits/sec). The fetch_url tool must NOT have been called
with any URL outside the allow-list.

**Cleanup:**

```bash
docker compose down -v
```

This stops all containers and removes the ml-data-cache volume (bert weights). To
keep the cache for future runs, use `docker compose down` without `-v`.

---

## Teardown (All Labs)

```bash
docker compose down -v
```

This stops and removes all containers and named volumes. Ollama model weights (host
setup) must be removed manually if desired:

```bash
ollama rm llama3.2
ollama rm mistral
ollama rm qwen2.5
```

---

## Safety and Authorisation Note

These labs do NOT involve attacking real infrastructure. The only "offensive" element
is the ReAct tool-calling lab, where the `fetch_url` tool is explicitly restricted
to a local allow-list. No real malware, no real PCAPs, no real victim data are used
anywhere in this chapter's labs. The phishing dataset (UCI id=327) contains
feature-engineered numeric vectors, not actual URLs or page content.

All tools in Lab 08c must only be pointed at the provided lab target
(`localhost:8080`) or a system you own and are authorised to test.
