Working with AI

AI Engineering

4 lessons · ~25 min total

The jagged frontier: what LLMs can and cannot do

~6 min

Build a working mental model of LLM capability so you judge where to apply them by testing, not by hype or vibes.

Key takeaways

LLMs optimize for plausible, not true: hallucination is the mechanism working, so design around it.
Capability is jagged: superhuman at one task, absurd at an adjacent one. Only your own task's measurement counts.
A demo shows the top of the output distribution; production serves the tail.

~7 min

Choose a grounding strategy by what each actually buys and the costs each hides, starting with the simplest thing that could work.

Key takeaways

Knowledge that changes or needs citation goes in retrieval; behavior and format go in fine-tuning; small corpora just go in the context.
Most RAG failures are search failures: the model cannot be more right than the passages it was given.
Escalate grounding complexity only when the eval set says the simpler mechanism has hit its ceiling.

~6 min

Build the eval set before the feature, so every prompt, model, and pipeline change is measured instead of vibed.

Key takeaways

An eval set is the test suite of an AI feature: built first, grown from production failures, run on every change.
The demo is the top of the distribution; the eval set is how you see the middle and the tail before users do.
LLM judges prefer verbose, confident, self-styled answers: calibrate them against human labels.

~6 min

Ship probabilistic components safely by bounding the blast radius and routing uncertainty to humans.

Key takeaways

Design from the worst case: enumerate what a wrong output can touch and cap it structurally, not hopefully.
Route autonomy by blast radius: reversible and cheap flows free with audit; irreversible or expensive gets a human gate.
Human review only works when verifying is faster than trusting: show the evidence with the recommendation.

Try a sample check

A sample from the AI Engineering track. The real checks unlock with an account.

AI Engineering

Your first lesson takes five minutes, and you can do it right now without an account.