Working with AI

AI Fluency

6 lessons · ~38 min total

Tokens, context, and what a model costs

~6 min

Use the mechanical vocabulary of an LLM (tokens, context, latency, cost) precisely enough to price a proposal and question an estimate.

Key takeaways

Everything in and out of a model is tokens, and both directions are billed: the number that matters is cost per request times volume, not the price per million tokens.
A context window is capacity for one request, not memory: anything the product seems to remember is being resent and repaid every time.
Reasoning models spend hidden tokens before answering, buying accuracy with cost and latency, and it is a dial you set per use case.

~6 min

Say what RAG is and is not, and ask the questions that separate a real retrieval plan from the word itself.

Key takeaways

RAG is search plus a prompt: retrieve the best passages from your material, then ask the model to answer from them.
Most retrieval failures are search failures, so the first question about a wrong answer is whether the right passage was retrieved at all.
Retrieval buys freshness, citations, and access control. It does not buy reasoning, and it will not fix a contradictory corpus.

~6 min

Read an eval result, and a vendor's benchmark claim, well enough to say what the number does not tell you.

Key takeaways

An eval is inputs plus judged-good outputs: a score is only as meaningful as the set, the judge, and the baseline behind it.
Public benchmarks measure general ability on shared tasks; only an eval on your own inputs predicts your outcome.
Averages hide the slice that hurts: ask for the breakdown across the cases that carry real cost, and for what the boring baseline scores.

~7 min

Use guardrail vocabulary precisely, including prompt injection, and tell prevention apart from measurement when someone says the system is safe.

Key takeaways

Evals measure how often the system is wrong; guardrails decide what a wrong output can do. Only guardrails act at runtime.
Guardrails stack from weak to strong: prompt instructions, input filters, output validation, action limits, then least-privilege isolation.
Prompt injection: any text a model reads can carry instructions, so protection comes from limiting what the system can do, not from telling it what to ignore.

~7 min

Understand the agent loop, tool calling, and MCP well enough to ask what a proposed agent can do and how anyone would know it worked.

Key takeaways

An agent is a model in a loop that can call tools: it picks an action, sees the result, and picks again until it stops or a limit stops it.
With tool calling the model only requests an action and your code runs it, so control lives in the tool list and the autonomy set per tool.
Reliability compounds down the loop: a 95 percent step is not a 95 percent ten-step task, so evaluate whole tasks and design for recovery.

~6 min

Run a vendor conversation about an AI product: where your data goes, which version you are on, and what leaving costs.

Key takeaways

For bought AI the three questions are: where does our data go, which model version are we on, and what does leaving cost.
A vendor can change the model under a feature and move quality in both directions, so ask for a pinned version, notice, and a window to rerun your evals.
An AI feature switched on inside software you already pay for is a new data flow, not an extension of the existing approval.

Try a sample check

A sample from the AI Fluency track. The real checks unlock with an account.

AI Fluency

Your first lesson takes five minutes, and you can do it right now without an account.