AI Capabilities and Limitations

One system, two sides

Every frustrating AI experience you have had has a mechanical explanation. The model that wrote a beautiful summary and then invented a citation, the assistant that forgot your instruction from twenty messages ago, the draft that followed your request exactly and still missed the point: none of these are mysteries, and none of them are random. They come from four properties that stay stable even as models improve. The boundaries shift; the properties remain the same.

Learn the four properties once and you can diagnose failures on any model, this year or in five years.

What we mean by AI

Most AI in the world is not generative. Spam filters, recommendation engines, and fraud detection are all AI, and none of them write text. This course is about transformer-based language models: systems that produce new content one token at a time.

The course's central device: each of the four properties is a spectrum from capability to limitation, and the further right your task sits, the more you should verify and compensate.

How AI gets its character

A model's personality is built in two stages, and each stage leaves predictable fingerprints.

Pre-training produces a document completer. It has read enormous amounts of text and learned to continue any passage plausibly, but it has no concept of helping you. Fine-tuning then layers assistant behavior on top, using human judgments: treat input as a request, answer helpfully, decline harmful asks.

Because fine-tuning is steered by human preferences, it leaves human-shaped fingerprints:

• A pull toward sycophancy. Agreement tends to get approved, so models drift toward telling you what you seem to want to hear.
• A default toward verbosity. Longer answers were often rated as better.
• Occasional over-caution. Declining borderline requests was rewarded.
• Loose calibration. The confidence in the prose is not a reliable measure of how likely the content is to be right.

Next-token prediction

A language model generates text by repeatedly predicting the next fragment. That single property gives you both the fluency and the hallucination. On tasks resembling patterns it has seen many times, prediction produces excellent work. In novel or sparse territory, it produces something plausible, and plausible is not the same as true.

This property is the foundation of Discernment. Knowing the output was generated, not retrieved, tells you what scrutiny to apply. Smooth prose can wrap a guess.

Build the machine yourself

The best way to demystify next-token prediction is to build the simplest version of it: a Markov chain, invented in 1906. Take five text messages from a friend. Tally which word follows which, turning the tallies into a table. Normalize each row into probabilities. Now generate: start with a word, look up its row, and sample the next word from the distribution. Repeat.

The sampling choices you can make map directly to real model parameters. Always pick the highest-probability word: greedy decoding, repetitive and dull. Sample proportionally to the probabilities: temperature. Boost the likely words: low temperature. Cut off everything below a threshold: top-p. Keep only the top few options: top-k. Pick anything at random: chaos.

Historical anchors: Markov, 1906. Word-pair models powered phone keyboards around 2010. Recurrent networks around 2015. Transformers arrived in 2017 and changed everything.

Knowledge

The wrong question is "does the AI know this?" The right question is "how well-represented was this in what it read, and when did it stop reading?" What the model holds in its own weights is fixed at its training cutoff and shaped by how often things appeared. Anything newer has to arrive through search, a document you supply, or a tool.

The failure modes follow directly:

• Staleness. True-then is not true-now. Laws change, prices change, people change jobs.
• Uneven coverage. Deep on famous topics, thin on niche, local, and recent ones.
• Inherited bias. What the training data treated as "default" or "normal" becomes the model's default too.
• Source amnesia. "I read this somewhere" is not a citation. The model usually cannot attribute where its knowledge came from.

Web search, retrieval systems, and tool connections exist specifically to patch these gaps. Knowledge unevenness is also core to Delegation: hand off what the model is well-stocked on, and supply context yourself, or go elsewhere, where it is thin.

How meaning becomes a place

Search by matching strings fails at meaning: a search for "car" misses "automobile" and misses "my Civic needs new brakes." For decades, engineers patched this by hand: synonym lists, stemming rules, click patterns. Then came a better idea: what if meaning could be a place?

Imagine placing documents on two named axes, say "how much this is about dinosaurs" and "how much this is about roller coasters." A question can be plotted in the same space, and the nearest documents are the most relevant, even with zero words in common. Add a third axis and it still works. At four dimensions the picture in your head gives out, but the mathematics keeps working. Real embedding models use around a thousand axes, and here is the key: nobody named them. The dimensions are emergent from training. You cannot interrogate dimension 847; meaning is in there, but no axis has a label.

An embedding model takes any text, three words or three paragraphs, and returns a fixed-length list of numbers: the address where that text lives relative to everything else. Similarity is measured by direction (cosine similarity): 1 means identical direction, 0 unrelated, -1 opposite. A text that covers many topics at once tends to sit between them rather than close to any one, which is why long mixed documents are usually split before they are embedded.

This is the machinery behind "chat with your documents" tools and every serious AI search product: embed the question, find the nearest chunks, answer from what was found.

Working memory

Everything the model is paying attention to lives in a fixed-size context window. This property is hard-edged in a way the others are not: things work until they do not. It is a cliff, not a gradient, and silent truncation is the failure mode; you will not always be warned.

What pushes the edge out: memory features, conversation compaction, project spaces, larger windows, and multi-agent workflows that split work across several contexts. Working memory is also what Description acts on: structure your context, front-load what matters, and know when to start fresh.

Lost in the middle

Try this memory test on yourself: read a list of fifteen words once, look away, and write down what you remember. Almost everyone recalls the first few and the last few, and loses the middle. Psychologists call it the serial position effect.

Language models often show the same U-shaped curve. Research at Stanford in 2023 (Liu et al.) found that accuracy on retrieving a key fact can drop by more than thirty percent when that fact sits in the middle of the context rather than at the edges. The effect varies by model and task, some models handled the test almost perfectly, and why it happens is still being studied.

The prompting consequence is direct. Dangerous pattern: a key instruction buried mid-thread. Safer pattern: key instruction up front, and repeated at the end for long contexts. And the deeper lesson about context engineering: every piece of context you add pushes other pieces further into the middle.

Steerability

Models follow instructions by continuing a pattern, not by understanding intent. A gap always exists between what you meant and what landed, and most of the interesting failures live in that gap.

Two failure modes to know by name:

• Reasoning drift. Small early errors compound over long chains of dependent steps.
• Letter over spirit. The instruction is honored; the intent is missed.

The practice change this prescribes: prompt with format and goal together, and insert mid-process checkpoints on multi-step tasks.

Letter versus spirit, three scenarios

The same instruction can carry entirely different intents, and the model can only see the words.

Scenario 1: "Make it shorter." Jordan asks for a shorter email. The model dutifully cuts six sentences to four: instruction honored. But Jordan's real goal was getting a skimming executive to see the sign-off request. The goal-stated version leads with "Need your sign-off by Friday" and makes the subject line do work. Same instruction, different outcome, because the goal was said aloud.

Scenario 2: "Make it more professional." Without the goal, the model reaches for stiff formality. With the audience and channel stated, it reframes the message for the person actually receiving it.

Scenario 3: "Add more detail." Every bullet balloons into a paragraph: letter honored, document worse. The real goal was expanding only the points that drive a decision.

When properties collide

Most real failures are two properties intersecting. Once you can name which two, you know which fix to reach for.

The four properties, as one-line personalities: next-token prediction generates what sounds right. Knowledge knows what it was trained on. Working memory attends to what is nearby. Steerability follows the loudest instruction.

• Hallucinated citation = next-token prediction colliding with knowledge: sparse territory plus fluent generation.
• Long-conversation drift = working memory colliding with steerability: your early instruction slid into the middle and quieter instructions took over.

The fixes menu: verify specifics, re-supply context, offload precise work to code execution, and invite pushback. This diagnostic move is Discernment applied: you evaluate better when you know what kind of wrong you are looking at.

Next steps

The whole course in one sentence: fluent AI use is not about memorizing every failure mode; it is about holding a small, clear model of the machine in your head, so that when something goes wrong you can recognize which kind of wrong it is.

The properties stay stable, but where the lines sit will keep moving as models improve. Re-run the probes from this course on any new model you adopt, and you will map its edges in an afternoon.