Your First AI App: The Lab Comes Alive · Lesson 5 ⏱ ~35 min

Words into actions: the NPC gets hands.

Talking is a party trick. Doing is a product. Today the NPC's replies become structured data your game can execute — ask it to show you a project and it walks you to the desk. This is the exact bridge between "chatbot" and everything currently being called an AI agent.

The idea: make it speak in data

Here's the problem with prose: your game can't act on it. If the NPC says "sure, follow me to the recipes desk!", that's a lovely sentence and the game has no idea what to do with it — sentences don't have coordinates. But you already know the answer to this, because it was the most transferable idea of Course 02: when code needs to act on something, that something should be data.

So we change the contract. From now on the NPC replies in JSON — words in one field, an action in another:

The target shape — what every reply will look like

{"say": "Right this way!", "action": {"type": "walk_to", "target": "recipes"}}

The chat panel shows the say. The game reads the action — the same way it reads lab-data — and executes it. Words for you, data for the machine, one reply.

Step 1 — change the contract

The prompt

I want my NPC to act, not just talk. Update the system prompt so the NPC always replies with pure JSON in this exact shape: a "say" field with its spoken words, and an "action" field. Allowed actions: {"type":"walk_to","target": a computer name from the lab} to lead me to a desk, {"type":"change_screen","target": a computer name, "color": a css color} to change a computer's glow, {"type":"emote","emote":"wave" or "dance" or "jump"} for a little animation, or {"type":"none"} when talking is enough. No text outside the JSON.

In the game: parse each reply, show only the "say" text in the chat panel, and execute the action — the NPC actually walks to the desk for walk_to (and I can follow it), screens really change color, emotes play a small animation. And make the parsing defensive: if a reply isn't valid JSON, just show the whole thing as speech and do nothing — a confused NPC should never crash my game.

That last sentence is load-bearing, and you wrote it before the bug ever happened. Models are brilliant and slightly unreliable narrators of their own format — now and then one gets chatty and wraps the JSON in "Sure! Here you go:". Your safety net means the worst case is a character who says something odd, not a game that dies.

Step 2 — play with it

Refresh and run the experiments. Each of these looks like a chat message; each is actually a command routed through a mind:

"take me to my oldest project" · "make the recipes screen green" · "celebrate!" · "which desk should I be proudest of? show me."

Watch that last one carefully, because two different things happen in order. First the model chooses — it reads your lab-data, weighs your projects, picks one. That's reasoning. Then your code executes — pathfinding, pixels, a little character trundling across the room. That's machinery. An AI deciding, and ordinary code doing: hold onto that pairing for about three more paragraphs.

Checkpoint — a decision, executed
YOU:  which desk should I be proudest of? show me.
NPC:  Easy one — my-page. Your first real build,
      and it's still live on the internet. Come on!
  ✳ NPC walks to the my-page desk ✳

NPC says JSON out loud — braces and all — in the chat panel? → The parser is missing valid replies. Copy one on-screen reply exactly and paste it to Claude Code: "the NPC replies with this text but the game shows raw JSON instead of executing it." One round.

🪤When the JSON breaks anyway — and once in a while it will — your try/catch quietly saves you: the words display, nothing executes, nobody crashes. This exact fragility is why the industry built formal "structured output" and "tool use" features into the APIs: industrial-strength versions of what you just did with a prompt and a safety net. Prompt-and-parse is the origin story of all of it — you've now lived the problem those features solve.

What you built has a fancier name

An AI that picks actions from a menu, and code that executes its choices with guardrails — that pattern has a name, and the name is on every tech headline you've seen this year: an agent. AI that books meetings: an action menu with book_meeting on it. AI that files support tickets, drives a browser, manages a calendar: bigger menus, same shape. Model chooses, code executes, guardrails catch the nonsense.

And one more turn of the screw: you've been living inside one of these all course. When you ask Claude Code to edit a file, a model replies with structured "edit this file" data, and the harness around it executes the edit — with you approving. Claude Code is this pattern wearing a terminal. You've now built the thing you've been sitting in. The circle is extremely closed.

💾Save point: "Commit — the NPC acts: structured replies wired to game actions." One lesson left, and it's the one this whole trilogy was building toward.

You can now…

  • Design a small JSON action schema an AI can reliably produce
  • Prompt a model into structured output — and parse it defensively
  • Execute model decisions in code, with a graceful fallback when it rambles
  • Say what "AI agent" actually means, without hand-waving
← Give it a mind Next: The infinite lab →
Want the course by email? Lessons in your inbox, plus new ones as I write them.

Running a business and would rather this get built for you? That's literally my job — start with the free audit.