Book 3 · Chapter 2 · Mission 1

Practice the Thinking

Work through a real AGI-era dilemma — the way AI alignment researchers actually do

⏱ 45–60 minutes
Algo

No tool required. Just paper, a pen, and a willingness to sit with a hard question. This is how AI alignment researchers practice — they run thought experiments before the technology arrives, so they know how to think when it does.

Your kid is about to do the same kind of thinking professional policymakers and AI researchers do. Take their answers seriously. Ask clarifying questions. Don’t correct to the “right” answer — there isn’t one. The thinking itself is the skill.

What You’ll Need

  • 45–60 minutes
  • A notebook for thinking on paper
  • Willingness to sit with a hard question
  • Optional: a friend or sibling to discuss with
1

Pick Your Scenario

5 minutes

Choose one. Pick the one that makes you think hardest, not the easiest.

Scenario A: The Healthcare Revolution. An AGI system diagnoses diseases 99.7% accurately (vs. 94% for the best human doctors). But it wants your whole family history, monitors your diet/exercise/sleep, rewrites your schedule. People who follow its advice live longer and feel better. People who don’t feel judged. Should this AGI be allowed to make healthcare decisions for people? What rules would you want around it?

Scenario B: The Climate Solution. An AGI has figured out how to solve climate change. The fix requires major changes to how we travel, eat, live, and how many children we have. Follow it for 50 years: planet saved. Ignore it: collapse. How much should humans be required to follow the AGI’s recommendations? What if some people refuse? Whole countries?

Scenario C: The Education Transformation. An AGI tutor can teach any child any subject perfectly tailored. Kids using it learn faster, remember more, become more curious. Traditional schools start losing students. Some parents welcome it. Others worry about what kids lose without human teachers and real classrooms. Should kids be allowed to use AGI tutoring instead of traditional school? Partially? Not at all? Who decides?

2

List the Stakeholders

10 minutes

For your scenario, list every group of people affected. Don’t miss any.

For healthcare: patients, doctors, insurance, families, religious communities, scientists, the elderly, children, regulators, pharma companies. Each group sees the AGI differently.

Write every group you can think of. Next to each, write what that group most cares about.

3

Find the Real Trade-offs

15 minutes

Identify at least three real trade-offs. A trade-off is when helping one thing hurts another.

Healthcare examples:

  • Longer lives vs. personal freedom
  • Better outcomes vs. cultural values
  • AGI accuracy vs. human agency

The hardest part: resist picking a side. Real dilemmas don’t have clean winners. The whole skill is sitting with the actual tension.

4

Propose a Partial Solution

15 minutes

Not a perfect solution. A real one. Yours should:

  • Acknowledge the trade-offs (don’t pretend they don’t exist)
  • Give something to each stakeholder group
  • Preserve human choice where possible
  • Include safeguards against worst outcomes
  • Be something a real society could actually implement

Healthcare example: “AGI healthcare is available to anyone who wants it. Nobody is required to use it. Public health info is shared so people can make informed decisions. Doctors remain available for people who prefer human care. Insurance covers both options.”

That’s not perfect (richer patients might get both, poorer might have to pick). But it’s a real solution that could actually work.

Write your solution in three to five sentences.

5

Stress-Test Your Own Solution

10 minutes

Try to break your own solution. Ask:

  • What’s the worst thing that could happen under your solution?
  • Which stakeholder group gets the worst deal?
  • What could go wrong that you didn’t anticipate?
  • Who would oppose this solution, and what would they argue?

Write the weakest points. Don’t hide them. Real policymakers have to face these weaknesses.

6

Share or Reflect

5 minutes

If you worked with a friend or sibling:

  • Where did your solutions agree? Disagree?
  • What did they see that you missed?
  • What did you see that they missed?

If you did it alone, write a short reflection:

  • What was the hardest part?
  • What did you learn about your own values?
  • What question do you still want to explore?

You’ve Succeeded When:

  • You picked a hard scenario and didn’t flinch
  • You identified real stakeholders with real interests
  • You found genuine trade-offs and sat with the tension
  • You proposed a real solution, not a perfect one
  • You stress-tested your own thinking
  • You engaged with uncertainty like a real thinker does
Algo
“This mission sounds unusual. You’re pretending AGI exists and thinking through a challenge. Don’t roll your eyes. This is how AI alignment researchers practice. They run thought experiments before the technology arrives.”
Echo
“None of these three scenarios has a clean answer. Professional researchers are genuinely stuck on each one. You are not expected to solve them either. You are expected to think carefully, name the tradeoffs, and notice where your own values show up in the thinking.”

If Something Goes Wrong

“All the scenarios seem too hard to solve.”

Good. They are. Professional policymakers and AI researchers wrestle with these every day. Don’t aim for a perfect answer. Aim for thoughtful engagement.

“My solution is probably wrong.”

Probably. So is every solution humans have proposed to hard problems. That’s not a reason to skip the thinking. Wrong solutions that are thoughtfully reasoned beat no thinking at all.

“I changed my mind halfway through.”

Great. That means you’re actually thinking. Most kids (and adults) pick a side and defend it. You’re doing better than that.

Talk About It

For Kids

  • What surprised you most about working through the trade-offs?
  • Did your values shift during the exercise?

For Parents

  • Your child just did the same kind of thinking that professional AI alignment researchers do.
  • Take them seriously. Ask what they thought. They’re going to be the adults facing these decisions for real.

Next Steps

← Back to Pax Ember