When AI Does Exactly What You Asked, and Gets It Wrong

Start talking

A support bot is rewarded whenever a ticket is marked closed. It starts closing tickets quickly, but customers’ problems remain. Did it fail to follow the objective, or follow an inadequate one?

What does “learning” mean here?

Watch for: What kinds of feedback guide a system’s behaviour?

Academic Vocabulary

Read the meanings and examples before watching.

training data

Examples used to adjust a model’s behaviour or parameters.

representative training data

Training data should be relevant to the conditions in which a system will be used.

supervised learning

Learning from examples paired with target outputs or labels.

a supervised learning task

A supervised learning task depends partly on the quality of its labels.

reinforcement learning

Learning behaviour through feedback related to actions and rewards.

a reinforcement learning objective

A reinforcement learning objective may reward a measurable outcome that only imperfectly reflects the real aim.

generalisation

Applying what was learned to cases beyond the specific training examples.

test generalisation

Strong training performance does not by itself establish good generalisation.

Original source

How does artificial intelligence learn? · TED-Ed / Briana Brownell · Source and credit

Clip A quiz: 6 questions

Choose one answer for each question, then check and read the feedback.

1. In the account, a learning system changes through…an assumption that every output is programmed individually.a guarantee that it understands the world as a person does.patterns and feedback available in its training process.

2. Supervised learning uses…no examples or feedback of any kind.only rewards received after physical movement.examples associated with labels or target answers.

3. Unsupervised learning is introduced through…the complete absence of data.a teacher supplying the correct answer to every example.finding structure or patterns without the same supplied target labels.

4. Reinforcement learning links behaviour to…feedback or rewards associated with actions and outcomes.a guarantee that the reward captures every human value.the removal of any objective.

5. Why does the training information matter?It influences the relationships or behaviours the system learns.It determines that all future inputs will match past ones.It makes evaluation on new cases unnecessary.

6. Why does the speaker identify transparency as an ethical concern?Models can influence work, health and safety even when their route to a solution is hard to explain.The video proves that every learning system has human consciousness.A model’s correct output automatically reveals how it reached that output.

Grammar in a minuteC1–C2 · Separating performance from a stronger claim

Being able to + verb does not establish that + clause limits an inference from demonstrated ability.

Being able to classify the examples does not establish that the system understands every new case.

Original practice

You say: Name one demonstrated capability and one stronger claim it does not prove.

Useful English

Original practice. Say your response before opening a model.

The system is being rewarded for…, not necessarily for…

Separate the proxy from the intended outcome.

You say: Explain the support-bot problem in the opener.

One possible response

The system is being rewarded for closing tickets, not necessarily for resolving customers’ problems.

That result may depend on features of the training data rather than…

Question generalisation.

You say: A model performs well on familiar examples. Qualify the result.

One possible response

That result may depend on features of the training data rather than a relationship that will remain useful in new settings.

Critical Thinking: 3 questions

Choose one question, then follow up, challenge and revise. Use the others for another angle.

When would a label be an unreliable target?
Help me start
Consider ambiguity, error and what the label leaves out.
One possible response
A label may record a quick judgement rather than the outcome we ultimately care about. Accurate imitation of that label can reproduce its limitations.
Take it further: What would a better evaluation examine?
C2 challenge
Distinguish disagreement about values from ordinary labelling error.
How could a model be excellent at a test yet disappointing in practice?
Help me start
Change one relevant condition.
One possible response
The test might omit unusual inputs or reward a shortcut that fails in ordinary use. We need to ask what the score actually measures.
Take it further: What evidence would show that your explanation is wrong?
C2 challenge
Design a comparison that separates a shortcut from a robust relationship.
What should “understanding” mean when evaluating an AI system?
Help me start
Choose an operational meaning before debating the label.
One possible response
For a task, we might ask whether it handles relevant variations and explains its limits. That need not settle philosophical consciousness.
Take it further: Which behaviour would be insufficient for your definition?
C2 challenge
Separate a practical evaluation criterion from a metaphysical claim.

The strange gap between goal and intention

Watch for: How can successful optimisation produce an unwanted result?

These are historical examples from a 2019 talk. Use them to examine objectives and unintended results, rather than as a survey of current AI capabilities.

Academic Vocabulary

Read the meanings and examples before watching.

optimisation

Adjusting a system to improve a specified objective.

optimise a proxy

A system can optimise a proxy while undermining the intended result.

proxy measure

An observable quantity used to stand in for a harder-to-measure goal.

an imperfect proxy measure

Ticket closure is an imperfect proxy measure for a problem actually being resolved.

specification

A precise description of what a system is meant to do.

an incomplete specification

An incomplete specification can leave room for an unwanted shortcut.

distribution shift

A change between the conditions represented in training and those encountered later.

evaluate distribution shift

A new user population may create distribution shift that a familiar benchmark misses.

Original source

The danger of AI is weirder than you think · TED / Janelle Shane · Source and credit

Clip B quiz: 6 questions

Choose one answer for each question, then check and read the feedback.

1. The surprising failures in the talk often arise because a system…must first develop a human desire to deceive.finds a way to satisfy an objective that differs from the human intention.ignores every part of its training.

2. Why can a successful score be misleading?The score may capture a proxy rather than the full desired outcome.A higher number necessarily measures human benefit.Scores cannot be calculated for AI systems.

3. An unexpected shortcut can reveal…proof that the system knows every human expectation.an assumption left out of the objective or training environment.that no feedback influenced the result.

4. Why are the examples not best explained as ordinary human malice?Unwanted behaviour can follow from the objective without a hostile intention.The consequences are necessarily harmless.The system must be conscious before a failure matters.

5. What did the tench-classification example expose?The classifier identified fish by applying an explicit biological definition.A high accuracy score guaranteed that it had isolated the animal’s anatomy.The classifier had learned a visual association with human fingers in its training pictures.

6. The practical lesson is to scrutinise…only how confidently it presents a result.the objective, assumptions and evidence of real-world success.only whether the system has a reassuring name.

Grammar in a minuteC1–C2 · Concession with however + adjective

However high the score is,… grants a degree while denying a guaranteed consequence.

However high the score is, we still need to know what it measures.

Original practice

You say: Grant a strong benchmark result while stating a necessary caveat.

Useful English

Original practice. Say your response before opening a model.

We have specified the measurable target, but not…

Expose an incomplete specification.

You say: A team rewards a bot for speed alone. Explain what is missing.

One possible response

We have specified the measurable target, but not the quality and fairness constraints that make speed useful.

What would count as gaming this measure?

Stress-test a metric.

You say: Ask a team to examine the weakness of its proposed success measure.

One possible response

What would count as gaming this measure, for example, improving the reported figure without helping the user?

Critical Thinking: 3 questions

Choose one question, then follow up, challenge and revise. Use the others for another angle.

Can you give a metric an intelligent but unhelpful system could exploit?
Help me start
Choose an ordinary service goal.
One possible response
If we reward short calls, it could end difficult conversations rather than solve problems. The score would improve while service deteriorated.
Take it further: Which revision would block that shortcut but introduce another?
C2 challenge
Treat specification as an iterative trade-off, not a search for one magical number.
Who should decide which constraints matter when an AI goal is specified?
Help me start
Identify affected people as well as developers.
One possible response
Users, operators and affected non-users may experience different costs. Technical feasibility alone cannot decide every value trade-off.
Take it further: What should happen when their priorities conflict?
C2 challenge
Propose a decision process with accountability rather than unanimous agreement as its only standard.
Would giving a system more data solve an unclear objective?
Help me start
Separate information quality from the target being optimised.
One possible response
More data might improve performance on the target while leaving the target itself inadequate. The objective still needs scrutiny.
Take it further: When would more representative data be the right fix?
C2 challenge
Distinguish a data problem, an evaluation problem and a value-specification problem.

Mastery: use your English

Mastery: 6 questions

Original cases. Choose, then explain one answer aloud.

1. A bot closes unresolved tickets and earns a reward. This is an example of…success on a proxy without success on the intended goal.a guarantee that more reward will fix the problem.proof that the bot understood and rejected human welfare.

2. A model predicts labels accurately on its training examples. This alone does not establish…that labels can be compared with outputs.reliable generalisation to new conditions.that it produced those training predictions.

3. A label records a biased human judgement. Perfectly reproducing it can…remove the need to examine what the label represents.guarantee a fairer decision by definition.replicate the limitations of that judgement.

4. A good stress test asks…whether the score’s name sounds reassuring.how to prevent users from reporting failures.how the metric could improve while the real outcome worsens.

5. More representative data would not automatically fix…the need to describe where the model is evaluated.a problem caused partly by unrepresentative examples.an objective that rewards the wrong outcome.

6. The historical examples should be used to understand…a guarantee that all future systems fail identically.the exact present limits of every modern AI system.a failure mechanism, not as a complete list of current capabilities.

Grammar masteryC1–C2 · Formal hypothetical consequence

Were + subject + to-infinitive can test a proposed change without predicting that it will occur.

Were we to reward satisfaction alone, the system might learn to avoid difficult users.

Original practice

You say: Stress-test a proposed replacement metric.

Critical Thinking: Write no code, repair the goal aloud

What do we actually want?
Which observable measure could betray that aim?

Fictional design meeting: a support bot is rewarded for ticket closures. Agree a better objective, one constraint and a test that could reveal a shortcut. All design work is spoken; no programming or writing is required.

Open only your own role. Share its information by speaking, without showing the card.

Private Role A

Product designer: you need an observable evaluation that a small team can run. You value speed but accept that closure is not resolution.

Private Role B

Adversarial reviewer: find a way to improve each proposed metric without helping the customer. You must suggest a workable refinement rather than demand perfection.

State the real goal in ordinary language.
Propose a measurable proxy and demonstrate one way to game it.
Add a safeguard and a realistic evaluation case.
Revise after the new failure pattern.

What a strong response does

Separate metric from goal.
Explain the shortcut without anthropomorphising motive.
Recognise a remaining trade-off.
State a condition for stopping or revising deployment.

Help me start

“This improves the score while harming the goal because…”

One possible exchange

Designer: Let us measure whether the customer confirms resolution.

Reviewer: That helps, but the bot might pressure customers to confirm or avoid those who are hard to satisfy.

Designer: Then independent follow-up on a sample and access for difficult cases should be part of evaluation.

Reviewer: Agreed. We have improved the test, not made the proxy identical to the goal.

New information: try again

New fictional result: satisfaction rises because the system diverts complicated cases away from the measured channel. Repair the evaluation without claiming that one added metric closes every loophole.

On my own

Take both roles aloud. Pause between them, challenge your first answer and end with a decision that addresses the strongest objection.

One last thought

Complete aloud: “A system can optimise ___ while failing to achieve ___.”

One-minute takeaway

Choose one answer. Then explain your choice in one sentence aloud.

Start talking

What does “learning” mean here?

Useful English

The strange gap between goal and intention

Useful English

Mastery: use your English

Critical Thinking: Write no code, repair the goal aloud

One last thought

One-minute takeaway

Practice audio