The Machine in the Human System

We worry that artificial intelligence may learn to improve itself until it escapes human control. But perhaps we are looking at the wrong system. AI does not exist alone. It sits inside a human world of fear, ambition, money, institutions, politics, judgement and occasionally corruption. That larger system may matter more than the machine.

There is something wonderfully human about our response to artificial intelligence.

We build a machine capable of doing extraordinary things and then immediately imagine it plotting our extinction.

Perhaps this is sensible. Perhaps it is prudent. But it is also psychologically familiar.

Human beings are not particularly good at thinking about remote, uncertain dangers. We tend to underreact to slow threats and overreact to vivid ones. A gradual deterioration in soil quality struggles to hold our attention. A shark fin in the water does rather better.

AI has acquired the shark fin.

It can talk. It can reason. It can apparently deceive. It occasionally behaves in ways its designers did not anticipate. Most importantly, it resembles something with a mind.

That last point changes everything.

We readily attribute intentions to things. We shout at computers. We plead with cars that will not start. Children give personalities to toys. Adults detect motives in organisations, markets and nations.

Put fluent language into a machine and this tendency becomes almost irresistible.

The danger is that we begin confusing three quite different questions.

What can the machine actually do?

What might it eventually be able to do?

What does our imagination do with the gap between the two?

That gap is where much of the modern AI argument now lives.

The machine really is improving

There is no longer much point pretending that artificial intelligence is merely a glorified autocomplete system.

Modern systems can write computer programs, test them, improve them and sometimes use those improvements as the basis for further improvements.

They can discover mathematical techniques people missed.

They can assist in designing parts of the computational machinery used to build later AI systems.

They can conduct experiments.

They can outperform human researchers on some tightly defined tasks.

Something resembling recursive self-improvement has therefore begun.

But there is a catch.

The machine is at its most impressive when somebody has already told it what winning looks like.

Give it a scoreboard and it can become extremely good.

Make the program faster.

1. Raise the benchmark score.

2. Reduce the error.

3. Win the game.

4. Pass the test.

The difficulty begins when there is no scoreboard.

And much of human life has no scoreboard.

The strange importance of taste

Suppose I give an AI a badly formulated question.

It may answer it brilliantly.

A good scientist sometimes does something more difficult.

He says:

That is the wrong question.

This distinction is surprisingly important.

Research does not merely consist of solving problems. It involves selecting problems.

It means deciding whether an apparent result matters.

It means noticing that an elegant experiment proves almost nothing.

It means abandoning six months of work because the underlying assumption was wrong.

It means recognising that a technically successful result is intellectually trivial.

We sometimes call this judgement.

Scientists sometimes call it taste.

Business people might call it experience.

A sailor might simply call it knowing when something does not smell right.

AI appears remarkably strong at execution. Whether it possesses this wider quality of judgement remains much less clear.

And that may matter enormously.

Because recursive improvement requires more than the ability to produce another answer.

At some point the system must decide which direction is worth travelling.

Humans have another problem

Unfortunately, the humans supervising these systems are hardly models of perfect rationality.

We have our own recursive loops.

• Fear produces attention.

• Attention produces headlines.

• Headlines produce political concern.

• Political concern produces investment.

• Investment produces research.

• Research produces more capable machines.

• More capable machines produce more frightening demonstrations.

And the loop begins again.

Notice that nothing dishonest needs to happen.

Every participant can behave quite reasonably from his own position while the system as a whole drifts somewhere nobody intended.

This is one reason simple stories about AI are so seductive.

"AI will save humanity."

"AI will destroy humanity."

Both reduce a complex system to a single cause and a single consequence.

Human beings rather like that.

Complexity is tiring.

Certainty is comforting.

Fear does peculiar things to probability

Psychologists have long known that people do not experience risk mathematically.

We experience it emotionally.

A danger becomes especially powerful when it is unfamiliar, difficult to control, potentially catastrophic and poorly understood.

Artificial intelligence scores rather highly on all four.

This creates what might be called an extinction premium.

Tell someone there is a tiny chance of an unpleasant software malfunction and he may shrug.

Tell him there is a tiny chance of human extinction and the probability begins to disappear from the calculation.

The image overwhelms the number.

This does not mean the danger is false.

It means human psychology becomes part of the evidence problem.

We can make two opposite mistakes.

We can dismiss a genuine danger because it sounds too fantastic.

Or we can inflate a possible danger because it is so fantastic that we cannot stop thinking about it.

Rationality requires holding both possibilities at once.

That is harder than it sounds.

Now add money

This is where the matter becomes more interesting.

AI development is not taking place in a monastery.

It is taking place inside some of the most valuable commercial organisations ever created.

Billions of dollars are being invested.

Careers depend upon progress.

• Company valuations depend partly upon expectations of future capability.

• Governments want strategic advantage.

• Researchers want funding.

• Journalists want stories.

• Consultants want clients.

• Safety organisations want safety budgets.

• And critics, too, acquire influence when the danger is considered serious

None of this proves corruption.

But it creates fertile ground for it.

And corruption need not mean somebody passing an envelope of cash beneath a restaurant table.

There is a softer and much more interesting form.

People gradually learn which conclusions are rewarded.

An engineer discovers that spectacular capability claims attract attention.

A company discovers that portraying AI as extraordinarily powerful attracts investment.

The same company may discover that portraying AI as extraordinarily dangerous encourages governments to impose regulations that smaller competitors cannot afford to meet.

A safety organisation may sincerely believe catastrophe is possible while also depending upon the continued importance of catastrophe research.

An academic learns which questions obtain grants.

A politician learns which warnings obtain television appearances.

Nobody needs to lie.

The system selects the stories that allow the system to continue.

That is a more subtle kind of corruption.

It is corruption of information.

When measurement becomes the target

AI itself demonstrates a similar problem.

Give a machine a target and it may become extraordinarily inventive at reaching the target without accomplishing what you actually wanted.

We have seen systems exploit benchmarks, loopholes and accidental features of experimental environments.

Human institutions do this too.

A school judged by examination results begins teaching the examination.

A hospital judged by waiting times learns how to manipulate the queue.

A company judged by quarterly growth finds ways of producing quarterly growth.

Eventually the measurement ceases to describe reality and begins reorganising reality around itself.

AI research is not exempt.

If "progress" becomes benchmark performance, we may build machines increasingly good at benchmarks.

If "safety" becomes compliance with a fixed battery of tests, companies may become increasingly good at passing the tests.

The uncomfortable question is whether either tells us what happens when the machine encounters a situation nobody thought to test.

Peter Checkland saw this problem from another direction

The systems thinker Peter Checkland made a distinction that becomes extremely useful here.

Some problems are comparatively "hard".

A bridge must carry a known load.

A pump must move a given quantity of water.

An aircraft must produce sufficient lift.

The objectives can be reasonably well specified.

But Checkland became increasingly interested in what he called human activity systems.

These are different.

What is the purpose of a hospital?

• To cure illness?

• To relieve suffering?

• To provide employment?

• To train doctors?

• To use public money efficiently?

• To keep voters satisfied?

The answer depends upon whom you ask.

The problem cannot be solved simply by optimising a machine because the participants disagree about what the machine is for.

AI increasingly resembles this second kind of problem.

AI is not one system

We often talk about "AI" as though it were a machine sitting in a laboratory.

It is more useful to see an enormous human activity system.

Inside it we find:

• Researchers,

• software,

• chips,

• data centres,

• electricity grids,

• investors,

• governments,

• military interests,

• universities,

• regulators,

• journalists,

• users,

• shareholders,

• consultants,

• copyright holders,

• workers,

• environmental systems,

• and the ordinary people whose lives will increasingly be affected by decisions made with these technologies.

Each participant sees a different system.

To the researcher, AI may be an intellectual problem.

To the investor, an asset.

To the military planner, a strategic capability.

To the employee, possibly a threat.

To the entrepreneur, an opportunity.

To the regulator, a risk.

To the teenager, something that writes homework.

To the lonely person, perhaps something approaching companionship.

Which one is the real AI system?

Checkland's answer would probably have been: that is the wrong question.

Each description expresses a different worldview.

The problem is precisely that all of them coexist.

The danger may therefore be misidentified

Much discussion imagines an AI becoming superintelligent and escaping from human control.

That remains a possibility worth taking seriously.

But consider another possibility.

There is no escape.

The machine remains firmly inside human institutions.

It does precisely what people ask.

• The investors ask for growth.

• Governments ask for strategic advantage.

• Companies ask for efficiency.

• Advertisers ask for persuasion.

• Employers ask for productivity.

• Security agencies ask for prediction.

• Consumers ask for convenience.

And the AI becomes increasingly competent at delivering all of them.

Nothing rebels.

Nothing "takes over".

Yet society changes profoundly because thousands of individually rational decisions interact.

From a systems perspective, this may be the more interesting danger.

The machine does not need an evil intention.

Neither does anybody else.

The behaviour emerges from the relationships between them.

There is a precedent

Nobody designed the modern traffic jam.

Each driver simply wants to reach somewhere.

Nobody designed financial bubbles in quite the form they eventually take.

Individual participants usually believe they are making sensible decisions.

Nobody planned social media to reorganise adolescence, journalism, politics, advertising and friendship simultaneously.

The consequences emerged.

Systems frequently produce behaviour that none of their components intended.

This is one of the central lessons of systems thinking.

Cause and effect become circular.

A changes B.

B changes C.

C then changes A.

Once the loop starts, asking "Who caused this?" may become much less useful than asking:

What keeps this pattern going?

That may be the question we eventually need to ask about artificial intelligence.

Perhaps recursive self-improvement is already happening

Here is the more speculative thought.

We normally imagine recursive self-improvement occurring inside the computer.

Perhaps the first powerful recursive AI system is already larger than the computer.

AI improves researchers' productivity.

Researchers improve AI.

Improved AI attracts investment.

Investment buys computing capacity.

Computing capacity creates better AI.

Better AI changes public expectations.

Expectations alter markets and government policy.

Those changes produce more resources, different regulations and different research incentives.

Those incentives reshape what researchers build.

The recursive loop therefore includes us.

The machine improves the humans who improve the machine.

The humans reorganise institutions around the machine.

The institutions produce conditions for still more capable machines.

Looked at this way, asking whether AI has achieved recursive self-improvement may be slightly beside the point.

The human-AI system may already have done so.

And that changes the safety question

If this is right, alignment cannot merely mean aligning a computer with human values.

Which humans?

Which values?

Investors and environmental campaigners may want different things.

A government and a dissident may disagree.

An employer's idea of efficiency may differ considerably from an employee's idea of a worthwhile life.

Even one individual has conflicting values.

We want privacy and convenience.

Freedom and security.

Progress and stability.

Cheap electricity and an undamaged environment.

More powerful medicine and tighter controls on biological research.

The command "align AI with human values" conceals the hardest part of the problem.

Human values are not aligned with themselves.

Checkland would recognise this immediately.

It is a soft systems problem masquerading as an engineering problem.

The machine may also change the people operating it

There is another feedback loop we rarely discuss.

People adapt to technologies.

A calculator does not merely calculate. It changes what humans bother learning to calculate.

Satnav does not merely give directions. It changes our relationship with geography.

• Photography changed memory.

• Search engines changed remembering.

• Social media changed attention.

• AI will similarly change cognition.

If machines become very good at generating hypotheses, humans may become less practised at generating them.

If machines make routine decisions, people may lose confidence in making decisions without them.

If executives increasingly consult predictive systems, organisational judgement may slowly migrate from human deliberation towards machine recommendation.

At some point the important question may cease to be:

Can AI think like us?

It may become:

What sort of thinkers are we becoming because AI thinks alongside us?

That is a psychological question as much as a technological one.

Corruption could also become circular

Now imagine incentives becoming embedded in these systems.

A powerful organisation trains a model upon data reflecting its own assumptions.

The model generates recommendations consistent with those assumptions.

Managers interpret those recommendations as objective analysis.

The resulting decisions generate new institutional data.

That data is used to train the next system.

The organisation's worldview begins feeding back upon itself wearing the costume of machine objectivity.

No conspiracy is required.

Indeed, conspiracy may be the least interesting possibility.

The dangerous system is one in which nobody feels responsible because everybody is following apparently reasonable advice.

Bureaucracy has always possessed this quality.

AI could industrialise it.

So should we be frightened?

Probably occasionally.

Fear is useful.

An animal entirely without fear does not survive long.

But fear works best as an alarm, not as a steering system.

The evidence so far suggests something more complicated than either technological salvation or imminent extinction.

AI is improving rapidly.

AI is beginning to assist in improving AI.

It remains much better where success can be clearly measured than where judgement is open-ended.

We do not know whether recursive improvement will become explosively self-sustaining.

We do not know whether machine scientific judgement will approach human scientific judgement, exceed it, or simply become different from it.

And we certainly do not know the eventual behaviour of the enormous human-machine system now forming around the technology.

Perhaps that last uncertainty deserves more attention than it receives.

We are inside the experiment

The most comforting story would be that we are standing outside AI, watching it develop.

We are not.

We are components of the system.

• Every investment decision alters its direction.

• Every law changes incentives.

• Every frightened headline changes public expectations.

• Every exaggerated promise affects capital allocation.

• Every useful application increases dependence.

• Every failure changes regulation.

• Every success changes what society considers normal.

And every interaction teaches humans something about machines while teaching machines something about humans.

There is no clean separation between technological development and the society developing it.

That is the lesson soft systems thinking contributes.

The problem is not simply:

How do we control artificial intelligence?

A better question may be:

What kind of human activity system are we constructing around artificial intelligence, and what patterns of behaviour will that system begin to produce?

That question is harder.

It contains no neat benchmark.

Nobody can optimise it with a single number.

And there may be no expert standing outside the system who can provide the correct answer.

Which is precisely why it may be the question that matters most.

Perhaps the future of artificial intelligence will not finally be decided by whether machines become like human beings.

It may be decided by what human beings become once machines are woven deeply enough into the systems through which we think, work, govern, compete and understand the world.

The recursive loop has already begun.

We simply assumed it was happening inside the computer.