The Horse, the Billiard Ball and the Family Firm

Artificial intelligence promises family businesses better information and better decisions. But a machine trained on an organisation’s own records may do something more dangerous: turn its inherited beliefs into apparently objective knowledge.

Imagine two objects.

The first is a billiard ball.

The second is a horse.

Strike the billiard ball with a known force, at a known angle, across a surface whose characteristics you understand, and you can make a reasonable calculation about where it will finish.

Now strike the horse with exactly the same force.

The mathematics becomes less useful.

The horse might bolt.

It might kick.

It might freeze.

It might remember you.

It might behave differently the next time you approach.

More importantly, the horse responds not only to force but to what the event means to it.

That small difference takes us from engineering towards psychology.

It also takes us towards one of the most important problems artificial intelligence may create for family businesses.

The comfortable world of machines

Mechanical systems appeal to us because they appear understandable.

An internal-combustion engine contains components whose relationships can be described. Fuel burns. Pressure moves a piston. The piston turns a crankshaft. The crankshaft eventually turns the wheels.

An electric bell offers another neat example. The electromagnet attracts the armature. This breaks the contact. The magnet releases it. Contact resumes and the cycle begins again.

Once we understand the components and the feedback, prediction becomes possible.

But even here we cheat slightly.

We draw a boundary around the machine.

The engine diagram shows pistons, valves and crankshafts.

It probably does not show the child developing asthma beside the road.

Pollution exists outside the boundary we chose.

The system did not tell us where its boundary lay.

We did.

That seemingly minor observation becomes enormously important when we move from engines to organisations.

Nature refuses to stay inside the box

Natural systems are harder.

Suppose goats graze a field.

More goats mean less grass.

Less grass may mean weaker goats.

Weak or dead animals support scavengers.

Changes in prey numbers alter predators.

Predators change the goat population.

• Grass recovers.

• Weather intervenes.

• Disease arrives.

• Humans fence part of the land.

The system changes again.

Older ecological thinking often described this as the "balance of nature", but modern ecology is less inclined to assume that natural systems always return obediently to one equilibrium. Ecosystems may oscillate, reorganise or move into quite different states.

There is no chief executive goat.

Nobody wrote the business plan.

The order emerges from relationships.

That is the first lesson family businesses might borrow from ecology.

A business may possess an organisational chart.

Its behaviour does not necessarily follow it.

Then people arrive

Peter Checkland encountered a similar problem when attempts were made to apply engineering-style systems thinking to organisations.

Engineering begins comfortably when the objective is agreed.

Build a bridge capable of carrying a particular load.

Design a machine that produces a particular output.

But what is a family business for?

• To maximise profit?

• To provide employment for the next generation?

• To preserve the founder's name?

• To maintain a property?

• To provide independence for the family?

• To support the village?

• To provide Mum and Dad with an income in retirement?

Ask six family members and you may receive seven answers.

Checkland's Soft Systems Methodology developed around precisely this difficulty. In human situations, the "problem" does not simply sit outside us waiting to be discovered.

Different participants may perceive different problems because they understand the situation differently. The person investigating the problem is also part of it.

That is a much more disturbing idea than it initially appears.

The family business is not a machine

Consider a family-owned hotel.

The accounts show that the restaurant is losing money.

To the accountant, the problem may appear obvious.

Close the restaurant.

To the founder, the restaurant may be the heart of the business.

To his daughter, it may be an outdated operation preventing investment elsewhere.

To a grandson working in the kitchen, it may represent his future.

To local customers, it may be one of the last places where the village still meets.

To the bank, it is an inefficient use of capital.

Which one has correctly identified the system?

All of them.

And none of them.

They have drawn different boundaries around the same situation.

Checkland's contribution was not to discover a better diagram.

It was to recognise that the diagrams themselves belong to the inquiry. He eventually argued that in complex human situations we should be cautious about imagining that "the system" objectively exists exactly as we have drawn it. Systemicity may reside partly in our way of thinking about the world.

This becomes very important when somebody puts an AI system in the boardroom.

The machine arrives bearing facts

Suppose the family hotel has struggled for three years.

The managing director believes marketing is the problem.

He asks the AI to investigate.

The machine is given access to sales reports, marketing plans, customer reviews, meeting minutes, email correspondence, budgets and consultant reports.

It produces an impressive analysis.

Marketing needs improvement.

The managing director feels vindicated.

More money goes into marketing.

The marketing department produces more campaigns, reports and customer surveys.

Meetings discuss marketing performance.

Consultants are appointed.

Their reports enter the company's files.

Six months later the AI is asked to reconsider the problem.

It now discovers even more evidence that marketing is central.

Of course it does.

The organisation has spent six months manufacturing information about marketing.

The machine has not necessarily discovered the truth.

It has discovered what the organisation talks about.

Those are not the same thing.

When confirmation puts on a white coat

Human beings have always suffered from confirmation bias.

We notice evidence supporting our beliefs more readily than evidence undermining them.

Family businesses can be particularly vulnerable because beliefs acquire history.

• "Customers come here because of the food."

• "Your grandfather always said never borrow money."

• "Our family does not advertise."

• "David was never interested in the business."

• "Sarah is the creative one."

• "The eldest son should run it."

Over the years these remarks can stop sounding like opinions.

They become facts.

Then AI arrives.

And AI may give them numbers.

This creates something psychologically new.

If Uncle Robert says:

"The younger generation has no commitment,"

the family can argue with Uncle Robert.

If an AI-produced report states that younger family members display "lower long-term organisational engagement", the same prejudice has suddenly acquired a tie and clipboard.

It looks objective.

The machine has laundered the opinion.

Corruption without dishonesty

We usually imagine corruption involving money.

• Someone takes a bribe.

• Someone alters the accounts.

• Someone gives a contract to a friend.

But information can become corrupted without anyone lying.

Suppose the founder believes expansion failed because his son managed it badly.

He says so at meetings.

Other executives learn which interpretation the founder prefers.

Minutes emphasise operational mistakes.

Consultants are briefed to examine implementation.

Reports discuss implementation failures.

Emails repeat them.

Five years later an AI system searches the company archive and concludes:

"The principal cause of the unsuccessful expansion was deficient execution."

Everyone nods.

The AI has found dozens of sources saying so.

But perhaps all those sources descend from the same original belief.

Thirty documents do not necessarily represent thirty independent observations.

They may represent one idea reproduced thirty times.

Family businesses understand this problem better than most.

Ask six cousins how Grandfather founded the company and you may hear the same story almost word for word.

That does not prove the story false.

But repetition is not independent verification.

The family archive can become an echo chamber

This may become one of the strangest consequences of corporate AI.

Companies are enthusiastically creating organisational memories.

• Emails.

• Meeting transcripts.

• Customer records.

• Staff evaluations.

• Policies.

• Strategy documents.

• Board papers.

• Financial histories.

AI can search all of them in seconds.

That sounds wonderful.

And frequently it will be.

But memory has never been synonymous with truth.

Families know this instinctively.

• Family stories are edited.

• Embarrassing episodes disappear.

• Successful decisions acquire more foresight in the retelling.

• Failures acquire villains.

• The founder becomes cleverer after death.

• Old arguments become principles.

The archive records not simply what happened but what people were prepared to write down about what happened.

Then the machine reads the archive.

The danger is not hallucination

Much public discussion worries about AI inventing things.

That is certainly a problem.

But in organisations there may be an opposite danger.

The AI faithfully reports exactly what it finds.

That can be worse.

An organisation may already contain distorted categories, political compromises, historical prejudices and convenient assumptions.

A perfectly accurate summary of them remains distorted.

AI does not need to hallucinate.

It merely needs to become very good at explaining the organisation to itself.

This is where machine-generated confirmation may begin masquerading as objective knowledge.

Family businesses add another layer

A public company can theoretically sack its chief executive.

A family cannot sack Uncle Peter from being Uncle Peter.

Roles overlap.

The shareholder is also the mother.

The sales director is also the younger brother.

The retired chairman still owns the building.

The badly performing manager may also be the person everyone expects to care for the parents in old age.

This makes the family firm a classic human activity system.

Purpose cannot be separated neatly from relationships.

A decision that appears economically irrational may make perfect sense when the family system is included.

Imagine a village shop employing an apparently unnecessary family member.

A conventional efficiency model recommends redundancy.

But perhaps that salary allows him to remain near his elderly parents.

His presence allows his sister to continue managing the business rather than becoming a full-time carer.

Her continued management keeps the business operating.

The apparently inefficient employee may therefore be supporting the entire enterprise.

Whether he represents "surplus labour" depends entirely upon where we draw the system boundary.

The spreadsheet did not make a mistake.

We asked it the wrong question.

This is the billiard ball problem again

We often want business analysis to resemble physics.

• Input.

• Output.

• Efficiency.

• Return.

• Prediction.

That works beautifully for some problems.

A new refrigeration unit consumes fewer kilowatt-hours.

That can be measured.

But people do not behave like refrigeration units.

Tell a machine that next year's electricity price will rise and nothing psychological happens to it.

Tell a family that the business may fail next year and the prediction itself changes behaviour.

• One brother delays buying a house.

• Another begins secretly looking for employment.

• Their father becomes more controlling.

• Suppliers hear rumours and shorten payment terms.

• The bank becomes nervous.

A prediction has entered the system it was predicting.

Social systems are reflexive.

The map can alter the territory.

AI makes maps very quickly.

The horse knows something happened

Which brings us back to the horse.

The important thing is not merely that its behaviour is difficult to predict.

Weather can be difficult to predict.

The horse possesses some capacity to interpret the event.

And humans plainly possess far more.

This raises an intriguing question about where "soft" systems really begin.

Consider killer whales.

Since 2020, members of the small Iberian orca population have repeatedly interacted with vessels and particularly their rudders. Scientists have documented hundreds of encounters. Yet describing the physical interaction does not establish what it means to the whales.

Play?

Curiosity?

Learned behaviour?

Some response to vessels?

Scientists remain cautious about assigning motive.

We do know, however, that killer whales can acquire behaviour through both individual and social learning. Research on other populations has shown behaviours spreading through social groups.

The philosophical point is more important than deciding what the Iberian whales intend.

Once an organism learns, interprets and changes behaviour according to experience, billiard-ball thinking becomes increasingly inadequate.

Something resembling meaning has entered the system.

Perhaps we need another category

Mechanical systems are comparatively easy to understand because the components behave consistently.

Natural systems add adaptation, ecological feedback and evolution.

Checkland's human activity systems introduce purposeful action and different interpretations of what the situation means.

I would go slightly further.

Perhaps we should speak of interpretive systems.

An interpretive system contains agents that do not merely react to events.

They construct some representation of what events mean and alter their behaviour accordingly.

Humans do this conspicuously.

Some other animals appear to do elements of it.

And AI now does something functionally similar, whether or not one believes that machines possess anything resembling consciousness.

This produces a very peculiar organisation.

The modern family company is becoming a system of humans interpreting machines that interpret records created by humans interpreting one another.

The loops multiply.

When the map begins drawing the territory

Now imagine the process continuing.

The AI tells the family that its premium customers value tradition.

The family therefore emphasises tradition in its marketing.

That attracts more tradition-minded customers.

Their behaviour enters the database.

The AI analyses the new information.

It now finds stronger evidence that customers value tradition.

The recommendation has helped create the evidence confirming the recommendation.

Nothing dishonest has happened.

Indeed, the AI may have been correct.

But cause and evidence have become entangled.

This is the point at which family businesses need to become especially careful about phrases such as:

"The data tells us..."

Data rarely speaks.

• People collect it.

• People categorise it.

• Machines process it.

People decide which questions to ask.

Then people interpret the answer.

Checkland would ask us to examine the worldview beneath the exercise.

• What transformation are we trying to achieve?

• Who benefits?

• Who owns the process?

• Who might be affected?

• What assumptions make the recommendation appear sensible?

And, most importantly:

What did we leave outside the system boundary?

AI may preserve the family mythology

There is an irony here.

Many people expect AI to destroy tradition.

In family businesses it may sometimes fossilise it.

Imagine a fourth-generation company training its internal AI upon 80 years of correspondence, decisions and management records.

The resulting system becomes extraordinarily knowledgeable about "how we do things here".

That could be priceless.

It could also become the most articulate conservative in the building.

A human successor may challenge Grandfather's assumptions.

The machine may quote Grandfather back at him.

Once historical precedent can be retrieved instantly, tradition gains a technological voice.

The family archive stops being merely memory.

It begins participating in decisions.

The dead acquire voting rights.

The great temptation will be certainty

Family businesses are emotionally expensive places in which to make difficult decisions.

• Should we sell?

• Should a daughter rather than a son become chief executive?

• Should an incompetent relative be removed?

• Should the old factory close?

• Should dividends be reduced to fund expansion?

People desperately want an answer that does not belong to them.

Consultants sometimes serve this psychological function.

So do accountants.

So do boards.

AI may become exceptionally powerful at it.

"I didn't decide this. The analysis did."

That sentence should worry us.

Because responsibility has not disappeared.

It has merely become difficult to see.

The machine should become another voice, not the voice

None of this argues against AI in family business.

Quite the opposite.

A system capable of reading decades of records, comparing scenarios, identifying contradictions and remembering decisions could become an extraordinary aid to family governance.

But perhaps its most valuable function should not be answering:

What should we do?

It might instead help us ask:

1. What assumptions produced this answer?

2. What evidence contradicts it?

3. Whose perspective is missing?

4. What changes if we redraw the boundary?

Are thirty documents really thirty sources, or thirty descendants of one old belief?

What would somebody who disagreed with us notice?

That would make AI less like an oracle and more like a participant in Checkland's learning process.

Which may be exactly where it belongs.

A family business is a conversation across generations

The family firm is never merely the thing shown in Companies House.

It is machinery, property, customers, money and contracts.

But it is also memory.

• Status.

• Marriage.

• Sibling rivalry.

• Sacrifice.

• Inheritance.

• Obligation.

• Stories about the founder.

• Ideas about fairness.

• Ideas about who belongs.

• And beliefs about what the family is for.

Those elements interact.

They change one another.

They sometimes keep the organisation alive long after conventional economics says it should disappear.

They sometimes destroy excellent businesses for reasons no spreadsheet could have predicted.

AI will enter this system whether we are philosophically ready for it or not.

The greatest danger may not be that the machine develops its own beliefs.

It may be that it becomes extraordinarily skilled at reflecting ours.

And because the reflection arrives accompanied by statistics, fluent prose and immense computational power, we mistake the reflection for an independent view of reality.

The billiard ball does not care what we believe about it.

The horse might.

The family certainly does.

And the machine is beginning to learn from all three.

The question for the family business is therefore not merely whether AI can produce better answers.

It is whether we remain able to recognise where those answers came from.

Because once an organisation's beliefs enter the machine and return wearing the clothes of objective knowledge, the oldest family story in the company may suddenly look like science.