In the corner of a small graveyard, an exhaust plume away from the wake-the-dead roar of London's Old Street roundabout, lies the tomb of an obscure 18th-century clergyman and statistician named Thomas Bayes. Among his companions in Bunhill Fields, the cramped resting place of many of Britain's most notable religious and political nonconformists, are Daniel Defoe, John Bunyan and William Blake. If his growing band of admirers is right, though, Bayes might soon gain the kind of international recognition that would place his neighbours in the celebrity shade.
Bayes published only two minor works during his life, one of them anonymously, and neither was of any lasting consequence. But among the notebooks on astronomy and electricity he left in his will, there was a tentative essay entitled 'Towards Solving a Problem in the Doctrine of Chances', which was published in 1763, two years after his death. The centrepiece of the work was an equation designed to predict events in conditions of uncertainty. And it is this methodology that, according to some informed observers, is now transforming the world of science.
'Walk into a research department in Cambridge, MIT or Stanford nowadays,' says Mike Lynch, Britain's leading software entrepreneur and a devout Bayesian, 'and you will meet people who will tell you that Bayes is more important than Marx and Einstein put together.'
For a quarter of a millennium Bayes's theorem, or Bayes's rule as it is sometimes called, enjoyed a limited and mostly discredited role in statistical mathematics. Recently, however, with the advent of cheap and available computers, its influence has rapidly spread beyond the dull grind of statistics to become something akin to a philosophical movement, with an almost theological appeal. Yet it's not a system of belief so much as a means of measuring belief.
Lynch is in no doubt that the clergyman's approach to what is known as probability inference has at last come of age. 'It has taken 250 years for people to catch up with him,' he says, 'and realise the potential of his ideas.' A 35-year-old billionaire with an expanding company and a receding hairline, Lynch is an enthusiastic advocate of Bayes. He speaks of the Pauline conversion that Bayesians typically undergo: 'People leave the lab one night non-Bayesians and return the next morning convinced Bayesians, and they can't do anything in life — make a cup of tea or pick a girlfriend — without viewing it from a Bayesian standpoint.'
Indeed such is the evangelical fervour of born-again Bayesians that it seems almost churlish to ask what exactly it is that the man came up with. The answer, as it turns out, is both stunningly simple and infinitely complex.
Not until after 1958 did an entry appear for Bayes in the Encyclopaedia Britannica, and even then its details were enigmatically sparse. Almost nothing is known for certain about the man whose speciality was uncertainty; even his birth year is subject to debate (it could be 1701 or 1702).
The biographical bones amount to little more than that he was the son of a nonconformist minister from London who himself became a Presbyterian clergyman in Tunbridge Wells, Kent. A fellow of the Royal Society, he was a keen Newtonian and he may have attended Edinburgh University. And that's about the sum of it. Were it not for his friend, Richard Price, another nonconformist minister and amateur statistician, Bayes would have been lost to history. Price, who inherited Bayes's notebooks along with £100, was a natural publicist and spin doctor. A staunch supporter of the American and French revolutions, he recognised the earth-shaking quality of Bayes's musings. And he did not play it down. Replacing Bayes's deprecating introduction with his own brand of hyperbole, Price claimed that the purpose of the essay was 'to confirm the argument... for the existence of the Deity'.
In fact, Bayes had done no such thing. If he was interested in the unknown, it was in predicting, not confirming, it. Specifically, Bayes had come up with a means of calculating the probability of a future event occurring based on previous events, current conditions and all other known and related factors. In other words, he had devised a formula for that most unconscious and subtle of skills: human intuition.
Every day, whether driving a car or deciding what clothes to wear, we analyse the probability of events — is it going to rain? Will a truck come round that corner? — by drawing on sensory apprehension and a background of stored information in such an unthinking manner as to seem almost instinctive. And as we learn each new factor — the distant appearance of a grey cloud, the sound of a horn — we are instantly able to change our estimation of the likelihood of that rain or truck arriving.
In turning that sophisticated process into a basic equation, Bayes, no doubt unknowingly, upended science. Traditionally, scientific method is based on making the imperfect real world fit into a perfect model. The beauty of Bayes's rule is that it accommodates and allows for the myriad anomalies of real life. Nevertheless, after a brief fashion, Bayes's rule was soon viewed by statisticians as a flawed concept. Only as recently as the 1950s was it re-evaluated.
'Everybody is a Bayesian unless they've been trained otherwise,' argues Professor Anthony O'Hagan of Sheffield University's department of probability and statistics, and a leading figure in Bayesian circles. But, as it happened, most statisticians were trained otherwise. The orthodox school was only interested in how often things happen. Therefore no other criteria but an event's frequency were considered when estimating the likelihood of its happening again.
Gradually Bayesians, with their all-encompassing, if 'subjective', outlook on life, have established themselves as the orthodoxy, although the more rigid, but 'objective', non-Bayesians continue to fight a bitter rearguard battle.
In itself that debate, while engrossing enough for statisticians, is a trifle arcane for most of us. The greatest impact of Bayesian thought is on computer engineering, a revolution that may lead to computers being able to act like us: intuitively. And the Bayesian who has had the greatest impact on computer engineering is Mike Lynch.
The progress of Lynch from a graduate student who couldn't get a bank loan to a billionaire businessman, inside a decade, is one that even the most cavalier Bayesian would not have dared to predict. 'No,' agrees professor Peter Rayner, of Cambridge University's engineering department. 'Not enough data.'
Lynch worked on post-graduate research into probability theory and Rayner was his adviser and Bayesian mentor. 'There were two things I noticed about Mike. First, he didn't often read his lecture notes but he invariably came up with novel solutions, even if they were not always right. He was clearly a very creative student. The other thing was that it was obvious he was going to make an awful lot of money.'
The son of an Essex fireman, Lynch won a scholarship to Bancroft's School before going to Cambridge where he took a degree in information engineering. 'Mike had a very sharp brain,' recalls Rayner. 'But in pure academic terms we've had a number of sharper students. He had confidence.'
In the offices of Autonomy, Lynch's internet software company in Cambridge, I waited in a large and overheated boardroom to meet this brilliant, self-confident billionaire. After a few minutes a man in a plaid jacket and jeans walked in and started tapping the thermostat. I was relieved to see a maintenance engineer dealing with the room's uncomfortable stuffiness.
'Hello,' he said. 'Bloody thing's broken.'
I smiled and then realised that he wasn't the company spark, but the bright spark who owned the company. At first sight, Mike Lynch is an unassuming, indeterminate presence: a round face with an acute expression, a goofy smile and quizzical eyebrows, young without looking youthful, and informal but not quite at ease. But when he talks he does so with a commanding wit and intelligence, so that you want to listen even if you don't quite understand.
Perhaps it was this conversational talent that inspired the manager of a rock band to give him £2,000 in a Soho wine bar one night back in 1991. Lynch was completing his PhD and, in his words, 'Bayesianism was just beginning to sweep like wildfire.' He wanted to set up a company and had enjoyed little success with more conventional routes for raising finance. 'The kind of venture capitalists you met in those days were the idiot sons of idiot stockbrokers,' he recalled. 'Complete waste of time. I went to the bank and met an incredibly nice chap who listened to me and admitted at the end that he normally did loans for people buying newsagent shops. We had an interesting discussion about confectionery. People always want sweets.'
Then there was an equally forlorn visit to the DTI. 'We had tea served in a full china tea service. We had a long chat and he showed me some of the things that the DTI were involved in. Classic British mistake: they loved anything quirky. So it was the sub-bicycle that could be ridden across lakes.'
In a sense, the bankers and civil servants were non-Bayesians. They simply looked at the frequency with which specialist software companies had succeeded and made what they thought was an objective decision against taking the risk. The inebriated rock manager, whom Lynch refuses to name, was a natural Bayesian. He looked at his own record of backing unknowns (what Bayesians call 'the prior'), listened to what Lynch had to say ('the evidence') and then took a subjective and intuitively calculated risk ('the model').
'I think he liked the enthusiasm of youth because he couldn't have had a clue what I was talking about, which was non-linear adapted pattern recognition. These days,' Lynch added, 'I've learnt the concept of marketing.'
He set up a company with the cyberpunk name of Neurodynamics and, like some sinister, futuristic brain from a William Gibson novel, developed a system for matching fingerprints for the Essex police force and another for reading car numberplates. The pattern-recognition solutions Lynch came up with were the result of complex Bayesian methods. In 1996, using the same principles, he set up Autonomy to exploit the great problem of the early computer era: unstructured data. Or, as it's more commonly known, prose.
You only have to use a search engine to realise how cumbersome computers are when it comes to deciphering written text. Type in the word 'octopus' and you'll be directed to a whole range of webpages from Octopus book publishers to a tribute to the Beatles' Octopus's Garden, but none of these will help you if you are interested in cephalopods. The problem is that computers can recognise individual symbols but are unable to identify the context of symbols. Autonomy software enables a computer to analyse the patterns of a document, regardless of its language, and then draw conclusions as to its relevance or importance.
'We don't need to know the meaning of the link between words, we just need the probability distribution,' Lynch explained. 'So if you see the word "cat", it's probably about felines. If you see the words "cat" and "nine", it's probably about felines. Nine lives. If you see the words "tail" and "cats", it's probably about felines. Cats have tails. If we see "cat", "nine" and "tails", it's probably about pirates. You start to build up these strange combinatory effects.
'The other important thing is conditional probability, which is the ability of one event to change the chances of another one. If we see the word "cat", it's about felines. If we see the word "burglar", it's about your house being broken into. If we see "cat burglar", the probability is not half and half. The meaning of cat has been modified and we can express that through probability distribution. Therefore you can develop probability calculations, which is what Bayesian inference is about. The beauty of it is that the belief, the idea of what it's about, is independent of words.'
At that point he caught sight of my dazed expression; a look not seen by anyone, I suspect, since my A-level maths teacher attempted to share with me his passion for calculus. 'That's as clear as mud,' said Lynch.
Now Europe's fastest-growing internet software company, Autonomy is a global concern with offices across America and Europe. Its clients include the US Department of Defence, a number of media conglomerates, General Motors and the BBC. But in truth, Autonomy has not yet achieved anything really sexy, unless you're excited by the idea of a computer automatically replying to an e-mail. But it is a giant leap on the way to computers making decisions. And Microsoft is just one of a number of companies that are following Autonomy's lead.
Lynch still finds time to get along to gatherings of fellow Bayesians to discuss probability theory: 'It's like going to a cheese-and-wine do, but there are blackboards around the room. You have to be careful because you can find yourself wiping someone's equation off the board with the back of your jacket.'
It's a typical Lynch comment, comic and demystifying. Like Bayes himself, who argued that maths was a matter of amusement rather than high seriousness, Lynch finds much of what passes for rational thought utterly absurd. 'One of the things I find hilarious is modern management textbooks. The idea that you've got this human brain that understands all the subtleties of employees, markets, what's going on — general feeling — and instead you're supposed to make decisions by looking at some stupid diagram.'
Certainly few management consultants could quibble with his results. A few weeks after I met him, I called him up to ask, among other things, if it was true that he was a billionaire. He told me people used the term because they divided the market price of his company by the percentage of his shares in it. Autonomy was valued at £4.7 billion.
I asked how much he owned. 'Er,' he said, '19 per cent. But I still eat at McDonald's.' What he didn't mention was that later on that day he would sell one per cent of shares, leaving him with 18 per cent of the company and £47.4 million in the bank. As it turned out it was a well-timed measure. The worldwide collapse in high-tech shares led to Autonomy losing more than half of its share value between October last year and January this year. Lynch felt moved to make a statement early in the new year to reassure shareholders that the future was still bright.
When not commuting to San Francisco, he lives alone — apart from his dog, Gromit — in a tiny Suffolk village with a Viking tower. There is no room or time yet for a steady girlfriend. But he savours the slow pace of the English countryside. 'In London or San Francisco you go into a restaurant and overhear people talking about C-programming or Java. In Suffolk conversation is about how to catch moles. It's the great unsolved problem, more so than Fermat's last theorem.'
He also keeps koi carp in a pond and maintains a small-scale railway in the garden. 'Pretty much the exact replica of a steam engine. I went to an agricultural auction, a real string-round-the-trouser-leg job. I managed to buy one for next to nothing.'
That £100 and the notebooks that Thomas Bayes left Richard Price turned out to be some inheritance. What is the probability distribution that in the future coachloads of superwealthy geeks from Silicon Valley will be paying homage in a cemetery just near the roundabout at Old Street?
1