Like Topsy, Darwin’s empire just grew, the largely uncoordinated work of many hands from many motives. From its beginnings in the seventeenth century until the Indian mutiny of 1857 decisively changed the government’s mind on the role of private companies in imperial affairs, the state was often two steps behind the adventurous trader and merchant and settler, reluctantly acquiring territory simply in order to keep its restless subjects under some sort of control. Even as the mature empire tried to systematise its administration and practices, local élites worked out how to use these structures for their own purposes, which had little or nothing to do with those of the metropolitan power and often worked against them. The advantages of empire did not all flow one way from the periphery to the centre. The British never succeeded in making their distant subjects over in their own image, and scarcely even tried. The empire was no medieval cathedral, slowly being shaped over time into a soaring and coherent edifice. It was messy and fragmented; it defied easy categorisation; it was riddled with contradiction; it was anything but the monolith of popular imagination; and it was always unfinished.

Darwin illustrates his theme of variation and diversity most persuasively in his discussion of what he calls the ruling methods of empire. They did not derive from a ‘standardized apparatus of power’, and nor could they. The image of ‘mustachioed titans of power in shorts’ imposing uniform rule across the empire, he warns, is a caricature of astonishing but lamentable resilience, including among scholars who should know better. London didn’t even try to establish that sort of uniformity. Instead, it accepted ‘wide local variations’ and left much to the discretion of the administrators on the ground. They in turn had to adapt their methods to the peculiarities of place: to the strength of local political traditions, to religious practices, to local economies and systems of land tenure, to the ease or difficulty of using coercive power in particular terrains, and, above all, to the fierce resentment aroused by attempts to impose the expense of empire on the local population – think, for instance, of the response of American colonists to the demand they contribute to the cost of defending them against Native Americans and the French.

In giving this sort of scope to local administrators, London acknowledged its incapacity to exert its authority from the centre. Its administrative structures should take some of the blame for that. The Colonial Office, the India Office, the Foreign Office, the Admiralty, the War Office, and – the bane of everyone – the cost-averse Treasury, all had a hand in the making of imperial policy. It would have been easier to herd cats than get consensus from that lot. The result, Darwin writes, was ‘a ramshackle empire, full of contradictions and quirks’. Imperialists who hung maps that rendered the empire in a uniform red were indulging the fantasies of empire rather than accepting its realities.

The pathos of empire has fascinated historians for generations. Empires decline and fall, as the Roman Empire did for Gibbon, or rise and fall, as the British Empire did in the titles of two of John Darwin’s recent books. Empires are among our most potent symbols of hubris, of dramas of self-destructive ambition played out over generations of conquerors and conquered, and vast tracts of the planet. From the Great War on, the British Empire was manifestly in decline, increasingly eclipsed by the new superpowers and collapsing on the impossibilities of defending an empire grown too large. But look below the heights of metropolitan political ambition, Darwin reminds us, and we see the smaller ambitions of millions of migrants from the British Isles who tried their luck in settler colonies such as those in America, Australia, Canada, New Zealand, and South Africa. Some failed, most succeeded, some wildly, without giving much thought to the grandeur of empire, and often acting in quiet defiance of it even when they were not in open rebellion, as, for instance, when wool growers began to squat outside the nineteen counties bounding Sydney. Their descendants would flourish in new imperial places, even as the ligaments of empire lost their hold.

The empire was most vulnerable, Darwin suggests, under a combination of critical circumstances: instability in Europe, and unrest in the colonies. By that test, the empire was in crisis from the middle of the eighteenth century, long before its period of greatest expansion. Defending British interests against French designs meant expelling the French from North America, to help which the government sought to tax the American colonists. Britain beat the French, and lost America. Within a decade, it was at war in Europe, and the empire again reeled. The settlement of 1815 restored the balance of power in Europe, and opened the period of the most rapid expansion of empire. But the long peace merely masked the problem: British statesmen knew by the end of the nineteenth century that the empire was too large to defend. When the new century brought decades-long turmoil in Europe, the task became an intolerable burden. Finally, the strategic defeats of 1940–42 inflicted ‘indelible injuries’ from which the British Empire would ‘never recover’: Indian independence became irresistible, the white settler colonies no longer looked to Britain for their defence, Britain itself was losing whatever strength in the global economy that remained to it, and the shift of strategic power to the east meant that, unlike after 1815, no postwar settlement could make Europe safe. By the time Europe had been restored to rough political (if not financial) stability after mid-century, the empire was gone. Whatever the intentions of the late imperialists of postwar Britain to the contrary, the commonwealth is not the empire reinvented, and has done nothing to preserve even the semblance of imperial power in the metropolis.

Of all the beneficiaries of the end of empire, the least expected is the royal family itself. The dissolution has been accompanied by an extraordinary flourishing of affection for royalty across the old empire. The two are obviously connected. Power and its exercise always stimulate more fear than affection, as they are intended to do. Had you asked a nineteenth-century Indian the meaning of a photograph of Queen Victoria at her spinning wheel, he would have given you a political answer: the cotton mills of Lancashire could flourish by exporting to India only at the expense of Indian manufacture and thus aspirations to self-rule, and she symbolised that face of empire. With the end of empire, the image of her spinning suggests merely a loopy domesticity, and that’s the way we now like our royalty.

Historians, no less than other scholars, are prone to search for a theory of everything. Not John Darwin. It takes considerable learning, great subtlety, fine prose, and steely nerves to render complexity intelligible while resisting the allure of grand but misleading generalisation. Happily for us, Darwin has them all. If you want only one book on the British Empire on your shelves, make it this one.