There is a paradox in public life in Britain today. The population is more highly educated than ever before, a larger proportion of young people attend university or further education today than in the past, the UK is among the highest-ranking countries for Nobel prize winners, and its cities are magnets for global talent in sectors such as finance, technology, arts, medicine and law.

Yet, despite nurturing an extraordinarily high stock of human capital over the last 50 years, Britain’s political class feels more inadequate and ineffective than ever before. Logically, a rise in the general intelligence of the population should result in a larger, more diverse and more impressive talent pool for the political elite. The reverse seems to be the case in the UK. Has there ever been a time when the standard of the political class has felt so out of line with improvements in wider society?

None of this is to be glib – it is practically a national sport in Britain these days for armchair critics to lampoon politicians, but politics is a much tougher gig than most people realise. Theodore Roosevelt was surely right to say in his defence of those in public life: ‘It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood.’

I write this essay as someone who has worked within the British political system and therefore has more sympathy than most for the people in it. For nearly three years I was the director of the prime minister’s Policy Unit in 10 Downing Street and had the remarkable privilege of observing the inner workings of Whitehall. I watched ministers, MPs, senior civil servants and political advisors grapple with the turbulence of the EU referendum result, parliamentary deadlock, a general election and the Covid-19 pandemic. It was an unparalleled education in the way government operates and the nature of those who run it.

Many of them are talented people dedicated to public service, who go to work every day to make the country a better place. Yet, for all the good people I met, I came away convinced that our political system does not prepare leaders as well as it should, nor does it attract enough of the talent needed to deal with the challenges the UK faces.

The truth is that people within politics are themselves becoming aware that there is a systemic problem in how we as a country identify and train MPs, Cabinet ministers, senior civil servants and advisors. Many of the old pipelines of talent are losing their appeal, or no longer exist. And for those who do arrive in politics, they are unlikely to experience the sort of ‘apprenticeship’ journey their predecessors did. In the past, Cabinet ministers had built careers in other professions, then spent years acquiring knowledge and experience on the back benches and as junior ministers.

Civil service mandarins were examined at entry – and frequently assessed – on their objective knowledge, not just their subjective qualities. All senior officials underwent rigorous training in thorough, structured courses at a national college. The political journey for many people running the country today is shorter, less demanding and more ad hoc.

The consequences of all this are that the quality of decision-making is often erratic, with government unable to think and act long term. Too often there is a lack of institutional memory and policy knowledge among legislators. Political parties struggle to generate new ideas and resort to familiar rhetoric invented for problems from decades ago. More than anything, a new generation of political leaders struggle to find a way to communicate their views at any length or articulate a vision with depth.

The quality of politicians (and public servants) is far from our only problem. Economic headwinds, global conflict, demographic change and other factors beyond individual control all play their part. But if politicians do not look like serious people, if they cannot develop credible answers or build support for their agenda, the result will be a declining faith in democracy. According to the Office for National Statistics, public trust in national government has now fallen below 40 per cent. Whether it is chicken or egg, politicians themselves end up acting in ways that reinforce the problem. When they go on reality TV shows to try to engage an increasingly detached electorate, they are signalling that politics is more about ‘personality’ than purpose and ideas.

The problem is not unique to Britain either. Henry Kissinger – who probably met more significant world leaders than anyone else in recent years – reflected that the rise of social media has reduced the attention span of politicians and voters. Despite being more likely to have earned a degree than in the past, elites have lost access to the deep literacy that was a given for previous generations, and which is vital to understanding the situation of their country. The most impressive world leaders of the twentieth century, from Charles de Gaulle to Lee Kuan Yew, spent time learning about the global forces facing their countries, the diverse needs of their people and how to build a team that could pursue a vision.

Today, the type of education and sense of duty that shaped the post-war generation is no longer present to the same extent. While we still have very good, talented individuals in society, we can no longer assume that the alchemy that attracts them into public life turns them into great leaders.

Why do we have this problem, and are there particular factors in the UK system that mean its political class is destined to fall short? There are structural reasons we can point to. First, what economists would call a classic ‘misallocation of capital’ problem: talented people are put off politics by the fact that it is poorly paid and increasingly low-status. The public dislike intensely the idea of paying politicians and civil servants more. A smart young person in their twenties would rather go into business or a profession where they can build high-status careers and gain esteem from their peers. If money is not a priority, they might be tempted into journalism or charity work. They are likely to live more comfortable lives and not endure the kind of hell-bent media scrutiny that is now a feature of political life.

With this talent drain, there is a doom-loop effect: the fewer good quality people become politicians, the lower status politics has and the less attractive it is to other good people. Britain’s hair-shirt approach to paying politicians, but also civil servants, inevitably narrows down the range of people who apply. It also means that politicians find it increasingly difficult to stay on. 132 MPs have stood down at the 2024 general election, including 75 Conservative MPs – the highest number since 100 Labour MPs stood down after the expenses scandal in 2010. Many face the near inevitable loss of their seat, but also the pressures of the job mean it is no longer as attractive to stick around.

This shrinking talent pool has been concealed for a long time by a growth in the number of politics courses in universities over the last 30 years. Young people are increasingly interested in politics for study.

However, the graduates of these courses tend to go on into the private sector – public affairs, corporate policy jobs in businesses, or non-government organisations. The Oxford PPE (Philosophy, Politics and Economics) course is no different, despite its infamous reputation as a pipeline for British politicians. And while the UK’s most prestigious institutions offer a world-class political education, it is noticeable how few UK students they teach. The student body at the Blavatnik School of Government in Oxford, which was set up over a decade ago with the aim of becoming the UK equivalent of the prestigious Kennedy School at Harvard, is composed of 80 to 90 per cent non-UK nationals. The London School of Economics’ student body is 70 per cent non-UK national. This attraction of global talent is tremendously important and lucrative, but to be clear, it is not resulting in better politicians at home. The UK is excellent at educating the world’s leaders but this is not the same as educating its own. These structural factors – low pay and low status – are real, but they are not inevitable. Why has society allowed public life to become so unattractive?

In the past, successful societies gave much thought to the question of what might be called ‘elite formation’. Just as any family or business might do succession planning, different tribes, nations, city-states and empires have found ways to identify the people who will carry forward the legacy of one generation to the next. One of the most famous stories of succession planning in Western culture is Moses handing over the leadership of the Israelites to his apprentice, Joshua. Just before they reach the Promised Land of Canaan, after 40 years of wandering, Moses is told by God that he will die before entering their new home as punishment for an earlier transgression. Moses, concerned for the fate of his people, pleads to God for a replacement, ‘so that the Lord’s community may not be like a flock that has no shepherd’. God commands Moses to appoint Joshua, who has effectively been training as an apprentice with Moses in the art of leadership and statecraft. Moses lays his hands on Joshua in front of the community, reassuring them that his successor is ‘full of the spirit of wisdom’.

As Adrian Wooldridge documents in his excellent 2021 book, The Aristocracy of Talent, different systems around the world throughout history have found ways of using meritocracy as a guiding principle, and creating institutions that would guide future leaders. In around AD 600, Imperial China introduced a national written exam, which was rigorous, fiercely competitive, open to anyone and acted as the gateway to a high-status career as a bureaucrat; by the seventeenth century, two and a half million people took the exam. Key elements still form part of the recruitment process for the Chinese civil service today.

In the modern era, European nations, and latterly the US, created a range of diverse institutions – grammar schools, public schools, universities – to teach the practical skills, as well as the moral character, that would be needed to cultivate future civic leaders. It is a well-worn cliché in Britain that the Battle of Waterloo was won on the playing fields of Eton, while in the US, the early Puritans founded the Ivy League universities (including Harvard and Yale) to train ministers. These went on to become the pipeline for future political leaders – Harvard alone counts eight presidents among its alumni.

So what has happened to this idea of elite formation in liberal societies? Arguably, something subtle has changed in the culture of these institutions and within the British elite itself since the 1980s or 1990s. Modern institutions today emphasise their ethos of meritocracy (even though they continue to have a highly socially privileged and white intake), but Wooldridge argues they seem to have lost a vital characteristic of elite formation, which is the idea of ‘noblesse oblige’. Students are no longer trained to think of themselves as custodians of a nation who ought to ‘give back’ in return for their status, but as highly talented individuals who have earned their position through intellectual superiority and therefore deserve to pursue their own ambition first and foremost. As one disillusioned Harvard graduate, Saffron Huang, wrote recently of her alma mater: ‘In place of elite formation is a production line of professional strivers – albeit ones with relative wealth and a valuable social network.’

In the UK, the business model of universities has underpinned this shift. In 1998 the government introduced tuition fees for students, which has shifted university funding from the state to individual ‘consumers’. Universities have responded by increasing recruitment of international students who can be charged higher fees. In marketing literature today, universities are less likely to present themselves as educators of a national elite, but as powerhouses for global talent. Undeniably, this globalising of higher education has many positives, but it is surely one reason why it is less steeped in concern for the future governance of the country.

Another profound change is in the British political parties themselves, which have cut back on their own outreach and education. In the 1950s, almost one in ten people in Britain were members of political parties. It is hard to imagine now, but political life was ingrained in wider civic life and it was far more common to meet party members through church, working men’s clubs, Rotary Club events and other local civic networks.

Both the Labour and Conservative parties had their own adult education colleges, which produced mass pamphlets and where thousands of party members and activists heard varied lectures on politics, economics and philosophy. They would have the opportunity to meet each other, as well as senior politicians. This vast social network has largely disappeared – there are now only around 500,000 members of political parties and the colleges have gone.

As a result, many people coming into elected politics today receive little education in public policy, economics or how government works. They arrive in Westminster expected to work their ideas out in isolation. There is no way for them to learn about the big challenges facing the country – from AI to geopolitics – except by their own volition.

The UK civil service shut down its prestigious national college at Sunningdale in 2010 as part of a wider cost-cutting exercise, and there have been complaints ever since that civil servants lack expertise. Taken in the round, these are extraordinary losses to the professional development of those running the country.

When Singapore became an independent country in 1965, it was lacking in natural resources, an independent military and even direct access to water. Prime Minister Lee Kuan Yew realised that excellence in public leadership would matter above anything else for his new country’s success. He made an early and explicit decision to hire on merit rather than ethnic quotas and to pay junior civil servants well to ensure they could attract talent. He was very explicit with the public about why. Today, the modern Singaporean civil service website foregrounds the word ‘excellence’ as one of its values and applications are highly competitive. The seriousness with which Singapore takes elite formation is astonishing.

By contrast, all the changes I have described in the UK have taken place almost without comment or concern in the last 40 or so years. We have become relaxed, even complacent, about the idea of elite formation. Could the reason for this contrast between us and Singapore reflect an even more profound difference? In Singapore, the very survival of the state has been in question since its early years of independence. Its sense of vulnerability has forced the construction of new institutions, a ruthlessness in selecting leaders and a sense of duty imbued throughout its culture. The UK has not experienced this sense of existential crisis for many decades. In the 1990s, elites believed they were living through the End of History and the End of Boom and Bust. They had beaten the twin evils of communism and economic decline.

In a multipolar world, with a stagnant economy and the prospect of geopolitical conflict forcing us into ever tougher choices at home, we must think about leadership again with the kind of seriousness it deserves. Like Singapore, we should worry much more about who is going to take over and treat it as a crucial element in the future success of the country.

This means institutions that are responsible for British elite formation thinking more consciously about how they can develop the discipline of statecraft and extend its spread to more people in politics and public life. It also means that those who care about this issue should – like social pioneers of previous eras – create new institutions that can take on this important work.