There are many explanations for the discontent in Western societies that has led to the current political instability – from inequality and the delayed impact of the financial crash to politicians deemed incompetent and aloof.

But there is one overarching explanation that encompasses most of the others: cognitive ability – the analytical intelligence that helps people to pass exams when young and process data efficiently in their professional lives – has become the gold standard of human esteem, and cognitive elites have come to shape society too much in their own interests. To put it more bluntly: smart people have become too powerful.

Sixty or 70 years ago, when we lived in somewhat less complex societies, the people who ran government and business were generally brighter and more ambitious than the average – as they still are today – but it was a time when skills and qualities other than analytical intelligence were held in higher esteem.

Today the ‘brightest and the best’ trump the ‘decent and hardworking’. Those other qualities like character, integrity, experience and willingness to toil hard, are not irrelevant but they command less respect.

A good society is one with a proper balance between the aptitudes of ‘head’, ‘hand’ and ‘heart’. The modern knowledge economy, however, has produced higher and higher returns to the highly qualified and reduced the relative pay and status of manual and caring jobs. An economic system that once had a place for those of middling and even lower cognitive abilities – in the semi-skilled jobs of the industrial era, on the land, in the military – now favours the cognitive elites and the educationally blessed. Other institutions that have stressed aptitudes other than cognitive ability have been in sharp decline across most of the West and especially in Europe. The decline of religion, family life, the military and traditional industrial employment, along with the increased demand for analytical and numerical skills in the computer and then digital age, has elevated cognitive ability above more traditional virtues and aptitudes. Think of Silicon Valley versus Sandhurst.

Moreover, for most of human history cognitive ability has been more or less randomly scattered through society. But in recent decades a huge sorting process has hoovered up the young exam-passers and sent as many as possible into higher education, leading in Britain to a 400 per cent increase in student numbers since 1990 and a precipitous decline in the prestige of so much non-graduate employment.

This does not mean that we live in a true meritocracy. Family income in childhood is still highly correlated with educational success. This has been reinforced by something described by the ugly phrase ‘assortative mating’, meaning that people in high status jobs requiring high cognitive ability are far more likely to pair up with similar people.

The children of these couples do not form a genetic elite of the highly able but they are far more likely to be brought up by two parents who are both well connected and understand what is required for children of even middling ability to enter good universities and higher professional jobs. They could thus be said to form a kind of hereditary meritocracy.

But surely modern, technological societies simply need more clever people, and so long as some of the biases just described can be ironed out and, through more spending on education, people from all backgrounds can have a fair crack at joining the cognitive elite, then all is well? I don’t think so. In the tradition of Michael Young’s Rise of the Meritocracy, Daniel Bell’s The Coming of Post-Industrial Society and Charles Murray’s Coming Apart, I believe that today’s achievement society has replaced one system of domination by another.

From the point of view of the efficiency of society, cognitive ability plus effort is a better selection criterion for high status jobs than inheritance of land or capital. But it is not necessarily any fairer or more humane. As Young pointed out 60 years ago, people blessed with advanced cognitive skills often feel less obligation to those of below average intelligence than the rich have traditionally felt towards the poor.

It is one of the most difficult balancing acts of open, modern societies, though one that is seldom articulated: how to constrain meritocracy and prevent a disproportionate degree of status and prestige (and financial reward) flowing to high cognitive ability jobs – and away from the ‘hand and heart’ jobs that are still so vital – without at the same time disincentivising the most able and ambitious people in our society.

The pleasure of mastering a task and performing it well is available to people of all levels of ability. It is inevitably the case that more complex and difficult tasks, such as designing a building or helping to invent a new drug, will receive, and deserve to receive, more esteem and reward than being a postal worker or an office cleaner (albeit, a significant proportion of high qualification jobs appear less useful and productive than many low qualification jobs).

A successful society must, however, manage the tension between the inequality of esteem that arises from relatively open competition for highly skilled jobs and the ethos of equality of esteem that flows from democratic citizenship. Economic inequality versus political equality.

To put it another way: an achievement society that wants to avoid widespread disaffection in the democratic age must sufficiently respect and reward achievement in the lower cognitive ability hand and heart jobs and provide meaning and value for people who cannot or do not want to achieve.

In the current age of disruption it seems clear that we have not been getting the balance right. Many people on the Left see this as mainly about income and wealth inequality.

Inequality has not, in fact, been rising sharply in many of the countries, including Britain, where there has been the biggest push-back against the status quo. It is true that slow or non-existent wage growth is harder to bear when a small minority, most notably bankers, seem insulated from austerity. But this misses an even bigger story about esteem and how valued you feel in the social order.

For we have often almost unwittingly come to confuse cognitive ability with human value and human contribution more generally. There is no reason why people who complete certain mental tasks more efficiently than others should be regarded as more admirable people.

Yet there is a clear trend in modern liberal politics to tell us that this is indeed the case. High cognitive/analytical ability and success in the knowledge economy is highly correlated with support for the modern virtues of openness, mobility and hostility to tradition – the opposite of parochialism and insularity. These virtues are also dominant in the expanded higher education sector of modern societies.

This was one of the themes of my book, The Road to Somewhere, in which I described the value divides in British society, revealed starkly by the Brexit vote, that have been exacerbated by this narrower focus on cognitive ability.

On the one hand is the group I call the ‘Anywheres’, making up about 20 to 25 per cent of the population, who are well educated (mainly with high cognitive ability) and usually live far from their parents, tend to favour openness and autonomy, and are comfortable with social fluidity and novelty.

On the other hand is a larger group of people, about half of the population, I call the ‘Somewheres’, who are less well educated, more rooted and value security and familiarity. They place a much greater emphasis on group attachments (local and national) than the Anywheres. Anywheres are generally comfortable with social change because they have so-called ‘achieved identities’ – a sense of themselves derived from educational and career achievements, which allows them to fit in pretty much, well, anywhere. Whereas Somewheres have ‘ascribed identities’ based more on place or group which means that their identity can be more easily discomforted by rapid change to those places.

The Anywhere/Somewhere divide does not always map directly onto the high cognitive ability/modest cognitive ability divide (there are, of course, Anywheres of average cognitive ability and Somewheres of high cognitive ability). But there is considerable overlap, not least measured by the cognitive element in the head jobs that Anywheres tend to do, and the hand and heart jobs that Somewheres tend to do.

It is certainly the case that Anywhere values and priorities, which have come to dominate modern politics and all mainstream political parties, tend to coincide with the values and priorities of those with high cognitive ability. And the Anywhere answer to everything from social mobility to improved productivity has been the same: more academic higher education.

Yet as David Lucas, the children’s author and illustrator, has persuasively argued, society needs the cognitive skills of the knowledge economy but we also need the craft skills of artisans and tradesmen, and the emotional intelligence of those in caring jobs.

Hand and heart skills have become chronically undervalued in the modern world, unbalancing our societies and alienating millions of people.

Everyone is in favour of social mobility – and bright people from whatever background travelling as far as their talents will take them – but today’s British Dream has become too narrowly defined as going to university and into a professional job. This is not surprising when more than 90 per cent of MPs are graduates.

In Britain there has been some attempt in recent years to offer other options to school leavers with improved apprenticeships and T-levels. But they cannot compete with the prestige of the university route, leaving our economy starved of essential workers – last year fewer than 10,000 young people started proper construction apprenticeships while half of the building workers in London are from abroad.

Meanwhile heart jobs in social care, parts of the NHS, early years education and childcare continue to be undervalued (and paid) because they are roles that used to be performed for free in the private realm of the family, mainly by women. The modern women’s equality movement has focused on breaking glass ceilings and equal competition with men in higher professional jobs and has rather looked down upon jobs and roles associated with traditionally female aptitudes for caring and nurturing. Hence, in part, the crisis in social care and in nurse recruitment.

We are encouraged to live increasingly ‘head’ lives, reinforced by most advances in technology that reduce opportunities for craft, and the need for human contact or attachment to specific places. By contrast, it is the relatively undervalued skills of hand and heart that promote belonging and attachment. (The modern obsession with cooking, on television and in the newspapers, may reflect the continuing desire for useful hand work on the part of so many people who no longer use their hands at work.)

All across the developed world, the one quality-of-life indicator that is going down is mental health. Mental well-being depends on a sense of meaning and purpose. We create meaning and purpose through attachment, belonging, stories, love. And through transcending a narrow sense of ourselves – the most powerful route to meaning is through self-sacrifice: giving to others.

There is nothing more spiritually rewarding than caring for others, or making things with your hands. It is the pleasure of belonging, of attachment, the pleasure of being embodied, in this place and time, the pleasure of being much more than a disembodied intellect, a brain in a jar.

Yet abstraction and detachment dominate our culture. Consider the ethos of the digital giants like Google and Facebook. And joining the world of cognitive achievers generally means detaching yourself from your roots. Justine Greening, Britain’s former secretary of state for education, said this in a speech in 2017: ‘All the years I spent growing up in Rotherham I was aiming for something better…a better job, owning my own home, an interesting career, a life that I found really challenging… I knew there was something better out there.’

The unselfconscious way in which a cabinet minister doubts whether it is possible for an able, ambitious person to live a fulfilled life in a town of 120,000 people (a 30-minute commute from Sheffield) reveals something flawed about modern Britain.

Many Somewheres cannot or do not want to leave their roots and join the Anywheres, and, in any case, half of the population will always, by logical necessity, be in the bottom half of the cognitive ability spectrum. Yet all of us need to feel we have a valued place in society even if we are not mobile, high achievers.

Anywheres, because they are generally more articulate and better trained in sifting evidence, flatter themselves with the belief that their values flow from reason and evidence and for that reason are morally superior. In fact they are generally just as subject to groupthink as Somewheres, as the UK’s environment secretary Michael Gove pointed out in his famous comment about experts (during the EU referendum campaign of 2016, he suggested that Britons had ‘had enough of experts’).

The Anywhere political class has ruled too much in its own interests, ignoring some of the basic political intuitions of the Somewheres: the importance of stable neighbourhoods and secure borders, the priority of national citizen rights before universal rights, the need for narrative and recognition for those who do not easily thrive in more head-based, cognitive ability-favouring economies.

And, in Britain, this lack of empathy for the Somewhere worldview has now left us with the Brexit backlash and a country more divided than at any time since the 1970s. Is that not ample evidence of the limits of cognitive ability? So how can a better balance between the ‘Three Hs’ be achieved? Markets do not simply respond to supply and demand but can also reflect the underlying value and priority that society places on different activities. Political pressure from those who do not share the interests of the higher cognitive elites will help to drive change. And there are several trends that suggest the head could be about to face a more even contest with hand and heart. A dystopian one suggested by Nicholas Carr in his book The Shallows is that we are all going to be made dumber by the internet. Carr argues that sustained exposure to the internet is reordering our synapses and making dependent on constant novelty. This may bring improvement in some fields such as decision-making and problem-solving but overall it will mean significant losses in language facility, memory and concentration. (Recent evidence that the Flynn effect of consistently rising IQ levels is going into reverse could offer corroboration of Carr’s thesis.)

There are two more positive trends that could help to boost both hand and heart.

The potential boost to hand, in the UK at least, comes from a combination of changes to employment patterns as a result of Artificial Intelligence and changes to immigration patterns as a result of Brexit.

Technological change and more open markets have mainly impacted middle and lower cognitive jobs in the past 40 years, causing a sharp decline in skilled industrial labouring jobs. But the next wave of AI is expected to have its most dramatic impact on high cognitive jobs (at least at the lower end) in accounting, law, medicine and so on.

This disruption felt by well-educated Anywheres could lead to a new sympathy for people performing relatively routine, hand and heart jobs, partly because the former accountants and lawyers may find themselves doing those jobs themselves.

At the same time, because of the widespread push-back against historically high immigration flows, in Britain and elsewhere in the developed world, it should become harder for employers to hire already trained people from other countries, especially in the middling technical and skilled manual jobs. As a consequence the training opportunities and the pay for such jobs ought to rise.

In some sectors such as IT and construction there may even be a ‘your country needs you’ drive to replace immigrants, raising the visibility, prestige and pay of such jobs.

The final benign trend, raising the status of heart jobs, is linked to two big and irreversible trends: the increased number of old people who will need significant levels of care in their final years and the rising power of women.

The #MeToo movement exposing the predatory behaviour of men in the entertainment industry, politics and elsewhere was welcome not only because of the constraint it helped to place on such behaviour but also because of the expression of female power that it represents. #MeToo would have been impossible 30 years ago because there were simply too few women in positions of authority in the media and politics.

One of the most important questions in British society over the next generation is whether the increased status of women in society will drive significant increases in the pay and prestige of caring (heart) jobs that are still mainly performed by women.

Could #MeToo mean a £20 an hour minimum wage in social care in the foreseeable future? Or are #MeToo women – educated upper level professionals – concerned only with a level playing field with men in high cognitive ability jobs? Are they too detached from the interests of more traditional women who work part-time in a care home?

To conclude: the combination of the rise and rise of cognitive ability, and head jobs, as a measure of economic and social success, and the hegemony of liberal (Anywhere) political interests, has led to the unbalancing of Western politics. The disaffection of large minorities, even majorities, in many countries, is intimately linked to the declining prestige of hand and heart roles. The stress on economic inequality does not properly capture this disaffection.

What are the lessons of history? Was there a better head, hand and heart balance in the past? What about the Victorian era that valued brains but placed more stress on character and virtue? Does that provide a model? Cognitive ability will remain central to the advance of human civilisation, and all societies will want to continue to nurture it. Can cognitive elites, and the liberalism they tend to espouse, be politically constrained without causing economic and cultural damage? What can the cognitively blessed reasonably expect, and how much power will they willingly concede?

Head work is not destined to become less important in our knowledge-based economies, but there are signs that the hand and the heart will be restored to positions of greater respect. Politics is, in fact, already demanding such a rebalancing.

This essay originally appeared in ‘Knowledge and Information – Perspectives from Engelsberg Seminar, 2018’, Bokförlaget Stolpe, in collaboration with the Axel and Margaret Ax:son Johnson Foundation.