‘Spy’ is a word that is familiar to the general public, the media and in fiction, but is rarely used by governments or within the intelligence community, other than to describe publicly the activities of a hostile agent who has been unmasked. As commonly employed, it is in fact a rather imprecise umbrella term, used to embrace a range of intelligence-related activities. Those who work for one of the United Kingdom’s intelligence agencies are properly called intelligence officers; these officers may recruit and employ agents, at home or overseas, who work in the interests of the British Government to acquire intelligence. Agents of foreign powers or groups working against British interests are also called spies, as are defectors or double agents, who have worked or still work for another power but provide intelligence to the Government. For the purposes of examining the relationship between the spy and the state, the word ‘spy’ will be taken here to embrace all these categories. The common factor is the collection of intelligence by human beings—what is known as HUMINT— rather than by technological means (though such means may, of course, be used by spies).

Spies are employed to acquire intelligence in order to protect the state or to further its interests and objectives, whether that intelligence is gathered within the state itself, or from other countries, organisations or interest groups. The core business of intelligence agencies is principally to acquire secret intelligence by covert means. Given the enormous power the spy and the state of technological resources, a great deal of stress is often laid upon ‘open source’ material gathered from publicly available sources; although such methods and materials are important, they are complementary to the core business of obtaining secret information covertly.

However it is acquired and however important its content, intelligence is only worth having if use can be made of it by the state. Someone must validate the intelligence, assess and analyse it and present it to the relevant authorities in a way that makes it useful. Intelligence customers also have a role to play in commenting on the reports they receive, an exercise that helps the originating agencies to readjust their efforts. This is an iterative process, reflecting the fact that it is the state that tasks intelligence agencies according to its requirements: the agencies are a part of the machinery of government. Intelligence is only ever a part of the picture when it comes to the formulation of government policy, and it is rarely decisive. But in the wide range of considerations that must be taken into account by a government when deciding how to react to events, to anticipate future developments or to develop long term strategies, intelligence is a piece of the jigsaw that can transform the picture and make it comprehensible, joining the pieces that do not otherwise seem to fit together.

What, then, is the relationship between the spy and the state? Some light can be shed on this complex issue by the history of the use of spies by the state in Britain, focussing particularly on the twentieth century, after the formation of the principal intelligence agencies.

Spies and spymasters

British rulers, governments and military authorities have always used espionage as a tool of statecraft. Henry VII set up a personal espionage system to protect himself after his victory at Bosworth Field in 1485; Henry VIII, Elizabeth I and James I all used secret funds to employ agents overseas, to bribe officials at home and abroad and to defeat conspiracies against the state. Monarchs and ministers very rarely dealt with spies themselves, for reasons of secrecy and deniability and to protect the integrity of sources and methods. Spies were handled by intermediaries, rather than by the principals: spies, in short, needed spymasters. Gradually, in the eighteenth and nineteenth centuries, these intermediaries became institutions rather than individuals. After the formal establishment of the Foreign Office in 1782, British diplomats serving overseas found themselves drawn into clandestine as well as representational duties, while both the army and navy developed intelligence capabilities of their own.

It was, however, the prospect of war with Germany in the early years of the twentieth century that led to the formation of the Secret Service Bureau in 1909, which divided after a year into two branches, one responsible for domestic and one for overseas intelligence—what we know today as MI5, or the Security Service, and the Secret Intelligence Service (SIS), later known as MI6. The intention behind the establishment of these agencies was not just to acquire intelligence, but also to provide a screen between those who commissioned it in pursuit of national objectives—the state—and those responsible for providing it—the spies. MI5’s priority was counter-espionage, directed against Germany in the few years before the First World War. For MI6 (or MI1(c) as it was then known), the priority was to recruit agents to gather intelligence overseas while participating in military movements and other preparations for war on the European continent.

The first two heads of the domestic and foreign agencies were Captain Vernon Kell and Captain Mansfield Cumming. Both had to invent themselves as spymasters, starting from scratch with very limited resources, and although their remits differed they shared common problems: how to recruit spies, whether officers to work in their embryonic organisations or agents to provide intelligence; how to judge the reliability and the quality of their intelligence, how to develop the administrative systems they needed to collate and assess that intelligence; and how to interact with their military and political masters to ensure both their own survival and that their efforts were put to some use. Between 1909 and 1914 both agencies were tiny, but they expanded hugely during the First World War, increasing their effectiveness and impact but also highlighting the problems they faced in dealing with spies of all nationalities.

The First World War was a formative period for British espionage, and many of the organisational structures, techniques and procedures developed during that conflict were to be used in the Second World War and beyond. In addition to MI5 and SIS, the First World War saw the institutional birth of the third and very significant arm of British intelligence: that concerned with codes and cyphers. The ground-breaking work carried out by Naval Intelligence’s Room 40 and the War Office’s 40 section MI1(b) during the First World War was, of course, a different form of spying—SIGINT (signals intelligence) instead of HUMINT. But the people who worked within those sections, and those who directed their activities, such as Director of Naval Intelligence Admiral ‘Blinker’ Hall, fall into the wider definition of spies. So successful were their activities that in 1919 the two code-breaking sections were joined into one agency, the Government Code and Cypher School (GC&CS, precursor of the Government Communications Headquarters, GCHQ).

The shape of the British intelligence establishment that emerged from the 1914-18 conflict was thus very similar to the present day intelligence architecture. And although the last century saw major changes in the global context and variations in the size, operational remit and public visibility of the agencies, many of the problems faced by the early spymasters and spies remained constant. These include both the work of the spy and spymaster and the interface between the spy and the state. The issues are not unrelated: if the state is to make use of intelligence, it must have confidence in its authenticity and reliability, imposing a responsibility on those who recruit the spy and those who assess the finished intelligence. At the governmental level, this means trusting the heads of the intelligence agencies and thereby what they produce. In the First World War, Mansfield Cumming struggled to maintain the trust of military leaders, who were disposed to resent what they regarded as an intrusion into their domain, while he retained, on the whole, the confidence of the Foreign Office. In the Second, Churchill trusted SIS Chief Stewart Menzies, not least because he brought him daily batches of decrypts from Bletchley Park; but loss of trust in MI5’s effectiveness led to Kell’s retirement in 1940. Anthony Eden’s confidence in his intelligence chiefs was shaken by their handling of an incident in 1956, when Commander ‘Buster’ Crabb disappeared while on a covert diving mission in Portsmouth Harbour. Edward Heath’s lack of regard for MI5 Director-General Martin Furnival Jones made him less disposed to take seriously reports on the scale of Soviet espionage in the UK, until convinced by Foreign Secretary Alec Douglas-Home.

These random examples show how at the governmental level the relationship between the spy and the state can be affected by personal predilections and prejudices, as well as by professional competence. Some British Prime Ministers, like Asquith or Heath, have had little interest in intelligence; others, like Churchill, have valued it highly; the majority have recognised the need for it, but prefer (like Macmillan, for example) not to have too close an involvement with it. In peacetime, as for example in the interwar period 1919–39, the heads of the intelligence agencies often found it difficult to bring their reports and suggestions to ministerial attention; Stanley Baldwin, who served as Prime Minister on three occasions in the 1920s and 1930s, was wary of secret intelligence and preferred to keep it at arm’s length. The development of interdepartmental machinery, like the Joint Intelligence Committee set up in 1936, provided a channel for coordinating intelligence and submitting it to ministers, but it was not until wartime that it became properly effective.

It is not surprising that the relationship between the intelligence agencies and military authorities was also closer and more harmonious in wartime, when the security and survival of the state provided a common motivation. By the same reasoning, governments pay more attention in wartime to intelligence as a tool of both strategy and tactics. When a period of conflict comes to an end, as in 1918, 1945 and 1990–91 at the end of the Cold War, there is a tendency for the state to try and cash in on its ‘peace dividend’, reducing (and often reorganising) its intelligence establishment in the soon-frustrated hope that it no longer needs so many spies. At the beginning of the 1945–51 Labour Government, Attlee and Bevin were determined to eschew the use of special operations in pursuit of their foreign policy—a determination that was soon dissipated by the realities of dealing with an aggressively expansionist Soviet Union. In the end, governments always need to make use of secret intelligence—of spymasters and spies—in pursuit of their objectives of defending the state and its interests, at home or overseas, and of promoting their policies. Even governments in which ministers were initially wary of the work of the agencies, such as Harold Wilson’s Labour Government of 1964–70, soon found a way of working harmoniously with the intelligence establishment and using its product effectively.

What are spies made of?

In order to produce reliable and insightful intelligence that will be useful to the state, spymasters must employ the right kind of people. The recruitment, handling and reliability of those employed as spies, whether as intelligence officers or agents, are always difficult and sensitive matters. ‘In its essential form the work of a British spy has barely changed. It involves persuading someone to betray secrets, a deeply personal, even intimate act, and one fraught with risks.’ The challenges faced by early spymasters Cumming and Kell were not very different from those faced by their successors: how to recruit the ‘right kind’ of people, inspired by the right kind of motives, who could be trusted with the secrets of the state and to collect the necessary intelligence; and how to deal with people who offer their services, rather than being recruited proactively.

Motivation is of course a complex issue. During the First World War, most intelligence officers joining MI5 or SIS transferred from the military, often because they were barred from active service for some reason, or had been invalided out. For example, Sir Samuel Hoare (later Viscount Templewood), a Conservative MP, was prevented by illness from serving at the front. Having learned Russian while serving as a recruiting officer in Norwich, he joined SIS and ended up heading missions to both Russia (where he was the first person to report the death of Rasputin) and Italy. The successful author Compton Mackenzie, invalided out of active service in 1915, was recruited to run counterespionage operations in the Aegean. Many officer recruits to MI5 had also been invalided out or were unfit for active service, as a wartime cartoon showed: ‘Oh, they knocked a piece out of my skull, so they sent me to the Intelligence Department.’ Intelligence officers recruited in this way might be very capable and highly motivated, but might also suffer from both physical and mental health problems. Both Kell and Cumming found some of their staff needed a good deal of pastoral care. Not all their recruits lived up to the description of the ideal intelligence officer: ‘He should be a gentleman, and a capable one, absolutely honest and with considerable tact and at the same time force of character . . . experience shows that any amount of brilliance or low cunning will not make up for the lack of scrupulous personal honesty. In the long run it is only the honest man who can defeat the ruffian.’

Being a spymaster, however, often meant dealing with ‘ruffians’, whether at home or overseas. The motivation of spies was often complex. Some sought excitement; some were bitter from a real or imagined setback in their lives or careers; some were driven by ideology; most wanted money; often, they were motivated by a mixture of all these. For example, the novelist Somerset Maugham was recruited partly because his occupation offered excellent cover, but he was also personally motivated by the search for excitement (though he wrote accurately of the tedium and fear of being a spy), and later by the wish to escape a rash marriage. Aaron Aaronsohn, a fervent Zionist, offered his network of spies in Palestine and Syria to the British in the hope of securing support for his cause. The SIS agent codenamed TR16, Dr Karl Krüger, was a German shipbuilding engineer embittered by an earlier court martial and greedy for money, who produced valuable reports monthly until January 1919, touring shipyards and reporting on German losses. TR16 was handled ably by Captain Richard Tinsley, a former merchant navy officer described by Army intelligence as an experienced blackmailer, ‘a liar and a first-class intriguer with few scruples’. But Tinsley also turned out to be a first class case officer and organiser until he was exposed as a British spy in 1916. TR16 was then handed over to Captain Henry Landau, a Cambridge-educated linguist with a natural aptitude for intelligence work— although it was reported that some took ‘a great dislike to him owing to his somewhat furtive manner’.

MI5 also had its share of unusual recruits, such as William Hinchley Cooke, who was bilingual in German and English and on the outbreak of war was working in the British Legation and was expelled with the rest of the staff. Though the London authorities mistrusted his German accent, Kell vouched for him (‘He is an Englishman’) and he became an extremely successful officer, arresting a number of German spies and serving in the Second World War as well as the First. MI5 also employed a large number of women, principally for registry work, some of whom were outstandingly able. While Cumming was chiefly concerned with recruiting spies overseas, Kell’s principal focus was initially on identifying German spies in the UK (at which MI5 was extremely successful), and later on counter-subversion, at home, in Ireland and in the Empire as a whole. The First World War also saw early examples of deception and the use of double agents, anticipating the enormously effective Double Cross system that saw almost all German agents ‘turned’ during the Second World War. In the naval and War Office cryptographic sections, it was the particular and peculiar skills of the members that made the units so successful: civilian mathematicians, philosophers, linguists, and brilliant intellectuals who needed just as much careful handling—by men like the Director of Naval Intelligence Admiral ‘Blinker’ Hall—as those with whom Kell and Cumming dealt. Some of those responsible for the major successes of First World War SIGINT repeated their achievements in the Second.

Indeed, all the intelligence agencies’ development and experience acquired in the First World War were to be the foundation for their work in the Second World War. During the lean interwar years, lessons learned about recruitment, agent handling and organisation had not been forgotten and were ready to be revived. There was also continuity in the challenges posed by reliability and motivation. Some of those recruited by SIS in the 1920s, for example, conform more nearly to popular images of the spy: larger than life characters like Sidney Reilly, Paul Dukes, Augustus Agar and Malcolm Maclaren. As Alan Judd has commented, ‘The myth of James Bond owed much of its potency to the facts of such lives’; but despite the difficulty of dealing with some of these figures, whose motivation was a mixture of greed, patriotism and thrill-seeking, they still produced valuable intelligence. Similarly, during this period MI5 was to recruit some of its most valuable officers and agent-handlers of the Second World War.

Again, however, it was the advent of war which provoked a major recruitment drive and led to the employment of a large number of ‘spies’, some low-profile and brilliantly methodical, others more colourful and hard to handle, but all needed in the service of the state at a time of supreme national effort. The three agencies were augmented during the Second World War by a wide range of ‘secret shows’, organisations involved either in the collection or interpretation of intelligence, the spreading of propaganda, or in sabotage and subversion. All these bodies employed people now commonly referred to as spies: a category that could embrace military planners in the War Cabinet Offices; specialists in technical areas such as photography or explosives; bilingual housewives dropped into Occupied Europe to liaise with resistance networks; undergraduates in mathematics or classics, or winners of crossword puzzle competitions talent-spotted for code-breaking operations at Bletchley Park and elsewhere. Spymasters faced the same problems as in the First World War in assessing reliability and effectiveness, but on a much larger scale. Inevitably, there were successes as well as failures, reinforcing the lesson that the most erratic or unpromising characters may prove to be good spies, while others who seem supremely well suited to the job may prove not just unreliable but even treacherous. Often, the distinction is impossible to discern in advance.

As in the First World War, the conflict of 1939–45 was to inspire many individuals to offer their services to the British state, from a mixture of motives. The man who has been described as the greatest double agent of all, Juan Pujol (GARBO), was motivated by a dislike for all extremism and for war to offer his services to the British Embassy in Madrid in 1941. Together with his gifted MI5 case officer Tomas Harris, an Anglo-Spanish art dealer who counted Guy Burgess, Kim Philby and Anthony Blunt among his friends, GARBO invented a web of fictional agents that convinced the Germans he was their principal agent in the UK. Others were motivated by more pragmatic considerations. Sir William Stephenson, a Canadian businessman who ran a commercial intelligence network during the interwar years, offered his services to SIS in 1939 when the increasingly threatening European situation was affecting his enterprises. He went on to work for SIS as the head of British Security Coordination (BSC), based in New York and providing intelligence from across the Americas, while promoting a useful personal relationship with FBI Director J. Edgar Hoover and Colonel ‘Wild Bill’ Donovan, who was appointed US Coordinator of Intelligence by President Roosevelt in 1941. And while Stephenson was undoubtedly prompted by commercial motives, he also relished the excitement and power that a wartime career in espionage brought him.

Wartime intelligence offers almost inexhaustible scope for examples of spying, good and bad, heroic or low-key, opportunistic or carefully plotted. Spying in the Cold War period, though rather less fully documented on the whole, faced similar challenges, both operationally and in terms of the relations between spies and the state. Neither space nor available evidence permit a full review of the later twentieth century, but mention should be made of the role played by a rather different sort of spy: those who take the difficult decision to betray their own governments, often from very complex motives. The intelligence provided by these spies can be of particular value to the state in times of crisis, for example Oleg Penkovsky at the time of the Cuban Missile Crisis, and Oleg Gordievsky, who made an outstanding contribution in the 1980s to British understanding of the Soviet Union. The intelligence provided by defectors like Igor Gouzenko in 1945, Oleg Lyalin in 1971 and Vasili Mitrokin can also provide valuable insights into the working of hostile intelligence bodies.

Even when the evidence available is incomplete, we know enough to say that the problems and issues faced by the earlier practitioners of intelligence are enduring ones; they may be tackled differently with modern tools and methods, but will not disappear. In the past thirty years, more and more documentation on the work of the intelligence agencies has become available: both MI5 and GCHQ have released records into the public domain, while SIS records on the files of other departments are also reviewed for release. The Foreign and Commonwealth Office has released the records of the Permanent Under-Secretary’s Department (PUSD), responsible for liaison with the intelligence agencies, up to 1951; and intelligence records in the Cabinet Secretary’s personal, miscellaneous correspondence have been released up to 1960. There is today more intelligence-related evidence available to the researcher than ever before. This evidence is already producing a rich and continually expanding seam of secondary literature, based on British and other sources, further illuminating the role of the spy and the relationship between espionage and the state that employs it. There will be much more to come.

Some commentators take the view that the role of the spy (or HUMINT) has been devalued by the power, scope and speed of technological intelligence gathering, the role of big data, and the availability of open source material. Even those who believe that intelligence agencies are overreliant on technical collection methods and cyber warfare (defensive and offensive) tend to take this line. But most intelligence professionals will agree that however powerful the technical tools, there is no substitute for human intelligence. HUMINT remains the ‘coin of the realm’ for intelligence agencies. Since intelligence is only worth having if use can be made of it, there can be significant problems in isolating and identifying relevant indicators from the vast amount of information collected by technical means. Success in doing so also depends on the skill and experience of those in charge of its collection and assessment: another sort of spy. In the complex twenty-first-century global context, all-source intelligence is required, as well as effective liaison with partners and allies, if the state is to have the information it needs to keep it safe and protect its interests.

But however powerful the technical tools, there is no substitute for having a human source—a spy—with access to the counsels of organisations or governments in whose intentions, potentially or actually hostile, a state may be interested. For historians, the concept of the spy and the state is likely to remain a focus of interest for the foreseeable future.