When I was growing up in Edinburgh in the 1950s, my father would depart for his office clutching a battered, brown leather briefcase. The briefcase, he told my brother and me, had accompanied him on a Commando raid in the Second World War. In September 1939 my father was reading modern languages at Cambridge. He had spent part of 1938 in Berlin and his German was fluent and colloquial. Called up by the Army, he went to the Military Equitation School in January 1940 to train for a small mounted unit intended to collect intelligence behind enemy lines — presumably a sort of Long Range Desert Group on horses rather than in trucks. During the Battle of Britain, he exchanged his horse for a motorbike and travelled between prisoner-of-war camps, interrogating shot-down German aircrew. In February 1941 he was sent overseas for the first time, as an intelligence officer on a raid to destroy the cod liver oil factories on the Lofoten Islands, off the coast of Norway. Those on the raid were told that the fish oil was for making bombs, not for consumption by growing babies. Lord Lovat, the Commando leader, came along for the ride and, according to my father, brought his stalking rifle; my father had his briefcase.

Shortly before he died in 2000, I asked my father what the purpose of the raid had been. He repeated the same story: to destroy the factories, a mission that had been successfully accomplished without loss of life. I told him he was wrong. The British had captured an Enigma machine, whose settings were changed every day in order to encrypt German signals, and — even more importantly — its enciphering tables. The coup had enabled Bletchley Park to begin to read German naval traffic. My father was not given to blasphemy, but his face expressed astonishment. I was amazed for a different reason. His credulity had trumped his rationality. He was an intelligence officer and in 1943 he would go on to work in MI (Plans) in North Africa, processing German signal intercepts into actionable information. In order not to reveal their sources, the British could not immediately attack Axis convoys sailing across the Mediterranean but had to give the enemy good grounds for attributing their responses to more obvious methods of detection, frequently aerial reconnaissance. This much my father had divulged to me in 1974, the year in which Frederick Winterbotham, a wartime MI6 officer, had revealed the prime source of allied intelligence for the first time in The Ultra Secret, so breaching a confidence which many, my father included, had kept for 30 years.

While the Commandos destroyed the fish factories, rounded up quislings and looted useful equipment, my father — or so I fancy — went to the telegraph office and put everything he could find into his briefcase. He gave me a copy of Heinz Guderian’s pamphlet, Die Panzertruppen und ihr Zusammenwirken mit den anderen Waffen (Mechanised forces in employment with other arms), which he acquired in this sweep and which either he had opted not to hand over on his return or his masters had concluded was now out of date (it was published in 1937). He may even have seized a codebook and so helped fulfil the raid’s objective but, if he had, he remained unaware of it. His gullibility was crucial to the operation — if he had been captured and interrogated, he would not have been able to reveal information he did not possess.

My father’s Lofoten briefcase lies at the intersection of the impact of new technologies and their role in wartime intelligence with older and more human methods of deception and influence. When Carl von Clausewitz wrote On War in the aftermath of the Napoleonic Wars, he was sceptical of intelligence’s value. Much of it rested on rumour which could not be tested and was liable to exaggeration. It was also probably out of date. The information which had proved so useful to Napoleon himself was topographical. Accurate mapping enabled him to plan his lines of march, to anticipate the possible loss of time created by obstacles such as rivers and forests, and to find viable routes so that his armies could threaten alternative objectives. But in many contexts, not least those of nineteenth-century colonial warfare, armies were like explorers, surveying as they went and collecting anthropological and ethnographic information, which would underpin the assumptions of empire. Technology played a key role in cartography, just as it did in the production of naval charts, but it only provided the broad contours of a campaign. The acquisition of tactical and time-sensitive intelligence was rudimentary. A cavalry patrol would struggle to penetrate far into enemy positions and was in danger of revealing its own commander’s intentions as it did so. Finding the enemy was even harder at sea, where unpredictable and variable weather could enable fleets to hide as effectively as did the expanse of the oceans.

At the beginning of the twentieth century, wireless revolutionised the potential applications of intelligence in war. By transmitting directives through the ether, it conveyed a commander’s intentions in a format which was publicly accessible, and did so in real time. To minimise these dangers, messages were relayed in code, but the processes of enciphering and decoding themselves took time and so undercut the gains in speed. In 1914, during the last week of the July crisis which led to the outbreak of the First World War, governments could sometimes intercept each other’s diplomatic traffic but still struggled to do so fast enough to get inside each other’s decision-making loop. A signal sent from an ambassador to his head of state at home had to be enciphered, transmitted and then decoded before it arrived on the statesman’s desk. In a fast-changing situation, the statesman at home might be reacting to yesterday’s events while his adversary might have learned his ambassador’s views through intercepts at least as quickly as he did.

When the war broke out, tactical intelligence was frequently too time-sensitive to permit such delays. As armies manoeuvred across Europe in August 1914, speed could trump security in the transmission of information and intentions. Units, most notoriously within the Russian 2nd Army in its invasion of East Prussia, communicated with each other in clear speech, so providing the enemy with real-time intelligence that, in this case, contributed to the defeat at Tannenberg. But they did so in the hope that, even with this advantage, the enemy would be too late in his responses. Ciphers in any case did not provide full security; any wireless traffic indicated enemy activity, especially when it increased in quantity, and proximity as its volume intensified. The only perfect way to avoid detection was to observe radio silence but that, in turn, incurred penalties. At Jutland on 31 May 1916, David Beatty used flags to signal to his battle cruisers, with the result that orders were missed or misinterpreted in the smoke. The British Grand Fleet was given the chance to intercept the German High Seas Fleet because German ships chattered to each other with helpful volubility, but, once it was at sea, British naval intelligence were fearful of sending its com- mander, Jellicoe, raw intercepts and, as a result, he lost his moment to achieve a ‘decisive’ victory.

For much of both world wars, British naval intelligence was able to read German signals. By the end of 1914 the Royal Navy had secured three German code books. In 1915 SMS Königsberg, a German light cruiser blockaded by British ships in the Rufiji delta in East Africa, thought its signals were being read, but its warning was not taken seriously; this was despite the fact that in March 1916 a blockade runner which observed radio silence managed to deliver supplies to the German forces in East Africa when one that did not failed. The High Seas Fleet attributed its encounter with the Grand Fleet at Jutland to bad luck, not to poor wireless discipline.

In the Second World War, the British again struck lucky. The Poles had acquired an Enigma machine and they brought it to Britain. Enigma enabled both speed of transmission and secrecy. Although it was compromised from early in the war, it took time for the British to be able to read Enigma signals (using an early form of computer) and there were periods when the British were effectively locked out. Despite what would seem, in hindsight, to be mounting evidence, Germany’s faith in the security of its signals remained as resilient in the Second World War as in the First. It put its trust in technology and, as a result, came to believe that evidence of information leaks had to be attributed to other factors. Human intelligence — spying — became one way of explaining the apparently inexplicable.

Spy stories and espionage myth-making flourished during and after the First World War. Although both sides used agents, their actual achievements were outstripped by the claims of post-war films and fiction. Mata Hari, the Dutch-born double agent executed by the French in 1917, embodied the allure of the female spy as femme fatale. This suited the intelligence services. Human intelligence was never likely to produce actionable information with the speed and in the quantity provided by signals intelligence, but that dependence could be covered by public self-deception and its appetite for fictional secret agents. The most successful British writer in this genre, John Buchan, was simultaneously running British propaganda. He almost certainly knew what technology was delivering through signals intelligence, but his novels highlighted only human intelligence.

Buchan’s last First World War spy story, Mr Standfast, takes the German offensive of March 1918 as its denouement. It ends with a battle in the air — the second technological revolution during the First World War to transform intelligence, particularly at the tactical level. The movements of the German armies as they swung into France from Belgium in August 1914 were tracked by French aircraft and revealed the moment when the manoeuvre that culminated with the victory of the Marne could be put into effect. When photography was allied to aerial reconnaissance, armies could collect information from deep inside enemy lines and have time to analyse it. They could check their developing assumptions by sending out further flights, and they could correlate their findings with other sources of intelligence, including signals intercepts and night raids. To be sure, bad weather, short winter days, darkness, deception and camouflage could enable the enemy to evade aerial observation, as the Germans did at Verdun in January 1916 before attacking in February, and as they were to do again in the Ardennes in December 1944. But the general effect was to make a surprise attack much harder to achieve and so to reinforce deadlock, most evidently in the First World War but also in the Second. Manoeuvre was now predicated on first achieving aerial supremacy over the battlefield so that the enemy was denied the opportunity to realise what was afoot.

The abundance of information furnished by new technologies in twentieth-century wars presented a fresh challenge for human agency. The issue now was less the collection of intelligence and more its assessment. Individual intercepts conveyed information that was often trivial or routine when analysed in isolation, and only gained significance when read in a wider context. The major contribution which the breaking of Enigma made to the conduct of the war was the cumulative ability to read the German order of battle, to know when units were being moved and whither they were bound. Similarly, aerial observation required successive images of the same location in order to detect change — the construction of new positions, the siting of guns or the effect of bombing. The expertise and experience required for the interpretation of signals or photographs might result in the correct conclusions but not in the right decisions. Commanders could still not know exactly what was in the mind of the enemy; they might be looking at a deliberate deception, as the Germans did in June 1944. They were encouraged by wireless traffic and dummy tanks to believe that a large army under George S. Patton was being mustered in England for a second Allied landing in France at the Pas de Calais. Commanders could be given and accept perfectly accurate intelligence but still only possess a partial insight into enemy intentions, and so misread either the time or the direction of an enemy attack. And they could simply reject what they were told because it did not conform with what they wanted to hear. Bernard Montgomery, commanding the 21st Army Group as it advanced into the Low Countries in September 1944, refused to accept the Ultra intelligence which placed two Panzer divisions in the location of the Rhine bridge at Arnhem, which he planned to take with lightly equipped airborne forces. Montgomery failed.

Despite the contribution made by technology to intelligence collection, it neither prevents wishful thinking nor provides foolproof answers to war’s conduct. In the recent conflicts in Iraq and Afghanistan, the United States and its allies have enjoyed massive advantages in signals intelligence but have suffered from a lack of human intelligence. In planning the aerial attacks on Iraq in 2003, the Americans did not learn from what they failed to recognise in 1990. They also privileged what they acquired electronically over cultural knowledge of Iraqi society or of Saddam Hussein’s government. In part, that was because they did not have the agents on the ground to give them human intelligence; instead, they relied on Iraqi exiles, who had been out of the country for too long. In part, too, it reflected an appetite for secret intelligence, fostered in both world wars, over open-source information. Secret services will prioritise an intercept over a newspaper report.

Today, however, the boundaries between intelligence services and open-source acquisition, and between the product of innovative technologies and human judgement, are becoming blurred. Much of the publicly available intelligence on Russian movements in eastern Ukraine has been acquired by independent, non-state bodies such as Bellingcat, the Netherlands-based investigative journalism group. Using open-source material, from the internet to mobile phones, it completed fine-grain analysis to show who shot down Malaysian Airlines flight MH17 on 17 July 2014. The amount of intelligence that is freely available swamps the capacities of human intelligence assessment. As a result, ‘big data’ is being analysed by artificial intelligence. The results carry significant implications for policymakers as well as for commanders in the field, but they do not remove the need for human judgement. One senior British officer tells a story of his time in Afghanistan, when photographic intelligence revealed evidence of Afghans working the ground overnight, so prompting the conclusion — reinforced at daylight by clear signs of recently moved earth — that they had been burying improvised explosive devices (IEDs). They had not; they were tilling the ground while it was cool and irrigating it when the water could not evaporate too quickly. The questions asked of artificial intelligence, just like those posed of any other form of intelligence, will shape the conclusions that emerge. Intelligence in war is as much an art as a science, as much a product of common sense as of technical brilliance. Wrong questions and false assumptions are just as likely to produce errors in the application of intelligence today as they were in the past.