By Any Means Possible: How Intelligence Agencies Have Gotten Their Data

Everyone knows how governments gather intelligence. There are dashing spies like James Bond (beautiful women optional) and deep cover sleeper agents. Satellites peer into buildings, while brilliant cryptanalysts, in a flash of insight, can crack the strongest “codes.” The messages they crack are, of course, intercepted either by stupendous technical feats¹,² or the derring-do of the aforementioned dashing and/or beautiful spies.

To say that this concept is false is just as misleading as to say that it's true. Intelligence agencies have always and likely will always acquire information by any means necessary. Their metrics are simple and twofold: Will the scheme work? If it works, is the cost—in people, dollars, and exposure risk—acceptable?

Academics think differently. Confronted with an encrypted message, an academic will want to attack the algorithm. Success can range from full key recovery to a distinguishability attack with a complexity of ${2^{230}}$ on a reduced-round version of it. Intelligence agencies just want the plaintext, and getting it because a disgruntled embassy clerk hid it under a bridge is a perfectly acceptable way to succeed. Cryptanalysis might be preferred, but only for its concrete advantages: it doesn't depend on the foibles and presence of an individual asset who might repent or be detected. The aesthetic aspect, that attacking the algorithm is somehow more “elegant,” doesn't enter into the question.

It pays to take a look back through history to see how intelligence has been acquired. The mechanisms used do include spies and beautiful women; they also include satellites, cable taps, strange and wondrous gadgets, and more.

Spies and Assets

Spying, of course, is an ancient tradition. One of the oldest recorded instances is given in the Bible, Joshua 2:1 (this story was probably composed no later than the 8th century BCE³ and perhaps two centuries earlier): “Joshua son of Nun secretly sent two spies from Shittim, saying, ‘Go, reconnoiter the region of Jericho.’ So they set out, and they came to the house of a harlot named Rahab and lodged there.” The chapter goes on to relate how she sheltered them and gave them information about the city's low morale. In return, the spies promised her safety during the forthcoming invasion.

For our purposes, there are three interesting aspects to this tale. First, they consorted with a prostitute, a profession no more respected then than today. Second, they relied on her for information. Third, they made a deal with her: they paid her (with protection from the invading army) for her information and services. These all illustrate practices still followed today: dealing with unsavory insiders, relying on them for information, and paying and protecting them. Joshua's spies could have tried to gather the information themselves, but that didn't work out very well earlier in the Bible (Numbers 13). If nothing else, trying to gather information firsthand is risky; indeed, even Joshua's agents were detected (Joshua 2:2): “The king of Jericho was told, ‘Some men have come here tonight, Israelites, to spy out the country.’” (To be sure, there is some disagreement on the effectiveness of these spies: Yair Zakovitch thinks they're bumblers,⁴ while John Cardwell of the CIA thinks that they did their jobs very well.⁵)

Seducing someone into betraying their country can be done literally, too. “Honey traps”—having someone seduce the target—are used in real life. David Kahn tells the story of an agent code-named CYNTHIA, who worked “not for money but for thrills.”⁶ Her amorous exploits were legion; her conquests included an Italian admiral and a Vichy France press attaché, and these “romances” yielded Italian and French naval codes. The same sorts of things go on today. The Soviets were experts at it; the Mossad found it easier to turn prostitutes into agents than to ask agents to prostitute themselves.⁷ Naturally, it isn't only heterosexual men who are seduction targets: women have been targeted by male agents (this was an East German specialty⁸), and homosexual affairs have often been even better for blackmail.

No one claims that these activities are moral. However, they're perceived as necessary. Most information from human sources—“HUMINT”—is supplied by insiders. This is of necessity: the insiders have the information, and outsiders are too conspicuous. Cases of spies actually penetrating an enemy organization are very rare, although not unknown (one example is the amazing story of Eli Cohen, an Israeli agent who was in line to become Syria's deputy defense minister⁹). Far more often, the role of an agent today, just as in Joshua's time, is to persuade someone else to give up the necessary information. The agent's role is to persuade insiders, pay them, and, if necessary, exfiltrate them. Yes, there's clever gadgetry, but this is generally for photography, communicating with sources, and so on.¹⁰

HUMINT is done this way because it works. That said, there are limits to what spies can accomplish. They can only report what they know or what's contained in documents to which they have access. A well-placed political agent is unlikely to have technical details on weaponry; conversely, a well-placed technician will have little idea what political decisions are being made. Beyond that, spying is risky, both personally and politically. Agents have a finite lifetime before they fall under suspicion or burn out; when these things happen, exfiltrating them becomes crucial. This is partly because it's good practice—few prospective spies will want to work for a country that abandons its assets—but also because captured spies carry a cost, in embarrassment and in what they can be forced to reveal about other operations.

Intercepting Communications

To cope with some of the limitations of human agents, intelligence agencies have long resorted to other means. One has been communications interception, thereby gaining an insight into the other side's actual operational plans. Kahn wrote, of an incident during World War I, that⁶

Militaries weren't slow to realize the benefits of communications interception. Communications interceptions are about as old as the use of communications for military purposes; not surprisingly, so are defenses. The ancient Greeks had their scytales, and the Romans had the Caesar cipher.⁶ During the Renaissance, diplomats communicated via sealed, encrypted letters—and rulers, from dukes up through the Pope, set up “black chambers” to open, copy, decrypt, and reseal these messages. When, during the US Civil War, the telegraph became important, interception of telegrams followed quickly. Jeb Stuart, a Confederate general, “actually had his own personal wiretapper travel along with him in the field.”¹¹

As technology improved, so did both attacks and defenses. Intelligence threats to telegraph cables became a major driver in British communications strategy. The ability to send messages via an “all-red route”—that is, solely via stations located in British possessions, which were colored red in the maps of the time—became a major driver in selecting paths for new telegraph links.¹² Conversely, they weren't slow to appreciate the intelligence capabilities they had acquired by virtue of being the hub of the world's telegraph network; indeed, the Official Secrets Act of 1920 contained a provision that effectively required copies of all international telegrams transiting the United Kingdom to be turned over to Naval Intelligence.

Radio made interception easier; naturally, the world's spy agencies built elaborate interception facilities. Defenders countered with encryption—they, too, resorted to technology. Hand encryption systems were too slow for volumes of traffic and were insecure to boot, so mechanized systems were developed: Vernam's one-time tape Teletype, Scherbius's Enigma, and more. Cryptanalysts countered with automation of their own, both off-the-shelf punch card machines and custom devices.⁶,¹³,¹⁴

The culmination of this trend was computerization of cryptanalysis. The first such machine, the Colossus, was also the first programmable electronic computer, built at Bletchley Park to crack the German Lorenz (“Tunny”) cipher.¹⁵ The trend has continued. One of the customers for IBM's Project Stretch, an effort to produce a computer 100 times faster than the IBM 704,¹⁶ was the NSA. The resulting computer, the IBM 7030, even had a special cryptanalytic add-on called Harvest, described in the open literature as a “Nonarithmetical System Extension.” The NSA is still building massive computing complexes.¹⁷

More elaborate interception techniques have been used as well. Spy subs have tapped undersea cables.¹ Hidden microphones have picked up the sound of rotor wheels being set.¹⁸ Specialized ships, satellites, and planes have all been used to collect radio signals.

Communications interception can work well, and if done by technical means, is often undetectable. If the activities are detected or disclosed, there can be considerable public outrage—witness the uproars about Echelon and the Snowden revelations—but intelligence agencies often shrug off such problems. There are, however, two obstacles: one, the increasing volume of communications means that there's a vast amount of data to collect and sift through in search of the really interesting material, which in turn translates to vastly increased cost, and two, the growth in strong encryption means that the actual yield is less. To be sure, the growth in machine-readable data—it's easier to process structured text than voice—and the amazing haul in metadata have at least partially compensated, but there's constant concern about “going dark,” a loss of ability to read the other side's communications. Some claim that the growth in metadata processing doesn't compensate, but Michael Hayden, former NSA director, has remarked, “We kill people based on metadata.”¹⁹

Overhead Surveillance

Spying moved overhead as soon as it was technically possible. In the US Civil War, balloons were employed from the beginning for reconnaissance and used in Europe even earlier²⁰: “The importance of gaining such a height for observation can be appreciated by all readers of military annals.”²¹

Naturally, progress didn't stop with balloons; reconnaissance aircraft played major roles during both world wars. In fact, aerial reconnaissance was so pervasive during World War II that the British used “accidental” sightings of German ships to disguise the real way they knew their location: cryptanalysis. Aerial reconnaissance was so common that it need not be concealed.

Two of the most famous planes in history, the U-2 and the SR-71, were spy planes. Built at Lockheed's legendary Skunk Works,²² one of their primary missions was observing the Soviet Union's nuclear capabilities.²³ The U-2 became vulnerable to improved Soviet air defenses, of course, but a replacement was ready in time: the first spy satellites.

These early satellites were a technological tour de force: for lack of suitably compact television cameras, they periodically ejected film canisters that were caught in midair by specially equipped airplanes. This sounds like an amazing spur-of-the-moment response to an intelligence crisis, but it wasn't. The US had been planning for orbital spying since at least 1950: a satellite would be a “novel and unconventional instrument of reconnaissance.”²⁴ In fact, one motivation for launching scientific satellites during the International Geophysical Year was to provide legal precedent for satellites overflying other countries; IGY was under UN auspices.²⁴

Modern spy satellites are much more sophisticated, of course. Indeed, the mirror technology for the Hubble space telescope was the same used for spy satellites.²⁵ In addition, many specialized types exist now, such as ELINT (electronic intelligence), radar ocean reconnaissance, missile warning, communications interception, and more.²⁶

Overhead reconnaissance works, but there are limits. Cameras can't see inside buildings. Clever adversaries can time their activities to evade satellites—some claim that India did exactly that to hide preparations for its 1998 nuclear tests.²³,²⁷,²⁸ Airplanes have less predictable coverage patterns, but of course they can be shot down. It will be interesting to see how drone-based platforms fare.

Enter the Computer

Legend has it that when Willy Sutton, the bank robber, was asked why he robbed banks, he replied, “Because that's where the money is.” Intelligence agencies follow a similar philosophy: they'll go where the data is. Today, much of the world's information is created on, transmitted from, and received by computers. This alone would make computers an interesting target. In addition, the explosive growth of strong cryptography has rendered traditional communications intercepts much less useful. The solution—capturing the data before encryption or after decryption—is obvious, if it can be done.

It can. Furthermore, the techniques necessary, such as hacking software, are useful for other forms of collection. For example, modern phone switches are nothing but computers with odd peripherals attached, and these computers can be hacked, too.

At this point, we know little of how computer espionage is done, by whom, or its scope. There have been a few published reports, mostly focusing on economic espionage.²⁹ Occasional failed operations, such as one that penetrated a mobile phone switch in Greece,³⁰ give some hint of possibilities. In other cases, government-grade spyware has been discovered—Flame, for example, used a previously unknown cryptanalytic attack on the MD5 hash function.³¹ We know neither the details nor the scope, but we've seen enough to know what's going on: as always, spying has followed technology, and now it's moved into cyberspace. That might be upsetting, but it can't be considered a surprise.

So where are we today? According to one published report, “the agency's top management corps for the past five or six years has consisted entirely of software engineers.”³² That's true of the new director, too: Michael Rogers, though originally trained as a cryptologist, “specialized in computer network attacks.”³³ The sought-after information—the intelligence “money”—has moved to computers, and the spies have followed.