Venona Project
In 1942 the United States Army's Signals Intelligence Service recruited Meredith Gardner to work on breaking German codes. During this period he also taught himself Japanese so that he could also work on their codes as well. He spent the rest of the war studying messages between Germany and Japan. "He worked initially on German ciphers and then on Japanese super-enciphered codes, in which messages were first encoded in five-figure groups taken from a code book and then enciphered by adding a series of randomly produced figures, known as an additive, which was taken from a second book." (1)
After the war Gardner was assigned to help decode a backlog of communications between Moscow and its foreign missions. By 1945, over 200,000 messages had been transcribed and now a team of cryptanalysts attempted to decrypt them. The project, named Venona (a word which appropriately, has no meaning), was based at Arlington Hall, Virginia. (2) Soviet messages were produced in exactly the same way as Japanese super-enciphered codes. However, "where the Japanese gave the codebreakers a way in by repeatedly using the same sequences of additive, the Russian system did not. As its name suggests, the additive appeared on separate sheets of a pad. Once a stream of additive had been used, that sheet was torn off and destroyed, making the message impossible to break." (3) According to Peter Wright, the author of Spymaster (1987): "Meredith Gardner... began work on the charred remains of a Russian codebook found on a battlefield in Finland. Although it was incomplete, the codebook did have the groups for some of the most common instructions in radio messages - those for 'Spell' and 'Endspell.' These are common because any codebook has only a finite vocabulary, and where an addresser lacks the relevant group in the codebook - always the case, for instance, with names - he has to spell the word out letter by letter, prefixing with the word 'Spell,' and ending with the word 'Endspell' to alert his addressee. Using these common groups Gardner checked back on previous Russian radio traffic, and realized that there were duplications across some channels, indicating that the same one-time pads had been used. Slowly he 'matched' the traffic which had been enciphered using the same pads, and began to try to break it." (4) As David C. Martin pointed out it was slow work: "When the cryptanalysts discovered that the same series of additives had been used more than once, they had all the leverage they needed to break the Soviet cipher system. Having used guesswork to deduce the additives for a Soviet message intercepted in one part of the world, they could test those same additives against the massive backlog of messages intercepted in other parts of the world. Sooner or later the same additives would appear and another message could be deciphered. It was an excruciatingly tedious task with less than perfect results. Since only a portion of the code book had been salvaged, many of the 999 five-digit groups used by the Soviets were missing. Knowing the additive might yield the proper five-digit group, but if that group could not be found in the code book, the word remained indecipherable. Whole passages were blanks, and the meaning of other phrases could be only vaguely grasped." (5) William Weisband immediately notified the NKVD that the Venona Project was on the verge of success. To make sure that the FBI was unaware that they knew that the code was about to be broken, they continued to use it. The "operatives" were instructed "every week to compose summary reports or information on the basis of press and personal connections to be transferred to the Center by telegraph." As Allen Weinstein, the author of The Hunted Wood: Soviet Espionage in America (1999) has pointed out the "Soviet intelligence's once-flourishing American networks, in short, had been transformed almost overnight into a virtual clipping service." (6) In February 1948 a Soviet official wrote an internal memorandum about the work of Weisband. "For one year, a large amount of very valuable documentary material concerning the work of Americans on deciphering Soviet ciphers, intercepting and analyzing open radio-correspondence of Soviet institutions (the Venona project), was received from (Weisband). From these materials, we came to know that, as a result of this work, American intelligence managed to acquire important data concerning the stationing of the USSR's armed forces, the productive capacity of various branches of industry, and work in the field of atomic energy in the USSR... On the basis of Weisband's materials, our state security organs carried out a number of defensive measures, resulting in the reduced efficiency of the American deciphering service. This has led to the considerable current reduction in the amount of deciphering and analysis by the Americans." (7) Weisband passed documents to Yuri Bruslov, his Soviet contact in the United States. In August 1948, Weisband became worried that he would be exposed as a spy and switched to dead-letter drops. Weisband also asked Bruslov to request asylum for him in the Soviet Union. He was too important a source and therefore they began paying him regular sums of $600 for his efforts. In 1949 Bruslov was replaced by Nikolai Statskevich. In December 1949, the FBI identified Statskevich as a Soviet agent. They also discovered that he was having regular meetings with Weisband. It was not until 1949 that Meredith Gardner made his big breakthrough. He was able to decipher enough of a Soviet message to identify it as the text of a 1945 telegram from Winston Churchill to Harry S. Truman. Checking the message against a complete copy of the telegram provided by the British Embassy, the cryptanalysts confirmed beyond doubt that during the war the Soviets had a spy who had access to secret communication between the president of the United States and the prime minister of Britain. The Armed Forces Security Agency requested copies of all transmissions handled by the British Embassy and began matching them against the encoded messages in the New York-to-Moscow channel, working backward through the code book and arriving at the additive. Gradually they were able to transcribe these messages. It now became clear that there had been a massive hemorrhaging of secrets from both the British Embassy in Washington and the atomic bomb project at Los Alamos, New Mexico. One message revealed that one of the scientists working on the Manhattan Project who was spying for the Soviets had a sister at an American university. This scientist had not been born in the United States. When the FBI carried out a full investigation into the scientists working on the project they found that Klaus Fuchs had a sister, Kristel, who had briefly attended Swarthmore College during the war. After the war Fuchs returned to England where he became head of the physics department of the British Nuclear Research Centre at Harwell. The FBI told MI5 about their suspicions and Fuchs was brought in for questioning. Fuchs denied any involvement in espionage and the intelligence services did not have enough evidence to have him arrested and charged with spying. However, after repeated interviews with Jim Skardon he eventually confessed on 23rd January 1950 to passing information to the Soviet Union. A few days later J. Edgar Hoover informed President Harry S. Truman that "we have just gotten word from England that we have gotten a full confession from one of the top scientists, who worked over here, that he gave the complete know-how of the atom bomb to the Russians." (8) Fuchs was found guilty on 1st March 1950 of four counts of breaking the Official Secrets Act by "communicating information to a potential enemy". After a trial lasting less than 90 minutes, Lord Rayner Goddard sentenced him to fourteen years' imprisonment, the maximum for espionage, because the Soviet Union was classed as an ally at the time. (9) Hoover reported that "Fuchs said he would estimate that the information furnished by him speeded up by several years the production of an atom bomb by Russia." (10) Following further information provided by Meredith Gardner and Klaus Fuchs, the FBI arrested Harry Gold and David Greenglass in July, 1950, Greenglass was arrested by the FBI and accused of spying for the Soviet Union. Under questioning, he admitted acting as a spy and named Julius Rosenberg as one of his contacts. He denied that his sister, Ethel Rosenberg, had been involved but confessed that his wife, Ruth Greenglass, had been used as a courier. (11) Gardner was also provide information to the FBI to trace another Soviet spy in the Manhattan Project. He was an American scientist, Theodore Hall, who was now teaching at the University of Chicago. He was interviewed by Alan H. Belmont in March 1951. "Although FBI agents put pressure on him to confess he gave nothing away, and they could find no other evidence against him beyond the Venona documents. Since Venona was still yielding fresh secrets at that time and promised to be a counter-intelligence gold mine for many years to come, the US security authorities believed they could not afford to let Moscow know they were cracking the code... So it was that, in the expectation that they might catch other fish in future, the FBI let Theodore Hall swim free." (12) In fact, the Soviets already knew about the breakthrough because of information provided by William Weisband, who worked with Gardner at the Armed Forces Security Agency. Peter Wright met Meredith Gardner in London after the arrests of the atom spies: "He was a quiet, scholarly man, entirely unaware of the awe in which he was held by other cryptanalysts. He used to tell me how he worked on the matches in his office, and of how a young pipe-smoking Englishman named Philby used to regularly visit him and peer over his shoulder and admire the progress he was making. Gardner was rather a sad figure by the late 1960s. He felt very keenly that the cryptanalytical break he had made possible was a thing of mathematical beauty, and he was depressed at the use to which it had been put." Wright revealed that he was upset that his research had resulted in McCarthyism and the executions of Julius Rosenberg and Ethel Rosenberg. Wright quotes Gardner as saying: "I never wanted it to get anyone into trouble." Wright added that Gardner "was appalled at the fact that his discovery had led, almost inevitably, to the electric chair, and felt (as I did) that the Rosenbergs, while guilty, ought to have been given clemency. In Gardner's mind, VENONA was almost an art form, and he did not want it sullied by crude McCarthyism." (13) Meredith Gardner and his team were able to work out that more than 200 Americans had become Soviet agents during the Second World War. They had spies in the State Department and most leading government agencies, the Manhattan Project and the Office of Strategic Services (OSS). However, they were at first unable to discover the identity of a spy with the codename Homer. His name was found on a number of messages from the KGB station at the Soviet consulate-general in New York to Moscow Centre. The cryptanalysts discovered that the spy had been in Washington since 1944. The FBI concluded that it could be one of 6,000 people. At first they concentrated their efforts on non-diplomatic employees of the embassy. In April 1951, the Venona decoders found the vital clue in one of the messages. Homer had had regular contacts with his Soviet control in New York, using his pregnant wife as an excuse. This information enabled them to identify the spy as Donald Maclean, the first secretary at the Washington embassy. Unknown to the FBI, the man MI6 had sent them to help with identifying British spies named in the Venona project, Kim Philby, was also a Soviet agent. Meredith Gardner later recalled that Philby was a regular visitor to Arlington Hall. He observed the strange intensity with which Philby had observed the decryption teams at work: "Philby was looking on with no doubt rapt attention but he never said a word, never a word." (14) As Ben Macintyre, the author of A Spy Among Friends (2014) pointed out: "Philby immediately relayed the bad news to Valeri Makayev (Philby's Russian contact in America), and demanded that Maclean be extracted from the UK before he was interrogated and compromised the entire British spy network - and more importantly Philby himself." (15) As a result of Philby's warning, Maclean and fellow spy, Guy Burgess, were able to escape to Moscow. In 1995-96 over 2,990 fully or partially decrypted Soviet intelligence cables from the Venona archives were declassified and released by the Central Intelligence Agency and the National Security Agency.Meredith Gardner and the Venona Project
William Weisband - Soviet Spy
Breaking the Code
Klaus Fuchs
Spies in Manhattan Project
Cambridge Spy Ring
Release of the Venona Archives
Primary Sources
(1) David C. Martin, Wilderness of Mirrors (1980)
Amid the confusion of war, Moscow had sent out duplicate sets of additives to various Soviet installations around the world. When the cryptanalysts discovered that the same series of additives had been used more than once, they had all the leverage they needed to break the Soviet cipher system. Having used guesswork to deduce the additives for a Soviet message intercepted in one part of the world, they could test those same additives against the massive backlog of messages intercepted in other parts of the world. Sooner or later the same additives would appear and another message could be deciphered. It was an excruciatingly tedious task with less than perfect results. Since only a portion of the code book had been salvaged, many of the 999 five-digit groups used by the Soviets were missing. Knowing the additive might yield the proper five-digit group, but if that group could not be found in the code book, the word remained indecipherable. Whole passages were blanks, and the meaning of other phrases could be only vaguely grasped.
Because of the laborious nature of the task, years would elapse between the actual transmission of a Soviet message and its decoding by the Armed Forces Security Agency. The first big break did not come until 1949, when the cryptanalysts found a duplicate additive in the New York-to-Moscow channel and were able to decipher enough of a Soviet message to identify it as the text of a 1945 telegram from Churchill to Truman. Checking the message against a complete copy of the telegram provided by the British Embassy, the cryptanalysts confirmed beyond doubt that a Soviet spy had somehow been able to obtain the verbatim text - cable number and all - of a private communication between two heads of state.
(2) Peter Wright, Spymaster (1987)
Shortly after the end of the war a brilliant American cryptanalyst named Meredith Gardner, from the U.S. Armed Forces Security Agency (the forerunner of the NSA), began work on the charred remains of a Russian codebook found on a battlefield in Finland. Although it was incomplete, the codebook did have the groups for some of the most common instructions in radio messages - those for "Spell" and "Endspell." These are common because any codebook has only a finite vocabulary, and where an addresser lacks the relevant group in the codebook-always the case, for instance, with names-he has to spell the word out letter by letter, prefixing with the word "Spell," and ending with the word "Endspell" to alert his addressee.
Using these common groups Gardner checked back on previous Russian radio traffic, and realized that there were duplications across some channels, indicating that the same one-time pads had been used. Slowly he "matched" the traffic which had been enciphered using the same pads, and began to try to break it. At first no one would believe him when he claimed to have broken into the Russian ciphers, and he was taken seriously only when he got a major breakthrough in the Washington-to-Moscow Ambassadorial channel. He decrypted the English phrase "Defense does not win wars!" which was a "SpelllEndspell" sequence. Gardner recognized it as a book on defense strategy published in the USA just before the date the message was sent. At this point, the Armed Forces Security Agency shared the secret with the British, who at that time were the world leaders in cryptanalysis, and together they began a joint effort to break the traffic, which lasted forty years.
Operation BRIDE (as it was first known) but later DRUG and VENONA, as it was known in Britain, made painfully slow progress. Finding matches among the mass of traffic available took time enough. But even then there was no certainty the messages on each side of the match could be broken. The codebook was incomplete, so the codebreakers used "collateral" intelligence. If, for instance, they found a match between the Washington-to-Moscow KGB channel and the New York-to-Moscow trade channel, it was possible to attack the trade channel by using "collateral," information gathered from shipping manifests, cargo records, departure and arrival times, tide tables, and so forth, for the date of the message. This information enabled the codebreakers to make estimates of what might be in the trade traffic. Once breaks were made in one side of a match, it provided more groups for the codebook, and helped make inroads on the other side.
The British and Americans developed a key device for expanding the VENONA breaks. It was called a "window index." Every time a word or phrase was broken out, it was indexed to everywhere else it appeared in the matched traffic. The British began to index these decrypts in a more advanced way. They placed two unsolved groups on each side of the decrypted word or phrase and after a period of time these window indexes led to repetitions, where different words which had been broken out were followed by the same unsolved group. The repetition often gave enough collateral to begin a successful attack on the group, thus widening the window indexes. Another technique was "dragging." Where a"SpelUEndspell" sequence or name came up, and the cryptanalysts did not know what the missing letters of the spelled sequence were, the groups were dragged, using a computer, across the rest of the channels, and out would come a list of all the repeats. Then the cryptanalysts would set to work on the reverse side of the repeat matches, and hope to attack the "Spell/Endspell" sequence that way.
It was an imperfect art, often moving forward only a word or two a month, and then suddenly spilling forward, like the time the Americans found the complete text of a recorded speech in the Washington Ambassadorial channel. Often terrible new difficulties were encountered: one-time pads were used in unorthodox ways, up and down, or folded, which made the process of finding matches infinitely more problematic. There were difficulties, too, with the codebooks. Sometimes they changed, and whereas the Ambassadorial, GRU, and trade channels used a straightforward alphabetically listed codebook, rather like a dictionary, so that the codebreakers could guess from the group where in the codebook it appeared, the KGB used a special multivolume random codebook which made decrypting matched KGB channels a mindbending task. The effort involved in VENONA was enormous. For years both GCHQ and NSA and MIS employed teams of researchers scouring the world searching for "collateral"; but despite the effort less than 1 percent of the 200,000 messages we held were ever broken into, and many of those were broken only to the extent of a few words....
Years later, I arranged for Meredith Gardner to visit Britain to help us on the British VENONA. He was a quiet, scholarly man, entirely unaware of the awe in which he was held by other cryptanalysts. He used to tell me how he worked on the matches in his office, and of how a young pipe-smoking Englishman named Philby used to regularly visit him and peer over his shoulder and admire the progress he was making. Gardner was rather a sad figure by the late 1960s. He felt very keenly that the cryptanalytical break he had made possible was a thing of mathematical beauty, and he was depressed at the use to which it had been put.
"I never wanted it to get anyone into trouble," he used to say. He was appalled at the fact that his discovery had led, almost inevitably, to the electric chair, and felt (as I did) that the Rosenbergs, while guilty, ought to have been given clemency. In Gardner's mind, VENONA was almost an art form, and he did not want it sullied by crude McCarthyism. But the codebreak had a fundamental effect on Cold War attitudes among those few indoctrinated officers inside the British and American intelligence services. It became the wellspring for the new emphasis on counterespionage investigation which increasingly permeated Western intelligence in the decades after the first break was made. More directly, it showed the worldwide scale of the Soviet espionage attack, at a time when the Western political leadership was apparently pursuing a policy of alliance and extending the hand of friendship. In the British traffic, for instance, most of the KGB channel during that September week was taken up with messages from Moscow detailing arrangements for the return of Allied prisoners to the Soviet authorities, groups like the Cossacks and others who had fought against the Soviet Union. Many of the messages were just long lists of names and instructions that they should be apprehended as soon as possible. By the time I read the messages they were all long since dead, but at the time many intelligence officers must have been struck by the sense that peace had not come in 1945; a German concentration camp had merely been exchanged for a Soviet Gulag.