WASHINGTON – The scope of a hack carried out by a leading Russian intelligence agency became clearer on Monday when the Trump administration recognized that another federal agency, the Department of Homeland Security, had been compromised. Investigators struggled to determine which parts of the military, intelligence agencies and nuclear laboratories were also vulnerable to the sophisticated attack.
US officials only discovered the attack in the past few weeks, and only then when a private cybersecurity firm, FireEye, alerted American intelligence that the hackers had eluded layers of defense.
It was evident that the Treasury and Commerce Departments, the first to be reported violated, were only part of a far larger operation, the sophistication of which baffled even experts who tracked a quarter of a century of Russian hacks on the Pentagon and American civil authorities.
According to SolarWinds, the company whose software was compromised, around 18,000 private and government users downloaded a Russian software update – a type of Trojan horse – that its hackers used to break into victims' systems.
Those who use SolarWinds software include the Centers for Disease Control and Prevention, the State Department, the Department of Justice, portions of the Pentagon, and a number of utilities. While the presence of the software is not in and of itself proof that every network has been compromised and information stolen, investigators on Monday attempted to understand the extent of the damage that a significant loss of American data could mean to a foreign attacker.
The National Security Agency – the leading US intelligence agency that both infiltrates foreign networks and protects national security agencies from attacks – appeared to be unaware of the SolarWinds network surveillance software breach until it was notified by FireEye last week. The N.S.A. himself uses SolarWinds software.
One of the most embarrassing violations was the Department of Homeland Security, whose cybersecurity and infrastructure security agency oversaw the successful defense of the American electoral system last month.
A government official, who asked for anonymity to discuss the investigation, made it clear that the Department of Homeland Security, which is charged with securing civil government agencies and the private sector, was itself a victim of the complex attack. The department, which frequently urges companies to clean up their customers when their systems are the victim of successful attacks, issued a disguised official statement that read only: "The Department of Homeland Security has received reports of violations. We are currently investigating the matter. "
According to a contractor who spoke on condition of anonymity, parts of the Pentagon were also affected by the attack, but officials were similarly shy.
"The D.O.D. is aware of the reports and is currently assessing the impact," said Russell Goemaere, a Pentagon spokesman, adding that for security reasons, the Pentagon "would not specify any systems that may be affected."
Investigators specifically focused on why the Russians targeted the Ministry of Commerce's national telecommunications and information administration, which is helping set guidelines for Internet issues, including setting standards and blocking imports and exports of technology considered national Security risk is considered. However, analysts noted that the agency deals with some of the most cutting-edge commercial technology and dictates what will be sold and denied to rival countries.
Virtually all Fortune 500 companies, including the New York Times, use SolarWinds products to monitor their networks. This also applies to the Los Alamos National Laboratory, where nuclear weapons are developed, and large defense companies like Boeing, which on Monday refused to discuss the attack.
The early assessments of the interventions – probably the work of the Russian S.V.R., a successor to the K.G.B. – suggest that the hackers were very selective about which victims they used for further access and data theft.
The hackers embedded their malicious code in Orion software from SolarWinds, based in Austin, Texas. The company said 33,000 of its 300,000 customers are using Orion and only half of those customers have downloaded the malicious Russian update. FireEye said that despite their widespread access, Russian hackers only exploited what was viewed as its most valuable target.
"We believe dozens have actually been compromised," said Charles Carmakal, FireEye senior vice president. "But they were all the most valuable targets."
The picture that emerged from interviews with corporate and government officials on Monday as they were trying to assess the extent of the damage was a complex, sophisticated attack on the software used in the systems that run corporate and government agency activities monitor.
After a quarter of a century of hacks against the defense industry establishment – many of them with brutal efforts to crack passwords or send spearphishing messages to trick ignorant email recipients into giving up their credentials – the Russian operation was a different breed. The attack was "the day you are preparing for," said Sarah Bloom Raskin, assistant treasury secretary during the Obama administration.
Investigators believe that in addition to the compromised Orion software update, Russian hackers used multiple entry points and that this may just be the beginning of what they find.
SolarWinds' Orion software updates are not automatic, officials noted, and are frequently checked to ensure that existing computer systems are not being destabilized.
SolarWinds customers were still trying to assess the impact of the Russian attack on Monday.
A Justice Department spokesman who uses SolarWinds software declined to comment.
Ari Isaacman Bevacqua, a spokeswoman for the New York Times, said, "Our security team is aware of recent developments and will take appropriate action when warranted."
Military and intelligence officials declined to say how widespread Orion was in their organizations or whether those systems were updated with the infected code that gave the hackers widespread access.
But if the government hadn't been aware of the security flaw in SolarWinds and had kept it a secret – which is sometimes the case to develop obnoxious cyber weapons – there would have been little reason not to install the most current versions of the software. There is no evidence that government officials withheld knowledge of the bug in the SolarWinds software.
The agency for cybersecurity and infrastructure security issued a rare emergency policy on Sunday warning federal agencies to shut down SolarWinds software. But this only prevents new interventions; It doesn't eliminate Russian hackers who, as FireEye said, planted their own "back doors", impersonating legitimate email users and fooling the electronic systems designed to ensure users' identity with the correct passwords and additional authentication.
"Such a supply chain attack is an incredibly expensive operation. The more you use it, the more likely you are to be caught or burned," said John Hultquist, director of threats at FireEye. "They had the opportunity to hit a large number of targets, but they also knew that if they went too far, they would lose their incredible access."
The chief executive officers of America's largest utility companies called urgently on Monday to discuss the potential threat to the power grid posed by the SolarWinds Compromise.
For the N.S.A. and its director, General Paul M. Nakasone, who also heads the US cyber command, the attack is one of the biggest crises of his tenure. Recruited almost three years ago as one of the most skilled and trusted cyber warriors in the country, he pledged to Congress to make sure those who attack the US pay a price.
In his audit hearing, he famously stated that the nation's cyber adversaries “are not afraid of us” and acted quickly to increase the cost to them. He delved deeply into foreign computer networks, attacked the Russian Internet research agency and sent warning shots over the bows of known Russian hackers.
General Nakasone focused heavily on protecting the country's electoral infrastructure, with considerable success in the 2020 vote. But it now appears that both civil and national security agencies were the target of this carefully crafted hack, and he will have to answer why private industry – rather than the multi-billion dollar corporations he got out of a war room in Fort Meade, Md. – was the first to sound the alarm.
Analysts said it was hard to know which was worse: that the federal government was blinded again by Russian intelligence services, or that White House officials said nothing when it was obvious what was happening.
But this much is clear: while President Trump complained about the hack that wasn't – the alleged manipulation of votes in an election that he clearly and fairly lost – he was silent about the fact that Russians hacked the building next to him : the United States Treasury Department.
In the short term, government agencies are now trying to get to the bottom of a problem with limited visibility. By shutting down SolarWinds – a step they had to take to halt future interference – many agencies are losing visibility of their own networks.
"You're flying blind," said Ben Johnson, a former N.S.A. Hacker who is now the chief technology officer of Obsidian, a security company.
David E. Sanger reported from Washington and Nicole Perlroth from Palo Alto, California. Zolan Kanno-Youngs and Alan Rappeport reported from Washington.