http://www.wired.com/news/technology/0,70040-0.html By Kevin Poulsen Jan 19, 2006 At half-past noon on Jan. 9, cable TV contractors sinking a half-mile of cable near Interstate 10 in rural Arizona pulled up something unexpected in the bucket of their backhoe: an unmarked fiber-optic cable. "It started pulling the fiber out of the pipe," says Scott Johansson, project manager for JK Communications and Construction. "Obviously, we said, 'Oop, we've hit something.'" As the fiber came spooling out of the desert soil like a fishing line, long-distance service for millions of Sprint PCS and Nextel wireless customers west of the Rockies blinked off. Transcontinental internet traffic routed over Sprint slowed to a crawl, and some corporations that relied on the carrier to link office networks found themselves electronically isolated. In the end, a hole dug out of a dirt road outside a town called Buckeye triggered a three-and-a-half hour outage with national impact. It wasn't even a very deep hole. "We ran into their line right away," says Johansson. Experts say last week's Sprint outage is a reminder that with all the attention paid to computer viruses and the latest Windows security holes, the most vulnerable threads in America's critical infrastructures lie literally beneath our feet. "No one wants something like this to happen," says Sprint spokesman John Taylor. "The fact is we are absolutely focused on restoring service to our customers ... and in this case we did so in record time." A study issued last month by the Common Ground Alliance, or CGA -- an industry group comprised of utilities and construction companies -- calculated that there were more than 675,000 excavation accidents in 2004 in which underground cables or pipelines were damaged. And an October report from the Alliance for Telecommunications Industry Solutions found that cable dig-ups were the single most common cause of telecom outages over a 12-year period ending in 2004, with the number of incidents dropping in recent years but the severity and duration of the outages increasing. In 2004, Department of Homeland Security officials became fearful that terrorists might start using accidental dig-ups as a road map for deliberate attacks, and convinced the FCC to begin locking up previously public data on outages. In a commission filing, DHS argued successfully that revealing the details of "even a single event may present a grave risk to the infrastructure." "We see people talking about the digital Pearl Harbor from the worms and Trojans and viruses," says Howard Schmidt, former White House cybersecurity adviser. "But in all probability, there's more likelihood of what we call the 'backhoe attack' that would have more impact on a region then a Code Red, or anything we've seen so far." Sprint claims it's still investigating who was at fault in Buckeye, but Johansson says that's a settled issue: Before his crew members disturbed so much as a pebble, they submitted their plans to Arizona's "call-before-you-dig" One Call center, then waited for each utility to mark off their buried facilities, if any. Contacted by Wired News, the center confirmed the call. According to Johansson, Sprint responded by giving the contractors the all-clear. "We had a no-conflict ticket from them, indicating that they had no line there," he says. Even that apparent gaffe wouldn't have been enough to cause an outage on its own. The Arizona fiber cut was on a transmission line that loops across the county in a solid ring -- a "self-healing" topology that guarantees a single break won't stop service, because traffic can always circle back in the other direction. But a few days earlier, another section of the same line buried in a railroad culvert near Reno Junction, California, suffered damage in a stormy mudslide. Sprint workers had to cut the waterlogged section of cable to make repairs. So when the contractor's backhoe ripped up the cable in Buckeye, the two cuts together effectively sawed off the entire westernmost section of the ring. But that conspiracy of bad timing and wet weather pales against the impact that deliberate saboteurs or terrorists could make with some rented backhoes and careful target selection. In 2003, then-Ph.D. candidate Sean Gorman famously mapped America's fiber-optic paths for his dissertation at George Mason University, and found it was easy to locate critical choke points from public records and data. Today, Gorman serves as CTO of FortiusOne, a startup that's helping financial companies diversify their electronic infrastructures, and consulting with the DHS. He says the vulnerabilities remain. "We've looked at scenarios where we (could) have multiple fiber cuts that effectively disconnect the West Coast from the East Coast," says Gorman. "It's not very difficult to figure out." Gorman blames this fragility in large part on the recent spate of telecom mergers and acquisitions -- with each one, he says, more and more of the nation's critical communications merge into fewer and fewer fiber-optic cables. Witness the Sprint outage, which affected customers of Nextel, which Sprint finished acquiring last month. Meanwhile, carriers don't want to spend the money to run redundant fiber-optic lines. A 2003 research paper (.pdf) from Sprint notes the company sought alternatives to "physically diverse protection paths" for its backbone network after confronting the "substantial capital investment" of running new cables, as well as challenges posed by geographic obstacles like mountains and bridges. Those geographic limitations have spawned another dangerous trend, says Gorman: Different companies tend to install their cables alongside the same limited number of roads and railways, often unknowingly. "The vast majority of providers are on just two routes" across the country, he says. (Presumably, one of them runs under Buckeye.) If there's widespread agreement on the danger, there's less of a consensus on the solution. Gorman argues that regulators should start taking into account the effect on national security when considering proposals to merge telecoms. "How many fiber paths are they planning on collapsing? How much diversity is the nation losing in the process? It's probably something that should be examined," he says. But former White House cybersecurity adviser Schmidt disagrees. "We built the infrastructure using facilities that were already there, because they were most effective," he says. "You have physical limitations, like bridging the Mississippi River.... Can you imagine they tell you tomorrow, 'We have to build redundancy in the system, so we're going to double your phone bill?'" Instead, Schmidt would like to see the government fund more research into network survivability. "Let's look at the R&D, let's start building this stuff so you can have alternative means of communications -- wireless, satellite. Because you're never going to be able to have 100 percent redundancy." For its part, Sprint insists that its network is diverse enough. "We do put a premium on redundancy," says Taylor. "In this particular case we had events simultaneously happen that are beyond our control." In the end, there's no simple way to prevent sabotage to critical communications lines, should the United States' enemies ever decide on that tact. So far, they haven't. But progress is being made on curtailing accidental damage, in particular by bolstering the system of regional One Call centers dedicated to preventing incidents like the Sprint outage, and the sometimes-fatal accidents that occur when an excavator digs into a buried natural gas or petroleum pipeline. Under state laws, anyone who's breaking ground generally needs to contact the local One Call center first. The center then sends out notices to all the utilities in the area, which are obliged to respond, generally within two days. If anything is buried in the dig zone, the utility dispatches a worker to mark off the location, usually by spray painting a kind of infrastructure hobo's code on the ground: A red line indicates buried cable, yellow is a gas pipe, green a sewer line, etc. Any digging conducted close to the marked facilities has to be conducted by hand, or using special equipment like a vacuum pump. The December CGA report -- the first comprehensive look at digging accidents -- found that nearly half of the 675,000 incidents in 2004 resulted from the excavator failing to contact the local One Call center. The most common facilities damaged as a result were gas pipelines, representing 51.6 percent of the damage. Telecommunications facilities came in second at 27.5 percent. Backhoes, trenchers and shovels tended to hit gas lines, while augers, borers and drills had it in for telecom cables. Most of the incidents only affect local facilities -- it takes bad luck to hit a major communications artery or pipeline. "But when they're hit, the damage is significant," says CGA executive director Bob Kipp. In one of the 2004 incidents, a construction crew in Walnut Creek, California, struck a buried petroleum pipeline, sparking an explosion that killed three people and injured six others. But utilities are hopeful for change. In 2002, Congress passed, and President Bush signed, a law mandating the creation of a national call-before-you-dig three-digit phone number that, like 911, would route automatically to the caller's local center. Last year the FCC decided on 811 as the magic number, and the CGA says it's on the verge of selecting a marketing firm to design a national Smokey the Bear-style campaign to promote the code when it goes live on April 10, 2007. "So instead of having 50 state campaigns with 50 different numbers, we'll get one campaign with one easily recognizable number," says Kipp. "If dad's going to go in the backyard and plant a tree, the kid may say, 'Dad, if you're going to dig, you might blow up something, or we might be without phone service.'" _________________________________ InfoSec News v2.0 - Coming Soon! http://www.infosecnews.org
This archive was generated by hypermail 2.1.3 : Thu Jan 19 2006 - 22:21:52 PST