[ISN] Vulnerable to risk: Other resorts susceptible to a similar outage

From: InfoSec News (isn@private)
Date: Mon Apr 19 2004 - 01:59:06 PDT

  • Next message: InfoSec News: "[ISN] Visa cards violated: BofA is reissuing after hack attack"

    http://www.lasvegassun.com/sunbin/stories/sun/2004/apr/15/516694304.html
    
    April 15, 2004 
    By Liz Benston 
    <benston@private>
    LAS VEGAS SUN
    
    The power outage that shut down the Bellagio and caused the evacuation
    of thousands of employees and customers could happen at any resort on
    the Las Vegas Strip and at any time, Clark County's top building
    inspector said Wednesday.
    
    That's primarily because the power failure occurred on the property
    and with Bellagio equipment rather than on an outer line supplied by
    Nevada Power Co., Clark County Department of Development Services
    Building Division Director Ron Lynn said.
    
    Some newer resorts such as the Bellagio own several power generators
    on site that can create instantaneous backup power to run their
    resorts should Nevada Power lines fail, he said.
    
    If Nevada Power's lines to the Bellagio had failed, the property's
    generators would have kicked on, flooding the property with the same
    power load in about the time it takes to flick a switch, he said.
    
    "(Resort guests) wouldn't have known the difference," he said. "Maybe
    the lights would have flickered and that's it."
    
    But that's not what happened at the Bellagio Sunday, he said. The
    failure occurred internally on resort-owned power lines, meaning the
    power carried from Nevada Power lines had nowhere to go and generators
    couldn't pick up the slack, he said. The generators are set up to go
    on if power doesn't come on to the property.
    
    An internal power line failure could mean bad news for any major
    casino, resort officials say.
    
    "The same kind of initial outage could happen to anyone," Harrah's
    Entertainment Inc. spokesman Gary Thompson said.
    
    What happens next could be more crucial, however, he said.
    
    Harrah's Rio hotel-casino has taken the rare step of building its own
    combustion turbine power plant that is designed to provide "more power
    than needed to get the resort up and operating very quickly" should
    the resort's main power line fail, Thompson said. With the power plant
    and following proper procedures, the resort would be able to power up
    the resort in a few minutes and wouldn't have to rely on emergency
    power and evacuate guests in the process like the Bellagio did, he
    said.
    
    The 4.92 megawatt plant, designed to provide at least 40 percent of
    the resort's power load, will be up and running in the next several
    weeks and is expected to cost the company about $7 million.
    
    Before adding the plant, the resort had a primary power source and an
    emergency power source. The emergency source is a separate cable line
    that is required of all resorts and provides minimal lighting so that
    guests can evacuate the property and powers safety features such as
    fire alarm system, smoke detectors and heat detectors.
    
    The Bellagio also has a main power line and an emergency, or "safety"  
    backup line to maintain minimal power for safety purposes. When some
    cables in the main line shorted, the property shut off power in all
    the main cables to repair the damaged ones and relied on its separate
    emergency line, which is fed by a separate line provided by Nevada
    Power, Lynn said.
    
    The Bellagio's power system meets code requirements and the resort
    followed proper safety procedures by shutting down its primary power
    source when a few of the main power cables shorted, he said.
    
    "We are concerned with the minimum standard for safety," he said.  
    Other features such as the amount of power needed to run the resort
    and backup generators on the site aren't code requirements, he said.
    
    "It's not a safety issue. That's a business decision," he said.
    
    Lynn said he isn't aware of any company in town that has a bulletproof
    system or dual, identical power lines that run separately from Nevada
    Power into a resort.
    
    "I find it highly unlikely that any hotel has a truly redundant
    (power) system," he said.
    
    Like the Rio, the Venetian hotel-casino has also taken some extra
    steps in an attempt to fend off potential power failures.
    
    In addition to the requisite emergency power line, the Venetian owns
    duplicate cabling on its property that runs into the resort, providing
    some backup should one of the lines fail, Venetian's Director of
    Facilities Kim Grange said. The property also has dual transformers as
    well as a computerized system that allows employees to immediately
    swap power loads from one line to another should problems arise, he
    said.
    
    The property could still run into problems if both power lines went
    out or if Nevada Power lines failed, however, he said.
    
    If Nevada Power lines were cut, the company's generators would only be
    able to power about 60 percent of the resort, which wouldn't allow for
    business as usual, Grange said.
    
    After the Bellagio incident, Venetian engineers examined the resort's
    system, which constantly undergoes preventative maintenance, he said.
    
    "I feel that we're very protected here at the Venetian because we've
    got a lot of redundant equipment," he said. "We've taken every measure
    to head off any unforeseen circumstances. But there are no guarantees.  
    Things can happen."
    
    The Bellagio incident was especially unfortunate because the cable
    failure occurred on a primary conduit adjacent to the resort, an
    "achilles heel" for the property, Grange said.
    
    While dual cabling at the Venetian provides some protection, the
    Venetian has so far decided against building its own power plant
    because Nevada Power rates are still reasonable and the cost wouldn't
    be worth the return on investment, he said.
    
    Besides giving the property immediate access to its own power source
    in the event of a power outage, the Rio's cogeneration plant is also
    aimed at cutting the company's power bill, Thompson said.
    
    The plant also helps out other power users by reducing the demand for
    power during peak times when power costs are higher, he said.
    
    Some observers have questioned whether there was some flaw in the
    layout of the power cables at Bellagio or whether there was some human
    error involved in either the initial cable failure or the repair
    process.
    
    MGM MIRAGE spokesman Alan Feldman said the resort has until now
    focused on fixing the problem and will now begin to devote more time
    to investigating what caused it.
    
    Many people have been speculating about what happened without actually
    studying the evidence, which is still coming in, he said.
    
    "Human error is one of the things we will look at," he said. "We have
    focused on the evidence but not enough to make any kind of
    determination about what happened. This was a complicated event. We
    will likely find out that there were many things along the way."
    
    Lynn said the county hasn't ruled out human error as a factor in the
    blackout but said it's still premature to speculate about whether
    something might have exacerbated the problem.
    
    "Our primary investigation is on what initially transpired and created
    the problem in the first place," he said. The county will begin its
    own investigation in earnest today by examining data collected by
    Bellagio computers about power supplies across the property, he said.
    
    The reopening of the resort couldn't have happened any faster than it
    did, Lynn said.
    
    Thousands of feet of burned-out cable needed to be taken out and
    replaced, a process that wasn't fully complete until Wednesday
    morning, he said. The power systems then took hours to test and
    retest, Lynn said.
    
    "It takes time to put the systems back on line," he said. "We ran some
    scenarios to see how they would run in an emergency system and that
    took a good deal of time."
    
    The testing process is similar to that performed at a new resort, he
    said.
    
    County building inspectors performed tests until Wednesday afternoon
    and approved the property to open around 4 p.m. The property opened
    just before 5 p.m. and by about 5:30 p.m. more than 1,000 guests who
    had been put up at neighboring hotels had returned to check in.
    
    
    
    _________________________________________
    ISN mailing list
    Sponsored by: OSVDB.org
    



    This archive was generated by hypermail 2b30 : Mon Apr 19 2004 - 04:21:04 PDT