[ISN] Disruptions Put Bank of New York to the Test

From: InfoSec News (isnat_private)
Date: Wed Oct 10 2001 - 00:41:50 PDT

  • Next message: InfoSec News: "[ISN] Cyberwar Foundering on Feuds?"

    October 6, 2001
    From the roof of the Bank of New York's Art Deco headquarters, just
    down the block from the New York Stock Exchange, one can easily see
    why the bank has been at the uncomfortable center of Wall Street's
    Before Sept. 11, nearly half of the bank's 19,000 employees, including
    those responsible for its main computer center, worked within view of
    the headquarters.
    Now the vista is dominated by the still- smoldering rubble of the
    World Trade Center; just beyond are two dust-covered buildings that
    had housed 5,300 employees and the bank's main computer center.
    The Bank of New York is woven into the fabric of Wall Street by more
    than just its location. It has played down the lending and
    deposit-taking functions of most banks and devoted itself to the
    plumbing of finance, accounting for and moving money and securities
    for traders, corporations and investors. Half of the trading in
    government bonds, for example, moves through its settlement system. As
    such, many Wall Street firms depend on the bank to ensure that their
    trading runs smoothly.
    Despite a reluctance to criticize publicly as the financial world is
    striving to show unity, some traders at those firms say the Bank of
    New York has had a harder time recovering from the disaster than
    others, with system failures and delayed processing leaving some firms
    in the dark about their financial positions or waiting extra days for
    money or securities to arrive.
    The Bank of New York, some say, was just too concentrated in
    Manhattan. The bank's penchant for lean operations, others surmise,
    may have kept it from having quite the systems or the manpower to
    recover quickly.
    Now, stock analysts wonder whether the bank may lose business from
    clients who want to be less vulnerable in the future.
    Thomas A. Renyi, the Bank of New York's chairman, acknowledges that
    there have been flaws in the bank's service. But he says the bank has
    bounced back from a nearly unthinkable disruption largely intact. "We
    are proud of what we have done," Mr. Renyi said, from his customary
    spot at the end of a vast wooden conference table next to his office
    in the headquarters at 1 Wall Street.
    "Clearly there were ups and downs," he said. "But our longstanding
    disaster recovery plans worked, and they worked in the extreme." Until
    Sept. 11, no company had ever had to cope with a sudden dislocation of
    so many people and computer systems, the bank's executives point out.
    Indeed, it is a testament to contingency planning across Wall Street
    that so many companies were back in business only days after being
    But the Bank of New York, because its systems are so central to the
    business of all the other firms, was under extraordinary pressure to
    resume running at full speed. The places where it lagged became
    especially noticeable.
    Mr. Renyi also gingerly suggests that the Bank of New York may have
    become a lightning rod for a wide range of complaints. Broken
    communication links were the bulk of the problems, he said, and it is
    not clear whether the fault, if any, lies with the sender, the
    receiver or the phone company in the middle. But those problems showed
    up when the bank fell behind on processing trades. "We are a highly
    visible provider and have a highly vocal customer base," Mr. Renyi
    said.  Michael Mayo, an analyst at Prudential Securities, said that
    many of the bank's customers were understanding of its problems, for
    "The Bank of New York had some bad luck and was given a grace period
    by their customers, but eventually the grace period will expire," Mr.
    Mayo said. "This is a big test for the Bank of New York's management.
    The spotlight is on the firm to get their problems fixed."  Bank
    executives argue that some of the criticism has taken on some aspects
    of urban legend, especially the notion that the bank was in disarray
    because the main backup for its computer center in Lower Manhattan was
    at another location in Lower Manhattan.
    The bank says that all of its several computer centers in Manhattan
    were always set to revert to centers outside the city in case of
    emergency, and they did on Sept. 11. The bank also insists that its
    spending for disaster recovery has increased substantially in recent
    In a few cases, though, the bank simply threw out its plan. The
    department that processes paper checks at its computer center, at 101
    Barclay Street, was supposed to move to Cherry Hill, N.J. But with
    communication so disrupted, the bank did not want that operation so
    far away.
    So the checks were sent to another check processing center in Lodi,
    N.J. But that center did not have machines needed for the bank's
    lockbox business, in which it opens envelopes that contain bill
    payments, both depositing checks and reading payment stubs to credit
    the right accounts. So the bank is only now catching up with
    processing some of those small payments.
    Mr. Renyi said the bank had quite deliberately planned to have
    different levels of backup for different functions. The crucial
    government bond processing, for example, had a system in which a
    second computer was receiving and processing all the data going into
    the main computer, making it ready to pick up at a moment's notice, in
    The bank had no such system for its 350 automated teller machines. The
    rationale was that its customers would be able to use other banks'
    machines in case of a problem. That is what customers were forced to
    do for more than a week after the attack. (The bank absorbed fees
    imposed by other banks.)
    As it turned out, though, even the expensive backup system was unable
    to get the government bond business up and running smoothly. That is
    largely because of problems maintaining the communications links that
    receive information on trades from its customers and report their
    positions back to them.
    "In many cases our backup sites were dealing with our customers'
    backup sites," Mr. Renyi said. And though the bank had established
    communications lines in advance connecting these various backup
    centers, they often were of low capacity and typically had not been
    fully tested and debugged.
    Even a week after the attack, the Bank of New York was having trouble
    with some crucial communications links, like its connection to the
    Government Securities Clearing Corporation, a central part of the
    government bond market. On several days that week, the bank had to
    drive computer tapes with its trades to G.S.C.C. offices. Though the
    trades were ultimately posted properly, clients were deprived of
    instantaneous reports on their positions.
    "Everyone had redundant telecommunications facilities, but a lot of
    them turned out to be routed through the same phone company offices,"
    said Thomas F. Costa, chief operating officer of the Government
    Securities Clearing Corporation. "We've all learned that when we have
    backup lines, we should know a lot more about where they run."
    The bond market, including the Bank of New York, is getting back up to
    speed and working through its backlog of work. Still, it could be
    months before the Bank of New York can return to its computer center
    to 101 Barclay Street. The bank is planning to build a new data center
    to house its operations in the intervening months. The bank hired
    I.B.M. to help it figure out where to put the center and to look at
    other ways to bolster its backup systems. But Mr. Renyi said the bank
    still plans to move back to Barclay Street so it can stay close to its
    Last week, the bank said the disaster would knock $125 million off its
    profits, although insurance will reimburse much of that. Less clear,
    however, is the long-term effect. Few analysts expect the weeklong
    disruption in A.T.M. service to cause many individual customers to
    leave. But the big Wall Street firms are a more demanding group. The
    only other player in government securities clearing is J. P. Morgan
    Chase, but State Street and Mellon Financial compete, as well, in
    other processing areas.
    Mr. Renyi says companies will certainly talk about whether they want
    to rely on a single provider of processing services, just as the Bank
    of New York, he said, is looking at whether it was too dependent on
    some vendors.
    But he remains optimistic that the bank will pick up some business
    that had gone to other banks. "We can say that we are the only one
    that has been tested in an extreme situation," he said. "If there is
    someone that has a blue-chip recovery plan that has been proven to
    work, it should be the Bank of New York."
    ISN is currently hosted by Attrition.org
    To unsubscribe email majordomoat_private with 'unsubscribe isn' in the BODY
    of the mail.

    This archive was generated by hypermail 2b30 : Wed Oct 10 2001 - 06:29:24 PDT