[This is a heartening thread from the nanog network operators list. Kudos to those firms like XO Europe who are taking extra steps to help their colleagues keep the Internet, especially New York City's portions, running during this difficult time. Without the Net, it would be more difficult for friends and family to keep in touch with their loved ones in the city. And, as Sean Donelan points out, sites like nyc.gov (now, alas, offline) are being used to provide vital information about emergency response and shelters. --Declan] *********** From: "Vincent J. Bono" <vbonoat_private> Subject: Re: 32 Old Slip AC Outage and 60 Hudson Verizon failure Date: Wed, 12 Sep 2001 09:28:24 -0400 Status: We are at 32 Old Slip on the 4th floor. AC power is down and our generator air intake is clogged and the sensors shut it down. Batteries are fine (we have 27 hours left) but the HVAC is on generator only. Since we are south of 14th street we are awaiting permission to enter the area to effect repairs. Luckily our staff members who were outside during the collapse are fine which was our main concern. Sincerely, GLOBAL NAPS, INC. Vincent J. Bono Vice President Engineering *********** From: "Vincent J. Bono" <vbonoat_private> Subject: Re: 32 Old Slip AC Outage and 60 Hudson Verizon failure Date: Wed, 12 Sep 2001 12:58:18 -0400 We have a man back at 32 Old Slip now. Most of our issues came from overheat. The generators are now up but utility power is still out. There is emergency lighting only. The generator ran our of fuel once and has been refilled from the building reserve but we are unsure about getting addiitonal fuel at this time. Sincerely, GLOBAL NAPS, INC. Vincent J. Bono Vice President Engineering PS FOCAL is on the 4th floor with us as well as TCG (AT&T Local). AT&T seems to have an equipment failure. -vb *********** From: robertat_private Subject: 32 Old Slip AC Outage and 60 Hudson Verizon failure Date: Tue, 11 Sep 2001 22:36:18 -0400 Our equipment at 32 Old Slip is not operational. This is apparently due to AC power failure and an inability to start the generators due to ash in the air (this may be a choice as well). Anyone running at 32 Old Slip (TCG/Eagle/Focal/GNAPs are the four I know of on the fourth floor) on DC power is on battery or offline. Our POP at 60 Hudson is up with the MFS/WorldCom provided backhaul circuit to our 34 Exchnage Place Jersey City location, but all Verizon circuits terminating at 60 Hudson are down/down. All MFS/WCom circuits seem to be ok so far... -Robert *********** From: Sean Donelan <seanat_private> Subject: NYC city net 161.185.0.0 off net Date: 11 Sep 2001 20:21:54 -0700 New York City's primary network 161.185.0.0/16 including the www.nyc.gov web site is off the net this evening. It was up all day, and providing public information about emergency response, shelters and transit status. I've tried from two different network providers. Applied Theory, NYC's primary name server, is up. *********** From: Alex Bligh <alex.blighat_private> Subject: [nom-steer] Disaster in New York; offer of assistance Date: Wed, 12 Sep 2001 09:43:00 +0100 No doubt everybody reading this will alread be aware of the disastrous events unfold in New York and WDC, and untold human cost. As we try and focus back on running networks, it has become evident that much of downtown NY is currently without power, and many telco hotels are running on backup generators. Some are not being started, or cannot start, due to the ash in the air, and some may stop. At least three colo facilities (WTC obviously, 7 WTC next door, Old slip) are down. 25 Broadway (at least) is running on generator power. It is unknown at this stage how much diesel existing facilities have. At least one C/O is gone, and Verizon is having extensive problems in the downtown area. Most transatlantic circuits land in Manhatten, and most are routed through down town area. Though we didn't see much circuit disruption yesterday (apparently UUnet lost some transatlantic capacity - no doubt they can provide more details), I expect more disruption as generators run dry. This is likely to be a difficult week for transatlantic operators. In the spirit of cooperation in adverse circumstances, XO Europe would like to offer free transit to any European ISP with a presence in the UK, in case of any emergency outage caused by the NY / WDC events (i.e. total lack of transit or severe degredation through). We can bring this up quickly in either Telehouse or Harbour Exchange. Obviously we will need to capacity plan this. Please email or ring me first. Don't just point default. XchangePoint has similarly offered to put in private interconnects at minimal notice if required under similar circuimstances (please contact Brett Wilde there). I would encourage other large ISPs with transatlantic capacity spare to join us in this offer. Please feel free to redistribute this message intact. Alex Bligh Senior Technology Officer, Europe XO Communications - http://www.xo.com/ (formerly Nextlink Inc, Concentric Network Corporation GX Networks, Xara Networks) *********** From: dennis opacki <dopackiat_private> Subject: Reducing power consumption in generator-supplied facilities Date: Wed, 12 Sep 2001 07:31:21 -0700 (PDT) What steps are folks taking to maximize run time of emergency generator equipment and battery in NY facilities currently facing diesel supply issues? Are second halves of redundant systems being temporarily taken off line and non-essential loads being shed in a effort to stretch the time to failure? In a previous life, such precutions were referred to as "rigging for reduced electrical power" and proved quite effective. Thanks, -Dennis *********** From: "Vincent J. Bono" <vbonoat_private> Subject: Re: Reducing power consumption in generator-supplied facilities To: "dennis opacki" <dopackiat_private>, <nanogat_private> Date: Wed, 12 Sep 2001 10:47:23 -0400 Hundreds of generators are not even running thanks to debris in the intake paths. At least in the area as close as our building was. I shudder to think of what's happening in the way of Carbon Monoxide gas release and general overheating issues for those units that do not have smart sensors on them. -vb *********** Return-Path: <jlapolitat_private> Subject: Re: World Trade Center attack Date: Tue, 11 Sep 2001 22:07:50 -0400 (EDT) Avi can you post this to nanog for me, my post access is gone: We have started to lose equipment at 32 Old Slip. The site is not currently off the air, but it is getting hot and several boxes have shut off due to heat. The site is currently (and understandably) unmanned, so we cannot get a status on the site from GNAPS our provider. We are a dial provider and have started to see a drastic reduction in inbound calls. It's unclear what is causing this at this point. -Joe *********** From: "Stephen J. Wilcox" <steveat_private> Subject: Re: Horrible world trade center crash, traffic effect (fwd) Date: Wed, 12 Sep 2001 13:57:29 +0100 (BST) So let see, this concerns North America, theres operators involved.. sounds nanogish. Just to double check the relevance heres a snippet from Telehouse The commercial power supply to 25 Broadway was lost at about 16:35 EDT today. We are continuing to support those loads on generator. We are conservatively stating that we have sufficient fuel for 24 hours or more, but actual fuel usage seems to indicate that could possibly continue for as long as 70 hours. Unfortunately, if the commercial power outage continues for a longer period, we may not be able to replenish the fuel given the state of the streets, etc. in the area. We'll watch this situation closely So yup, theres a good chance everyone will be affected sooner or later. Does that qualify as relevant!? FYI, good stories, pictures and video on www.sky.com Steve *********** ------------------------------------------------------------------------- POLITECH -- Declan McCullagh's politics and technology mailing list You may redistribute this message freely if you include this notice. Declan McCullagh's photographs are at http://www.mccullagh.org/ To subscribe to Politech: http://www.politechbot.com/info/subscribe.html This message is archived at http://www.politechbot.com/ -------------------------------------------------------------------------
This archive was generated by hypermail 2b30 : Wed Sep 12 2001 - 11:45:12 PDT