Here is the latest status on the server.
"Datacenter flooding
Oct 29 2012 08:27:43 PM PT InterNAP has sent us the following notification:
Please be advised that the NYM-EXT1 facility is experiencing significant flooding in the sub-basement of the 75 Broad Street building as a result of Hurricane Sandy. The flooding has submerged the site's diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost. We urge all Internap customers to take necessary remote action, if shutting down equipment in a more graceful fashion is possible before the outage occurs.
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now."
"We hope that flood waters will recede enough that generators can function and/or that utility power can be restored before this happens, but we can't tell if that will be the case.
If a complete power outage takes place, our managed services should not be significantly affected beyond the downtime. This is because all are configured to come back online properly when power is restored, and managed game servers (with the exception of Minecraft servers, which are snapshotted hourly) rarely write to disk, meaning that disk corruption is unlikely. However, unmanaged services would also go down. If you run an unmanaged VDS or machine, we recommend that you back up critical files (such as databases) that might be writing to disk at the time of any later abrupt shutdown, and that you consider stopping those services in advance to be safe."
Post Merge: October 31, 2012, 11:45:16 AM
Back online, I might as well post all the stuff about it, very detailed
"New NYC connectivity loss (but not loss of power)
Oct 31 2012 03:37:22 AM PT We lost contact with all our equipment in NYC again at approximately 5:48am EDT this morning (Oct. 31). InterNAP has not sent out an official announcement yet about this problem, but their NOC answered our telephone inquiry by stating that one of InterNAP's other facilities in NYC is without power, and as a result, our facility does not have connectivity. They tell us that the generators are still working properly for us, which means that your service is continuing to run in the background and should simply pop back online when they get the link fixed, continuing as it was at the start of the outage.
Update @ 7:04am EDT: Connectivity has been restored by InterNAP -- at least temporarily. We hope it sticks.
NYC power restored
Oct 31 2012 12:36:37 AM PT InterNAP has related to us the following:
The generators have been refueled, and engineers have successfully restored power to the site via generator. Please begin checking your systems now. There is still no customer access being granted at this time so please do not make plans to visit the data center until you have been notified by Internap that building access has been restored.
Followed by:
Subsequent to power returning via generator as previously notified, the power was lost again briefly. The onsite engineers are investigating and work to stabilize power is ongoing. The NOC will continue to update customers as the work continues, and confirm once Remote Hands services are available.
Followed by:
As previously notified, the NYM-EXT1 facility has had generator power restored, and engineers have continued to work to stabilize both the generator and additional site infrastructure in support of our customers.
At this time site engineers have been able to confirm that the generator is running normally, and is at full fuel capacity. The brief drop in generator that took place has been fully investigated and it is not expected to be a concern moving forward. Additionally site engineers have returned the UPS systems to service and confirmed that the UPS' are operating normally and helping to support critical loads. Site engineers have also returned all CRAC units to service at this time. The local site engineers believe that the generator and UPS systems are operating in a stable fashion at this time and as noted, we have ample fuel on site and are coordinating additional fuel delivery.
From what they have said, we don't expect further power outages. However, some of InterNAP's upstreams in NYC are still offline and may go up and down, so you may see higher-than-normal pings or slight packet loss at times as InterNAP has to route around them and compensate.
After the last notification, we worked to systematically bring all services back online. All but one of our game server machines are online right now (the customers on the machine that is offline are being contacted separately). Voice servers are also working normally, and VDSes should all be online. If you have a unmanaged dedicated machine of your own at this location and you are not able to connect to it, please contact us ASAP so that we can pull up its console and investigate.
We greatly appreciate your patience as we and InterNAP dealt with the damage from this powerful storm! Total downtime for most services was a bit under 15 hours (11:48am EDT through 2:15am EDT). If you require a day's SLA credit for the downtime, please let us know, and we will be happy to issue this.
NYC location remains offline
Oct 30 2012 05:28:53 PM PT Our NYC location remains offline and InterNAP is working to bring fuel to the generators in order to turn back up the power. They are declining to provide any sort of ballpark ETR at this time.
Please refrain from attempting to change your server. Some customers have tried to move their servers to other locations because of the downtime, but those moves are guaranteed to fail, and they will require significant manual work to clean up after the location is running again. Similarly, server creations in NYC are guaranteed to fail, including creations onto managed VDSes. If you require an emergency temporary server at another location until the location is running again, please contact us through the "Help!" page in your control panel.
NYC location now offline
Oct 30 2012 08:48:00 AM PT Our NYC location went offline at approximately 11:48am EDT (lasting much longer than we expected!), signaling to us that the diesel fuel for InterNAP's generators has been exhausted and that the pumps in the sub-basement are still not working to bring more up. At this time, InterNAP does not know how long it will take before the facility can be brought back online, but as soon as we receive a time estimate from them, we will post it."
