RKNet Blog... Thing

Author’s Note: The information below is based upon reports which were handed to me during the incident through various sources. I am well aware that at this point there are inaccuracies. I intend to maintain this post as-is to serve as a record of what occurs when information is not disseminated in an organized fashion. For a full report on what happened, there are several sources, but it may be best to go with Rackspace’s finished report on the incident. Once again I appreciate all the feedback, and I am glad that I was able to provide some insights to at least a few people during this business crisis. Thank you.

Just heard from my head IT guy at my office:

Rackspace had a major power outage in its Dallas location due to a truck which crashed into a utility pole, causing it to crash into a transformer, causing it to explode a huge ruckus resulting in a transformer blowing up.

There are backup generators which kicked in immediately to prevent serious data loss and server damage, but it wasn’t enough to power the heating and cooling.

They realized that the infrastructure of generators wasn’t enough to handle it all when the temps were going up in the data center so they shut back down.

So what did they do? They trucked in several (around 6-10) 100 kilowatt large generators to power the HVAC.

These are being hooked up presently and service should be restored soon. They’re flipping servers back online one at a time to ensure maximum hardware integrity.

It’s estimated that it will take around 12 hours to restore the transformer. This means that Rackspace will be running on generators for a while. Could ge interesting.

Key update from my techie!

Unfortunately, the report I was given at 9:19pm stating that the site would be back up within 5 to 10 minutes was not accurate. According to Rackspace, they are still working to restore power-and as of right now they do not have a time estimate for restored service. The tech I spoke to felt that it would be at least two more hours until service was restored.

Also noteworthy: Popular site LaughingSquid is suffering the wrath of this outage as well. They have established a temporary wordpress blog with some detailed rackspace status updates here. I also heard via their twitter stream that 37signals is down due to this outage as well.

And we’re back! 10:55pm EST
Laughingsquid and the company I work for are back up and running. Saw another green light over at the digg page for this article. However it appears at this time that 37signals is still down for the count. I’ll provide more insight from my IT connection if Rackspace has provided any.

11:30pm EST - Well, 37signals is definitely back to life. Still no word from my IT connection. Knowing him, he’s probably locked himself in the server room to avoid a bombardment of sales people asking questions about the situation. Translation: I probably won’t hear a lick of new info until tomorrow. Keeping my fingers crossed. In the meantime, Valleywag’s numerous sources will undoubtedly have one among them with their finger on the pulse of the situation.

11:45pm - heard from my IT dude again. got clarification on the original report (see above), but nothing super-new.

12:51pm EST - Got an update from IT. He went and checked my.rackspace.com. Valleywag has a pretty comprehensive report from them as well, but here are the latest two blurbs from the Rackspace team.

Nov. 12th 9:30PM CST — As of 8:45 p.m. CST, temperatures are stabilizing in the DFW data center. In cases of servers that were proactively shut down to avoid overheating, we are starting the process of bringing the affected machines back online in a phased, gradual way. We are sorry for service disruptions caused by these events and understand how critical this is for your business. Throughout this process, we are making every effort to minimize impact on customer environments and return affected machines back to service as quickly and smoothly as possible.

We continue to work with vendors to re-establish utility power to the facility and will keep providing updates here in the portal.

Nov. 12th 11:30PM CST — As of 10:50 p.m. CST, all DFW servers that were proactively powered down earlier this evening, to avoid overheating, have now been powered back up. The Data Center Engineering team has been working to resolve the power issues caused by tonight’s traffic incident. The team is preparing to transfer machines affected by tonight’s power outage from generator power back to utility power. The servers and devices that were affected by the unrelated event over this past weekend will remain on generator power. We anticipate transferring the machines affected this evening back to utility power within the hour and expect the transfer to be non-disruptive to customer environments. We apologize again for the inconvenience these events have caused and have all hands on deck working fanatically to minimize the impact on your business.

Final Update: with everything that’s gone on in the past day regarding this rackspace outage, I just couldn’t keep up with all the info, and I certainly didn’t want to harrass the company’s RS rep into coughing up more info. I provided what knowledge I had at the time, some of which has been rightfully refuted by further, better information. (As pointed out in some of the comments below.) At this point, news on the accident which queued off this chain of events is available at Valleywag.

The best news is that the person who had the accident is okay. Everything after that is debate over guarantees, serious reputation management challenges for Rackspace, and damage assessment for those who lost time and money.

Wired’s Underwire blog has just delivered quite the scoop!

The Burning Man festival’s traditional “burning man” figure burst into flames a full 5 days ahead of schedule, during last night’s total lunar eclipse.

Keep your eyes on their article, as they’ll update when they have more information as to whether this was a technical issue or arson.

Check out the article now for a huge picture of them hosing down the traditional figure after the incident.

Just to keep things brief:
I dreamt I had a fish tank with a few kinds of viscious, snappy fish in it, and a small octopus and some other kind of shellfish or something. I accidentally screwed up the whole aquairum by collapsing one of the sides leaving only a tiny bit of water at the bottom, and nearly crushing the fish. I set everything back up and added in the appropriate salt water for this collection of pirhana like fish. (One looked like one of those clap-trap deals from Donkey Kong Jr.)
Next time I looked into the fish tank, it was full of irridescent octopi. They would look like small, but relatively normal, bluish and reddish octopi, then they would contract and glow in neon colors with cartoon faces. I watched this in fascination, as more were arrising, born out of the dead bodies of the fish that did in fact die because I had broken the tank.
That’s all I remember now. There was so much more. There has been so much more lately but I unfortunately haven’t had time to record the overall details. I have been told I need to watch Until the End of the World. …I can’t imagine why.

Anyone who still reads at my “old location” knows that I’ve been having a bit of a family crisis. So yesterday while driving to the Portsmouth Regional Hospital, I was fussing with the radio, trying to find some kind of music that didn’t suck. This task is often challenging in this area, as they tend to play the same five Pink Floyd songs, and a mix of recent popular dreck. (I most often listen to 96.7 since they tend to have a pretty enjoyable mix of 60’s, 70’s and 80’s.)

However, periodically I’ll tune in to 106.1 - Portsmouth Community Radio - in time to catch some DJ’s set that introduces me to the interesting, oddball stuff I so thoroughly enjoy sometimes. Yesterday was one of those happy accident days. The two charming ladies setting up the records at the station that day played a unique track called “Japan” by a duo known as Cocorosie.

My immediate impression from the first 30 seconds or so was that Lamb Chop had grown up and started dropping a lot of acid. In two words: Far fuckin Out. As it went on I started to pick out the warm and quirky lyrics themselves, and the choice of instruments that fit the theme. The tune was relatively simple, but that wasn’t a shortcoming, it was a feature that made the song more accessible. At one point the song faded out, and I thought that the next song on the playlist was a lovely, operatic vocal song with minimalist instrumental background. As it turned out, that was an intermezzo of sorts. It was such a striking musical experience that I managed to remember the artist and song title hours later. (A rare feat for me, no matter how much I like what I hear.)

A quick googlin’ turned up one of my favorite lyrics sites: Always on the Run.net. They’ve got a brief bio, lyrics galore, and a couple MP3s available from the group (available from a small icon by their name on the main page).
Hit it up, check it out, and let me know what you think.

The obligatory “if you like” list:
The Flaming Lips
Ween
Poe
That one operatic song from The Fifth Element
Modest Mouse
Lamb Chop, grown up, on acid (not the actual band named Lambchop, mind you)

Hairnet Paradise

Rainbowarriors