AtariAge Downtime

[AtariAge]

Since I know many members of DP also frequent AtariAge, I figured I would post this here so at least some people know why AtariAge vanished off the web earlier this evening.

There was an electrical explosion at 4:55pm CST today at the datacenter where the AtariAge server is located, and the entire datacenter has been offline since then. This affects over 9,000 servers, and at this time they will not be able to get the datacenter back online (even partially) before sometime Sunday afternoon. There's absolutely nothing I can do from my end, except hope that they are able to get things back online sooner (although I don't really have my hopes up, given how they described the damage).

If anyone needs to contact me for any reason, you can do so at albert@austinsystems.com, since I won't have access to my AtariAge email until the server is back online.

..Al

[punkoffgirl]

Holy cripes, an explosion? That's some freaky stuff! I take it as good news that they're able to get it back up, period, then!

[s1lence]

Where was that at Al?

[Ze_ro]

Wow, that's pretty crazy... hope everything works out. Was there any data loss?

--Zero

[Kaboomer]

Thanks for the explanation Al. Hope they get you going again soon.

[jcalder8]

That is crazy. I hope you get the site back up and running soon.

[Mianrtcv]

Kaboom...

Hope all gets worked out asap.

[AtariAge]

Wow, that's pretty crazy... hope everything works out. Was there any data loss?

--Zero

Fortunately, no. The explosion destroyed three walls in an electrical equipment room, but no customer servers or network equipment were apparently damaged. Sounds like a significant infrastructure loss, though. Tomorrow morning they'll probably have a better idea of how long it'll take to get the center back online.

..Al

[AtariAge]

Where was that at Al?

In Houston.

..Al

[NoahsMyBro]

Wow. That's a little stunning. Glad to hear there was no data loss, and thanks for keeping us in the loop, so to speak.

I actually tried to log in over at AA about an hour ago, and just assumed it was down temporarily and would be back up tomorrow AM. I didn't really think much of it at the time.

[Sothy]

I deny any involvement with this explosion.

[tom]

Wow, that's something awful.

[Damaramu]

Explosion in Houston? Hmm....I'll see if I can dig up any info on that.

[deathtrappomegranate]

Ouch!

Thanks for the info. Hope it gets fixed soon.

[Brian Deuel]

Hmm... I wonder if Other World Computing (http://www.macsales.com) is on those servers too. Although their main index page is up, their store pages have been down since last night.

[digitalpress]

At least you have good reasons for your downtime! Once I explained that "an alpaca kicked over the server". Strangely, that was one of the most reasonable excuses I had. That week, anyway.

Thanks for the update, Al, and hoping for a speedy electronic recovery. If there's anything I can do, just give me a shout.

[tom]

Maybe Joe can rent (hire?) you a little corner here at DP, we're a nice bunch you know.

[Dan Iacovelli]

sorry to hear that al, I think rob at atari-users.net might provide you some free space for AA while it's down.(or maybe joe as stated above)

[therealred5]

I just hope it's back up by tomorrow.
I mean, what would I do when I'm at work?
I might actually have to do my job, and nobody wants that.

Kidding aside, hope everything works out quickly.

[AtariAge]

Thanks for the kind words and offers everyone. It would be a gargantuan task to move AA to another server and it would take quite a bit of space. If the whole datacenter burned down, I would already be working on getting a server elsewhere, but hopefully it'll be back online today. Still no ETA on when things will be back online, but they are posting updates every hour..

..Al

[Dan Iacovelli]

Thanks for the kind words and offers everyone. It would be a gargantuan task to move AA to another server and it would take quite a bit of space. If the whole datacenter burned down, I would already be working on getting a server elsewhere, but hopefully it'll be back online today. Still no ETA on when things will be back online, but they are posting updates every hour..

..Al

good to hear, that they are working on the problem.

[Flack]

We had an explosion of sorts on one of our data centers a few years ago. Apparently one of our remote computer specialists managed to install a battery backwards in a UPS. I've replaced dozens of those and I know for a fact you cannot install them backwards. When I talked to the computer specialist I said, "how on Earth did you install a battery backwards?" His response? "Wasn't easy ..."

Hope you get your stuff back online soon. Being offline is frustrating. It feels so ... early 90s. :)

[swlovinist]

Best of luck to you and your quick recovery. AA is an awsome site and you have alot of support over here. Hang in there!

[Lord Helmet]

Well that answers that question. I noticed it was down last night.

[Gentlegamer]

In Houston.Holy crap!

[Sniderman]

Holy crap-noodles! I was wondering what was the cause of the extended outage, but I just figured it was "one of those random hiccups." But an explosion? Damn! Well, I hope all gets fixed quickly!

[Jumpman Jr.]

Cool. Thanks for letting us know.

I have some pending sales over on AA, and I hadn't been able to access the site for a while. I thought to myself "Hmmm.. maybe there is a post on DP about it."

[cyberfluxor]

Sorry to hear. Hoping for the best of a quick recovery on the site. It's a shame that over 9000 servers are affected too!

[MarioMania]

There's a forum I go too sweetin..it's down also

[AtariAge]

There's a forum I go too sweetin..it's down also

There are quite a few sites down today. For instance, the Austin Ridge Riders site (a local mountain biking club) at http://www.austinridgeriders.com is also down as a result. That's the second of two websites I frequent that have been hit by this outage (not including AtariAge).

Their most recent update leads me to believe I'll be lucky to see service restored today at all..


We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.

Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation.

..Al

[MarioMania]

How did the explosion happened

[AtariAge]

How did the explosion happened

I'm not sure they know yet, but I'm hoping they post pictures of the damage later.. Right now 100% of their attention is focused on getting the power restored and everything back online.

..Al

[Ze_ro]

Looks like the news has even made it to Slashdot (http://tech.slashdot.org/article.pl?sid=08/06/01/1715247&from=rss). Don't even try to click that link to ThePlanet's forums...

--Zero

[MarioMania]

Keep us posted Al

[rick weis]

thanks for letting us know Al!

Rick

[AtariAge]

Looks like the news has even made it to Slashdot (http://tech.slashdot.org/article.pl?sid=08/06/01/1715247&from=rss). Don't even try to click that link to ThePlanet's forums...

--Zero

Yeah, their forums are quite slow right now, but since that is where they are posting updates I have been checking on them fairly frequently.

..Al

[Omegamatrix]

My internet provider was doing upgrades last night so I had no internet for about seven hours. When it came back I could get everything, but AA so I thought they blocked me somehow. Hopefully everything will be back up soon.

[AtariAge]

Geesh, just tried to get to sram.com (they make various mountain bike components and I was trying to look up some documentation) and they are down as well. A quick look at the site's IP address reveals that they are also hosted at the same facility..

..Al

[Dan Iacovelli]

the planet is also where statcounter host thier website as well.

[Thomas Jentzsch]

Good to know about the reason. I was already getting nervous.

[mnbren05]

Did someone cut the blue wire?

[phaxda]

Wow, I have really missed AtariAge so glad to hear it's not just me, but on the other hand....an EXPLOSION? Wow.

Thanks for the updates; see you all back on there when things come back up!

[MarioMania]

sega8bit.com forum is down also

[Wraith Storm]

NOOOOOooooooo! How am I supposed to chat about my 1040 STe now???

Seriously though, I'm glad to hear that there was no data loss and hopefully no one was hurt. But that's crazy... An explosion?!?!

[philosophyst]

Thanks for the heads up Al, I was wondering what the hell was going on. I recently switched from IE to Firefox and thought it had something to do with that. I hope you get back up and running soon.

Thanks to DP also, without this post, I would still be clicking that little "AA" link on my toolbar thing every 20 minutes or so.

[deathtrappomegranate]

The hosting company still isn't sounding too optimistic about an early return to service.

Al, LMK if I can help.

[FireTiger]

Good to know I was going crazy!

Ummm- I know I think I was doing something... oh yeah...

How about that feedback forum?

:D

[shadow460]

First and foremost, was anyone hurt in the blast?

OK, I didn't see anyone blow that data center up. I didn't see you throw that man off the balcony, either. All I wanna know is, are you happy with your haircut?

I thought it might have been something on my end, also. That's why I came here, wondering if the site was down.

[Rom Hunter]

Thanks for letting us know, Albert.

Hope everything will be online and OK soon.

8)

[DaBargainHunta]

Thanks for letting us know. I love AtariAge. Keep up the awesome work with it.

[shannon]

Well that sure explains alot! I thought my provider was going bonkers on me!

Feeling the withdraw. :lol:

[Andrew Davie]

There was an electrical explosion at 4:55pm CST today at the datacenter where the AtariAge server is locatedl

God is sending you a personal message, Al.
Cheers
A

[AtariAge]

Just a quick update. The Planet has started to get servers back online, but unfortunately the AtariAge server is located on the first floor of the facility which apparently suffered the most damage. There are 3,000 servers in "Phase 1" (first floor) and 6,000 in "Phase 2" (second floor). The servers in Phase 2 have been coming online this morning, and they hope to restore power to the servers in Phase 1 this evening. I'd be delighted if they actually meet this deadline, but right now I'm not expecting AtariAge to be back online before tomorrow.

For those who are curious, The Planet has created a page where you can view the latest updates (much faster than trying to view their forum):

http://service-update.theplanet.com/

..Al

[RangerG]

I now realize what an AtariAge addict I am (although my wife has said so for years ;-). I hope it gets worked out soon.

[megamaniaman]

Well it looks like the power outage might brought Atariage members and Digital Press members closer together, so there is a silver lining.

[HatNJ]

Hope all endsup well Al.
Like others I thought it was my end or ISP

Take care
Mike

[Phosphor Dot Fossils]

I'll add my voice to those offering to help. I've got server space and mad amounts of bandwidth and I'm not afraid to use it. LOL

[Kaboomer]

I also spread the information on this weeks ChaseTheChuckwagon.com "Auction Talk" episode as we have many AA members registered. I have had a few email me asking if I knew what was happening. I also mentioned this thread to all who asked.

[AtariAge]

I also spread the information on this weeks ChaseTheChuckwagon.com "Auction Talk" episode as we have many AA members registered. I have had a few email me asking if I knew what was happening. I also mentioned this thread to all who asked.

Thanks, much appreciated. :)

..Al

[hex65000]

Yikes!
I was wondering why I couldn't hop on to AA this morning. (AA: The adiction. Ha!) You gotta feel bad for the guys in that data center though... Situation that's pretty much beyond their control and a ton of customers going "Want it up now!" That's not fun at all.

Hex.
[ Has had his career plans tampered with and now needs a new plan... ]

[deathtrappomegranate]

For those who are curious, The Planet has created a page where you can view the latest updates (much faster than trying to view their forum)
Thanks Al.

Since the forum URL was posted on Slashdot, it's become very slow.

[PacManPlus]

Hey Al

Hope it all comes back soon - good luck with them.

Bob

[Sniderman]

Maybe a stupid question, but would it be possible to cobble together some sort of static info page hosted on another server and have www.atariage.com and/or the AA Forums link point to that info page? Or does the main hosted server need to be up to have the redirected pointer in place?

I've a feeling this comes across as a n00b Web question, but I was trying to think of a way to get the info out there to other AA folks who may not have stumbled across this thread and are still wondering what's going on.

[EDIT: Or how about a mass e-mailing using the addresses used for signing up for an AA account?]]

[Thomas Jentzsch]

Maybe a stupid question, but would it be possible to cobble together some sort of static info page hosted on another server and have www.atariage.com and/or the AA Forums link point to that info page? Or does the main hosted server need to be up to have the redirected pointer in place?
Good idea. I wonder why The Planet hasn't set up such a page. Probably it isn't possible?


[EDIT: Or how about a mass e-mailing using the addresses used for signing up for an AA account?]]
I doubt Al has a backup at home. ;)

[AtariAge]

Maybe a stupid question, but would it be possible to cobble together some sort of static info page hosted on another server and have www.atariage.com and/or the AA Forums link point to that info page? Or does the main hosted server need to be up to have the redirected pointer in place?

If I had known it was going to be down this long, I would have done this. I would have to change DNS at the registrar and it could take up to 48 hours to propagate for some people, and then when the server came back online I'd have to change it again (although I could lower the TTL drastically for that). At this point, since they are giving an ETA for late tonight to have the servers back online, I'm not going to do that.


I've a feeling this comes across as a n00b Web question, but I was trying to think of a way to get the info out there to other AA folks who may not have stumbled across this thread and are still wondering what's going on.

[EDIT: Or how about a mass e-mailing using the addresses used for signing up for an AA account?]]

I may do that later today, although I need to install MySQL on one of my local machines and import the forum database first (which is a bit time consuming).

..Al

[iwan-iwanowitsch]

Good to know about the reason. I was already getting nervous.

Heck Yeah, me too! (I tried to hop on AA every free minute......)

[iwan-iwanowitsch]

AA is back!

[AtariAge]

AA is back!

I saw your post before I had noticed this myself. :)

:D

..Al

[iwan-iwanowitsch]

I saw your post before I had noticed this myself. :)

:D

..Al

You are getting old, Al!:snuggle::-D

[cd-w]

... and unfortunately it is down again :(

Chris

[iwan-iwanowitsch]

... and unfortunately it is down again :(

Chris

S%it! :frustrated:

[moycon]

Perhaps they just rigged it on a temp service today to restore power and are now cutting over to a permanent fix.

[deathtrappomegranate]

Work is expected to continue through the weekend, so we're likely to have more outages for some days yet. Hopefully they'll be short.

[tom]

Al stated on AA that the site will be down at times to do some work/checking/maintenance until everything runs smooth again. Not to worry.

[Thomas Jentzsch]

The provider website mentions problems with the air condition in Phase 1 where the AA server is located too.