Tuesday's Site Outage
 

What a way to start the new year! At 8:12 am today the data center that hosts my server had a major failure that caused 6 hours and 40 minutes of downtime. Sorry for the inconvenience.

For all you techies out there here is the most recent status report on the failure:

The site does still seem to be a little slow but that should clear as the data center is brought back to full capacity.

Thanks for your patience.

Thanks Don....
 

....for keeping us posted. :)

I noticed a few of my other common sites were experiencing difficulties also...the biggie being the Manchester Union Leader site....now you have me curious as to if they are utilizing the same site!

Thanks again and Happy New Year,

Skip

EDIT: Wow, just checked in at the Union Leader and at the top of the fold had the same explanation as yours! Wow, you really do run with the big fish now!!!

Savvis is a huge data center and this outage must have affected thousands of customers including banks, government and media sites, like the Union Leader.

While the outage was a minor inconvenience for us, the big fish must have been going nuts. Fortunately a simultaneous failure of redundant systems is very rare and unusual.

I was a bit concerned when I saw even the DNS servers were down and could not be reached. Glad it is back up. Seems to be at full speed.

Affected Metrocast
 

I sent an email last night to DRH, Pepper, IG and Island Sib. This morning when I logged on I had an email message telling me that the delivery was delayed to DRH and Pepper. Both have cable modems from Metrocast. Then I couldn't access the Union Leader or this website. I did manage to pull up Foster's Daily Democrat which must go thru a different server in New England. Now as the Pink Panther would say..."the mystery is solv-ed.":D

You know Don, I wonder about this though. The issues had to do with routers and circuits today. Seems to me that they weren't very redundant. I'll be curious to know more, but it doesn't seem like everything failed over like it should, which makes me question their policies and procedures.

Hmmmmm.............
 

Oh, the site was down? Thank god, I thought I was still drunk from New Years!!:eek:

They certainly could have used triple redundancy in this case but you make a good point Steve. If I was running a critical business server at SAVVIS I think I'd be asking some tough questions and considering my alternatives. For the travel & tourism sites that run on my server the downtime was more of an inconvenience than a crisis and I wouldn't move the server unless this becomes a regular event.

I'm sure that in response to concerned (irate?) customers they will soon have a plan to make sure this can never happen again.

It seems the problem wasn't directly with any servers or routers at the SAVIS location, it was the fiber links from them to the world. They need to take a hard look at their transport providers.

I believe they did have internal router issues as well. Not 100% sure though if it was related to failover, or a perfect storm of events.

Could this be a larger problem?
 

I have heard and read about several fiber optic cable failures here in Eastern Mass that started last Friday. The one in Ipswich MA has been worked on 24/7 since last Friday. Verizon is in the process of replacing several thousand feet of Fiber Optic cable. I read about some other towns with similar issues over the weekend. Is there any way they could be related, ie; over loading of existing circuits when they transfer the circuits off to another line while they replace the failed one??? Seems a little coincidental to me.

Blue Thunder

the magnitude of yesterday's outage
 

Lycos was down too: http://computerworld.com/action/arti...intsrc=hm_list