Just after 12:00am ET, our host had a power problem in a portion of the hosting facility. Apparently, when they got everything fixed, they forgot about a couple of racks, including ours, as our circuit breaker had indeed popped, taking down our entire rack of servers. Unfortunately our network admin (me) was unavailable (fallen asleep) shortly before midnight and wasn't taking calls (is a heavy sleeper and didn't hear the phone). So in total, we were down about 8.5 hours. We have a 100 percent SLA agreement with the host, so we will receive a refund for the time in which we were down. I apologize for the downtime and will plan accordingly to ensure that we have a better chance to recover quickly should this happen again in the wee hours. George
No worries, everyone needs some sleep. We need you well rested to continue to run this site. Hire someone with a sleeping disorder or a meth habit to watch the breaker.
Well, sounds to me like we need a new network administrator... Kidding, kidding... Please don't ban me...
I do happen to know a reliable insomniac who`s currently unemployed. If indeed you need all the angles covered in the future.
Yup, the host lost a generator and the backup one didn't kick on automatically. When they switched it manually, we popped our breaker and our servers fell over again. It's generally not a good idea to let a bunch of servers start up at the same time. So we weren't the only ones that it happened to. A small portion of their network was still down before I troubleshot it.
