Earlier today we suffered an outage at our mkc1 (Kansas City) data center.  Here are some details about today's event:

• At 2:50 AM (CDT) we detected elevated temperatures on our equipment.  We immediately contacted the facilities people to look into the problem.
• At 3:52 AM the facilities people responded that they were working on the issue and temperatures should start to come down.  Shortly after that we noticed that the temperatures were still rising.
• At 5:05 AM one set of UPSs shut down due to the elevate temperatures.  This reduced our capacity by approximately 50%.
• At 5:32 AM the facilities people responded that the problem was worse than expected.  A specialist was dispatched to investigate the problem.
• At 6:00 AM I was on-site at the data center monitoring the situation.
• At approximately 7:00 AM The A/C specialist was on-site.
• At 8:24 AM The appropriate replacement parts were in the process of being put in to place.
• At approximately 08:30 AM we began restoring our system to full capacity.
• At 9:24 AM The A/C system was back to full capacity.

It was explained to me that this problem was due to multiple simultaneous failures in the A/C system.  The first temperature rise was due to fan failures which resulted in a compressor shutdown in the primary system.  This problem appears to have started about an hour our two prior the warning we received from our equipment.  Some time around 4:00 AM a similar failure occurred in the backup system.  The resulting temperature rise caused the UPSs to shutdown a short time later.

To help prevent this from occurring again, I've been told that additional backup A/C capacity will soon be added.  The added system will also use a separate cooling source for more protection.

Please let me know if you have any questions about this.

-Bob