Today is cascading failure day. Woo hoo.
High-value (and, unfortunately, high-maintenance) customer's hosted instance failed overnight. High-value customer tried to page our 24/7 paging system. Paging system failed because when we migrated that system last week as part of our office move, we neglected to make a critical DNS change that had the Internet thinking that the domain on which the paging system lives was still getting mail at the old FQDN...which is shut down because there is no internet in the office and won't be for another 10 days.
So who got the call? I did. I just finished wrapping up the "Lessons Learned" portion of the exercise (at least my part of it) after working non-stop for four hours.
And now I have a fucking headache and my jaw is killing me where I got a replacement crown yesterday. And it's meeting day, to boot. Arghhhh.
Leave a comment