This is an admittance of failure, by one party, me. Today was working on a load of issues at work, and lost access to my home IM server. This usually means network manager on my laptop has dropped the VPN, and just need to redial.
However today, slightly different, in that my VPN GW had fallen over. Luckily this is a virtual machine on one of my servers in London, so just have to reset the guest via VMWare. Unfortunately, this is on one my older servers, still running VMWare Server 1.x, and also for added fun, I can only access the Host via the VPN, seeing the problem here?
Well I do have a ‘backdoor’. Anyways I digress, whilst I was resetting the guest, I thought I’d take a look at what snapshots I had for it.
After clicking the second button, I quickly remembered, VMWare Server 1.x, only has one snapshot. Oh dear, within a blink it was restoring to the previous snapshot, with no way to stop it.
Checking further, this snap shot was old, very old, about 3 months old. So it booted back up. Here comes the second fail. The server in question, happens to be the master, of my MySQL servers, so decided to quickly replicate it’s self, wiping all the new data from the other nodes. Is also my primary NS, so replicated those too. It also does a shit load of other things, that I wont go into, but that broke lots too.
Alas, shouldn’t be a problem though, as I have a four hourly rsync of the configs taken, and replicated off site. Here comes the third and forth fail. 3) the MySQL dumps, appear to have stopped working, looks like there was a regression in one of the scripts. 4) I’d been working on the offsite nas a few months back, and didn’t want the rotation of backups to occur whilst I was doing the work. You guessed it, I forgot to switch it back on. Fuck!
These are all my own fault, I have no way to deny. I spend most of my working day fixing other people’s silly mistakes like this, and setting up systems / procedures so this doesn’t EVER happen, you’d think I’d actually tick a few things off my personal to do list, huh?
Nevermind, managed to fix it all in the end, well mostly… fail fail fail fail fail