Micro Tech Systems March 2019

(208) 345-0054 MicroTech Pages

March 2019

The Worst Night How ‘Bad Luck’ Can Help You Improve

M ore than 10 years ago, a new client signed on with MicroTech Systems for server upgrades and integration — and we learned the biggest lesson. This was back when storage and backups of digital platforms were kept on tapes, and we had recommended this client to back up their two servers to this standard technology. So, at 5 p.m. one workday, we took their first server down, put in the tape drive to back it up, and powered the server on again. Everything worked seamlessly, and we did the same thing with the next server until we hit a hitch in the plan. One of the hard drives on the server failed, and while this isn’t ideal, it’s not a huge problem. A server is still salvageable with one hard drive failure because it’s designed for that situation. As we called Dell to mitigate the hiccup in our plan, the second server’s second hard drive failed. When a second hard drive fails, you have data loss; there’s no way to revive it from the dead at this point. It was as if you could have cued the “you’re doomed” music. Here we were, just beginning a relationship with a new client — an attorney, nonetheless — and we were facing big time data loss on this server. In my 20 years of working in the IT field, I have only faced major data loss twice, and this was the first. Around 1 a.m., I sent my partner home to get some sleep so someone of sound mind would be here when the office opened at 8 a.m. In the meantime, I scrambled to move what services and data that I could from the dead server to the first one so the lawyers would at least have something they could do when they came into work in the morning.

Around 5 a.m., I was walking past the dead server when it began showing signs of life. This is another unheard-of phenomenon in the IT industry. Servers don’t just jump back from a total loss. I couldn’t believe it, but I ran with it. I quickly threw all the data from the “zombie” server onto the first server to save it from being lost again. I don’t know how it happened — it shouldn’t have happened — but we were able to save more than 95 percent of that company’s data. I left the client’s office around 11 a.m. and collapsed into bed, having spent the most nerve-wracking night of my life trying to revive a lost server. Visions of being sued for all MicroTech had were leaping into my mind all night as I tried to grapple with possibly costing a client big bucks. If something strange were to happen to MicroTech’s servers and we lost our business, I could still look myself in the mirror. But, if we cost a client their business, I wouldn’t be able to look at myself the same. Now this story could easily be construed into a case of bad, horrible luck that turned suddenly very good, but I believe there’s a different lesson in this story. If we had done a better job of due diligence and prepared for the possibility that the servers could fail, we would have been in a much better position for recovery of data. It wouldn’t have been ideal, but we could have been more prepared for the failure and ultimate stroke of bad luck. This experience motivated us to update our onboarding process with new clients, and it makes us more prepared in the field. We can’t always control when a server dies; some things are left to chance. But you can guarantee we’re prepared now.

–Randy Amorebieta

By 3 a.m., I was on my knees saying, “Man, if I could ever use some help, it would be right now.” Someone must have been listening.

page 1


Made with FlippingBook - professional solution for displaying marketing and sales documents online