A techy colleague of mine was taking advantage of a rainy weekend to update his internal infrastructure. For the sake of our story, we’ll call him Bobby. Bobby was migrating his entire server over to brand new hardware on a new rack. Without getting too technical, his server contained several virtual servers running in tandem, controlling his overall network, storing his data, and running his line of business applications. In other words, it was critical to his business. If it were to go down, his employees would be nearly crippled.
Saturday morning, he texted me the specs of the new hardware and a picture of all of the hard drives he was putting into his new server. Jokingly, I told him good luck, and that I hoped he had a backup on hand (I knew he had one, we use the same BDR device).
Server migrations can take a lot of time and labor. The initial labor of installing the server isn’t too bad (that’s the fun part), but there is a lot of waiting and nail-biting as the data transfers over, updates are run, and everything is tested.
Not Everything Goes According to Plan
Around 7pm on Saturday, I texted Bobby to see how he was doing. “Not good, I’ve had to restart the process a couple times. A couple drives that came in were dead on arrival.”
This can happen, hard drives are mechanical, and therefore very sensitive. A lot can happen during shipping that could impact them. It’s usually not a big deal when this happens, hard drive manufacturers have measures in place to return and replace dead drives. That said, it can definitely be a roadblock when you are trying to get this kind of project done within a small time frame. I asked him if he had enough working drives to finish his migration and he said yes.
Hurry Up and Wait
Bobby started the migration process again and left things running overnight. When he got back to the server, nearly nine hours later, the data was only a third of the way migrated. The process was going much slower than anticipated, and Bobby was expecting to spend most of his Sunday getting the new server configured. Instead he was still waiting on data to transfer over.
Sunday night was going to prove to be a late night. The data finished transferring over later that evening, and Bobby had a lot of work to do before 9am Monday morning in order to give his staff access to the files and applications they needed. He texted me the situation and asked what I’d do.
I told him “At this point, I’d flip your BDR (the backup device that he uses) into a temporary server. All of the data is there, it’s all configured, and your staff can run off of that temporarily until the new server is fully operational.”
Bobby told me he’d see how the night proceeds. He wanted his staff to feel the speed and performance of this new server, something I definitely understand, but sometimes you just need to know when you are beat. I wished him luck and said to call me if he needed anything.
The Very Long Night(mare)
Bobby worked through the night. He said that every time he checked the clock, 90 minutes would go by. Soon it was 7am and in a few short hours his staff would start remoting into the network to work. The network was down. The server was down. A little more than half of the applications and services he needed were running. He ran into a few snags throughout the night - nothing catastrophic, but it slowed Bobby down.
At this point though, things were a mess. He almost considered moving things back to the old server for now and starting over, but at this point there wasn’t enough time.
Should he tell his staff to come in late? Should he give everyone the day off?
That’s when Bobby turned to his BDR. The BDR (which stands for Backup & Disaster Recovery) takes complete backups of his network every 15 minutes. He disabled it during his migration, so it was essentially storing the entire state of his server from Saturday morning.
BDR to the Rescue
Bobby fired up his BDR and set it to take over for his server. The BDR wouldn’t perform as well as the new hardware - it wasn’t designed to be as beefy as the new server he had been installing, but it would be able to run all of the applications and dish out all of the network permissions and give everybody access to all of the files they need to work. Plus, it would do all that while still performing backups.
In about a half hour, his network was up and running. He tested everything, and stuck around for his employees to log in to make sure they had everything they needed.
Then, Bobby slept.
Doing It All Over Again
Bobby didn’t sleep for very long. He texted me around noon. “Now that my staff is working and saving data, I just realized I’ll have to redo the entire migration!”
Technically true - the data that was migrated to the new server on Sunday wasn’t up-to-date. If he finished the server configuration, he’d lose a day of changes.
I called him and told him about a feature his BDR had. It’s one of those features that you don’t think about because you usually only have to use it in the worst emergencies, but it’s a heck of a life saver!
The Bare Metal Restore
A Bare Metal Restore is a really cool name for an equally cool (okay, I’m nerding out over this) feature. The idea behind it is this:
Let’s say your server just dies. A brownout shorts it out, or lightning strikes it. It can be literally any scenario, but let’s assume that your server is absolutely toast.
You quickly order a replacement server and have it overnighted to your office. In the meantime, you are limping along by using your BDR as your temporary server. Unfortunately, you couldn’t order the same exact hardware for your replacement server, you had to get something different.
In a normal situation, that means you need to start from scratch, install everything, migrate the data, etc. Expect it to be a long, arduous process before your network is back up and running.
The Bare Metal Restore from a BDR overcomes this. You don’t necessarily need the same exact hardware. A Bare Metal Restore will push all of the data from the BDR to the new server and get it in an operable state, and let you reconfigure things later.
After discussing the situation with Bobby, we determined that the Bare Metal Restore would work for him (it isn’t designed to be a workaround for all server migrations, but in his particular case, it was going to work fine).
He continued to let his staff work off of the BDR for a few days, and on Friday night, he started the Bare Metal Restore. He gave it plenty of time to migrate the data, and came back on Saturday evening, and his new server had everything on it ready to go. He had some tasks to do after - updating drivers, tweaking settings, and that sort of thing, but he could do that on HIS time. The network was back up and running off of the new hardware, finally!
The Moral of the Story
Bobby is a very smart, experienced tech. He didn’t do anything wrong here, other than try to perform a complicated server migration over a weekend. Things can go wrong, and it wasn’t his fault. When things go wrong, it could cost you a lot of time, money, or in his case, loss of sleep.
Having a solid backup solution can save your day in a situation like this, and there is no better solution than the BDR. We use it. We encourage all of our clients to use it. No other solution holds a candle to it.
If you want to talk about your backup, or have your backup tested, or get help with your next IT project, give us a call at (604) 513-9428.
Comments