“Can we shut it down by pulling the plug?” “Yes.”: the migration that was supposed to take an hour

In 2020 there was a migration to do, the kind that on paper is a formality. A server lived in a PoP that was about to be decommissioned and had to move to the new one, the current one, which at the time still smelled new. The plan was touchingly simple: go there, shut the server down, carry it away, switch it back on.

Estimated time: one hour.

The star of the show was a Dell R710 running ESXi 5.5. A server with a few years on it already, but the kind that does its job without complaining. Which, as we will see, is not the same as being ready to start again.

The question

In front of the rack, only one step was missing: the shutdown. At that point the colleague turned to our IT guy and asked the question that, in the history of IT, has preceded more disasters than any Friday-night update:

“Can we shut it down by simply unplugging the power cable?”

And our IT guy answered:

“Yes.”

Three letters. By the time they were out, the fate of the day was already sealed.

The server that never came back

The plug was pulled. The server shut off, like every server whose power you cut.

Silence.

Then came the moment to switch it back on, and the server decided not to take part. It did not start. Not then, not later, not ever again in the form in which it had been left.

What was supposed to be a one-hour move turned into an outage lasting hours.

The plan for the day

The original plan was: shut down, load, transport, power on. The real plan, written on the fly and with no enthusiasm whatsoever, was this:

1. Rebuild the operating system: ESXi 5.5 to be set up again, from scratch.
2. Remount the disks, that is, manage to explain to the freshly born system that those disks, deep down, were its own.
3. Bring back to life everything that ran on top, while the outage went on.

Each of these steps, under normal conditions, is a job to tackle calmly, with a coffee and a backup of the configuration. At that moment there was hurry and an outage in progress; calm, coffee and backup were missing.

The mood

Our IT guy, that day, wanted to cry. It is an important detail, because he was also the one who had said “Yes”.

Some colleagues, meanwhile, were laughing. It must be said that the scene lent itself to it: a man in a technical room, in front of a dead server, mentally replaying the conversation from a moment before. Laughter, in these cases, is a form of solidarity with questionable audio quality.

What a terrible experience. Today, though, it can be told, and that is the only compensation available.

What this story teaches us

• A server is not a lamp. It is shut down from the operating system. With ESXi that means shutting down (or migrating) the virtual machines, putting the host in maintenance mode and only then halting it. The plug comes after, and it is the last step, not the first.
• Before moving a server, save the configuration. The host's too, not just the data. If the system does not come back up, what saves you is knowing how it was built.
• A server that has not been rebooted in years is a server you cannot be sure will start again. A test reboot, done calmly before moving day, costs a few minutes and tells you a lot.
• Size the window for the worst case, not the best one. “One hour” is the estimate for when everything goes well. If the migration involves a real service, the window should be announced wide.
• To the question “can we do this irreversible thing?” the right answer is not “yes”. It is “hold on, let me check”. It takes ten seconds, and sometimes it is worth a whole day.

“One hour,” they had said.