Key Takeaways:
-
A pilot is an experiment, not an early success. It is only worth running if it is allowed to fail, and if you decided beforehand how you would recognize that.
-
The base rates are against you. Bloor Research found in 2011 that 38 % of data migration projects overrun or get aborted, at an average project budget of $875,000. A pilot exists to find out early which side of that line you are on.
-
The slice decides the value. Migrating the easiest part of a system proves nothing. Choose the slice that carries the most unknowns.
-
Measure duration, not just success. How long a run takes is the number the cutover plan is later built from.
-
A pilot without a rollback plan is a move. Decide how you get back before you go. Otherwise the decision has already been made.
A data migration plan proves its worth in the pilot, and a pilot is set up correctly when it defines, before the first run, which result would disprove it: a run time above X, an error rate above Y, running costs above Z. A pilot meant only to show that "it works in principle" confirms an assumption nobody doubted and still costs six weeks.
We supported such a pilot this month, and were reminded how strongly the outcome depends on which slice you pick.
Why a pilot at all
Because the averages are misleading. Bent Flyvbjerg and Alexander Budzier examined 1,471 IT projects and found an average cost overrun of 27 %. The average was not the problem. One in six projects was an outlier with a 200 % overrun. You do not plan for the average project. You plan so that you notice early whether you are in the dangerous sixth.
That is the pilot's job. Nothing else.
The reflex: start with the easy part
The obvious slice for a pilot is the part of the system where least can go wrong. Little data, few dependencies, no external interfaces. The result is very reliably a successful pilot, followed by a project that walks straight into the problems that were left out.
The pilot is not there to prove the cloud works. It does. It is there to find out what gets expensive in this system during this move.
How we pick the slice
We look for the smallest slice that touches as many open questions as possible at once:
- A data volume large enough for run times to become measurable. A thousand records tell you nothing. A million does.
- At least one external interface, because that is where the network and permission questions live.
- At least one non-relational data type. Files, images, attachments. Reliably the part the plan forgot.
- One business process that runs end to end, not just a table.
That is less comfortable than the easy slice. It is also the difference between an experiment and a demonstration.
What gets measured
Run time. The most important number in the pilot. It is what you extrapolate the real run from, and it determines whether a weekend cutover is possible at all. Measure export, transfer, transformation and import separately. The bottleneck is rarely where you expect it.
Error rate and error classes. Not only how many records fail, but into how many distinct classes the failures fall. Twenty errors in one class is an hour of work. Twenty errors in twenty classes is two weeks.
Running cost. This is where the second run earns its keep. On the first, nobody notices what an environment left running costs, what data leaving the cloud costs, or how much storage a registry accumulates when every build leaves an image behind. In the second month it is on the invoice.
Repeatability. Can the run be started again without someone cleaning up by hand first? If not, it is not a process. It is a one-off.
The second run is the more informative one
The first run shows whether it works. The second shows whether it is a process.
On the second run you notice that someone created a directory by hand in between. That a key was placed manually. That the transformation is not idempotent and produces duplicates when re-run. That a table has to be emptied first and nobody wrote that down.
These are exactly the things that become a problem at three in the morning during the real cutover. So for us a pilot that has run only once is not finished.
{{feature:data-migration}}
The rollback plan
A pilot is reversible by definition. If it is not, the decision has already been made and the pilot was a ritual.
The rollback plan answers three questions. What happens to the data already sitting in the target environment? Which accesses and keys have to be withdrawn? And at which point does rolling back cost more than carrying on? The last one is the interesting question, because it marks the real decision deadline.
What counts in the end
A good pilot ends with a list of uncomfortable numbers and a clear recommendation. A bad one ends with a deck saying everything worked.
We set migrations up so the pilot run is technically the same run as the later production run, only against a different slice. Otherwise you measure something in the pilot that shares nothing with reality except the headline. If you are putting a migration plan together right now, this is what we build.
{{cta:onboarding-playbook}}
Sources
- Bloor Research (Philip Howard): Data Migration research, 2011 update
- Flyvbjerg, Budzier: Why Your IT Project May Be Riskier Than You Think, Harvard Business Review, September 2011