Key Takeaways:
-
A pilot tests the conditions for the scale-up. So it runs the way the scale-up is planned: same process, same tooling, same rails, only against a smaller slice. A pilot built as a special case teaches you about the special case.
-
Learning is the deliverable. Decide before the first run which result would change the plan: a run time above X, an error rate above Y, running costs above Z. A pilot you cannot learn enough from is worthless, however smoothly it runs.
-
Proven software, proven processes. The pilot is where the organization learns the interplay, so keep the moving targets few: take what is proven, and validate the adaptation to your use case, not everything at once.
-
The base rates are against you. Research puts the share of digital transformations that fall short of their goals at around 70 %. A pilot exists to find out early which side of that line you are on.
-
The slice decides the value. Migrating the easiest part of a system proves nothing. Choose the slice that carries the most unknowns.
-
Measure duration, not just success. How long a run takes is the number the cutover plan is later built from.
-
A pilot without a rollback plan is a move. Decide how you get back before you go. Otherwise the decision has already been made.
A data migration plan proves its worth in the pilot, and the pilot’s job is to test the conditions the scale-up will run under: the same process, the same tooling, the same rails as the full migration, only against a smaller slice. Learning is the deliverable, so the setup decides before the first run which result would change the plan: a run time above X, an error rate above Y, running costs above Z. A pilot built as a special case, or meant only to show that "it works in principle", teaches nothing about the scale-up and still costs six weeks.
We supported such a pilot this month, and were reminded how strongly the outcome depends on which slice you pick.
Why a pilot at all
Because the averages are misleading. In the largest study of IT project risk to date, Bent Flyvbjerg and Alexander Budzier examined 1,471 projects and found an average cost overrun of 27 %. The average was not the problem. One in six projects was an outlier with a 200 % overrun. You do not plan for the average project. You plan so that you notice early whether you are in the dangerous sixth.
That is the pilot's job. Nothing else.
The reflex: start with the easy part
The obvious slice for a pilot is the part of the system where least can go wrong. Little data, few dependencies, no external interfaces. The result is very reliably a successful pilot, followed by a project that walks straight into the problems that were left out.
The setup fails the same way. A pilot that runs on one-off scripts, hand-holding and a path the scale-up will never take produces a result that does not transfer. Whatever you learned, you learned about the special case.
The pilot is not there to prove the cloud works. It does. It is there to find out what gets expensive in this system during this move, under the conditions the full migration will actually face.
How we pick the slice
We look for the smallest slice that touches as many open questions as possible at once:
- A data volume large enough for run times to become measurable. A thousand records tell you nothing. A million does.
- At least one external interface, because that is where the network and permission questions live.
- At least one non-relational data type. Files, images, attachments. Reliably the part the plan forgot.
- One business process that runs end to end, not just a table.
That is less comfortable than the easy slice. It is also the difference between an experiment and a demonstration.
Proven parts, one moving target
A pilot is not only a technical test. It is where the organization learns the interplay: who delivers which data, who approves a result, how long a decision takes when a run is blocked. That learning only happens if the number of moving targets stays small.
So we run pilots on software and processes that are already proven, and keep one variable under test: the adaptation to your own use case. If the pilot also introduces a self-built importer, a tool nobody has operated before and a process invented for the occasion, every surprising number has four possible explanations. You finish the pilot without knowing what you learned about.
Do not invent, learn. Take what works, and validate the adaptation, not everything at once.
What gets measured
Run time. The most important number in the pilot. It is what you extrapolate the real run from, and it determines whether a weekend cutover is possible at all. Measure export, transfer, transformation and import separately. The bottleneck is rarely where you expect it.
Error rate and error classes. Not only how many records fail, but into how many distinct classes the failures fall. Twenty errors in one class is an hour of work. Twenty errors in twenty classes is two weeks.
Running cost. This is where the second run earns its keep. On the first, nobody notices what an environment left running costs, what data leaving the cloud costs, or how much storage a registry accumulates when every build leaves an image behind. In the second month it is on the invoice.
Repeatability. Can the run be started again without someone cleaning up by hand first? If not, it is not a process. It is a one-off.
The second run is the more informative one
The first run shows whether it works. The second shows whether it is a process.
On the second run you notice that someone created a directory by hand in between. That a key was placed manually. That the transformation is not idempotent and produces duplicates when re-run. That a table has to be emptied first and nobody wrote that down.
These are exactly the things that become a problem at three in the morning during the real cutover. So for us a pilot that has run only once is not finished.
The rollback plan
A pilot is reversible by definition. If it is not, the decision has already been made and the pilot was a ritual.
The rollback plan answers three questions. What happens to the data already sitting in the target environment? Which accesses and keys have to be withdrawn? And at which point does rolling back cost more than carrying on? The last one is the interesting question, because it marks the real decision deadline.
What counts in the end
A good pilot ends with a list of uncomfortable numbers and a clear recommendation. A bad one ends with a deck saying everything worked.
This is why we set migrations up so the pilot run is technically the same run as the later production run, only against a different slice. The pilot then measures the scale-up, not a rehearsal of it, and every number it produces still holds when the slice gets bigger. If you are putting a migration plan together right now, this is what we build.
Sources
- Tabrizi, Lam, Girard, Irvin: Digital Transformation Is Not About Technology, Harvard Business Review, March 2019
- Flyvbjerg, Budzier: Why Your IT Project May Be Riskier Than You Think, Harvard Business Review, September 2011