Key Takeaways:
-
File or API is the second question. The first is where the data lands: in a folder that waits for a script, or in a system that validates the delivery and starts the processing on arrival.
-
The transport is not the process. A file sitting on an SFTP server is nothing yet. The process begins when a system has received it, checked it and triggered the next step, with a receipt to show for it.
-
The transport itself can be modest. Hohpe and Woolf list file transfer as one of four fundamental integration styles in Enterprise Integration Patterns. It fails only when the file is the end of the pipeline instead of the start.
-
Real time is an add-on, not a default. It is claimed far more often than it is needed, and it is complex. The benefit has to be checked before the complexity is bought.
-
Approval processes cost more than code. An SFTP transfer is built in an afternoon. Exchanging keys with a large counterparty takes weeks. Plan for the second, not the first.
The first decision in data onboarding is not file versus API. It is where the data lands: in a folder that waits for a script, or directly in a system that validates the delivery and starts the processing on arrival. We build for the second. The transport can be an upload, an SFTP delivery or an API call, depending on what the counterparty can do. Real time is an add-on, and an expensive one, so the benefit has to be checked before the complexity is bought.
We spent several weeks this month exchanging public keys for a partner's SFTP area. Not a line of code. Keys, forms, follow-up questions and ticket numbers. That experience settles the transport question less than you would think, and it sharpens the real one: once the file finally arrives, what receives it?
Straight into a system, not into a folder
The anti-pattern is familiar: files land on a server, a script picks them up on a schedule, and the knowledge about what happens next lives in that script and in one person's head. Nothing is checked on arrival, nothing confirms receipt, and when a file is missing, nobody notices until the numbers are missed.
What we build instead is an intake that treats every delivery as the start of a process:
- Validation on arrival. The delivery is read and checked the moment it lands. Wrong format, missing columns, broken encoding: rejected with a reason, not discovered three steps later.
- A receipt. The delivering side gets a record of what arrived and what was rejected, per record.
- A trigger. Receiving the data starts the processing. No cron job, no "did anyone run the import?".
- Idempotency. The same delivery twice must not create anything twice.
- An archive. Processed deliveries are kept, not deleted. What was transferred can be inspected months later.
- An alarm on absence. The most common failure is not the broken delivery. It is the delivery that never arrives.
None of this depends on the transport. A file delivered into that intake is a fine integration. The same file lying in a folder is a liability.
What decides the transport
With the intake settled, the transport question becomes small and pragmatic:
What can the counterparty do? With a small partner, an API is live in a week. In a large organization every new access passes security review, network approval and operations handover. If SFTP is what their process supports, SFTP into a proper intake beats a beautiful API that arrives two quarters late.
How current does the data have to be? The honest question is: what concretely happens if the information is four hours old? In most cases, nothing. Then a scheduled delivery is the robust choice.
How often do the systems change? An API is a contract. If the source changes every two weeks, the contract changes with it. A file tolerates an extra column.
Gregor Hohpe and Bobby Woolf sorted the styles out back in 2003: file transfer, shared database, remote procedure call, messaging, each trading coupling against timeliness. The style is a choice per flow, not an identity.
{{feature:data-migration}}
Real time as an add-on
Real-time integration is possible, and sometimes it is right: a booking that must not be sold twice, an inventory level that changes by the minute. But it is genuinely complex. Retries with growing intervals, a state per record rather than per run, a queue that survives an outage on the other side, and a versioned contract both sides remember the same way.
That complexity is a price, so it needs a benefit to pay for it. Our rule: the bulk lands in the intake on schedule, and the few events that provably cannot wait get the real-time path as an addition. Proven means someone has answered the four-hours question with a concrete consequence, not with a preference.
Where that leaves the original question
Build the intake once, then choose the transport per counterparty and per flow. The interface-versus-file debate dissolves into a smaller, calmer decision, and the weeks of key exchange stop being a project risk. They are just lead time for one of several ways into the same system.
If you are wiring up a data onboarding right now and want a second opinion on the intake, that conversation is free.
{{cta:onboarding-playbook}}
Sources
- Hohpe, Woolf: Enterprise Integration Patterns, Addison-Wesley, 2003. See the File Transfer pattern.