.jpg)
.png)
The most damaging automation hire is not an incompetent one. It is a competent one who builds a working automation that no one can maintain.
Six months after go-live, something breaks in production. The automation engineer who built it is no longer engaged. No one internally knows how the workflow is structured. There is no runbook. There is no documentation that a non-technical person can use to diagnose the problem. The fix requires hiring the same person again, or starting over.
This is not hypothetical. It is the standard outcome when the wrong evaluation criteria are used to select an automation partner.
Most manufacturers evaluate automation engineers on two things: portfolio (have they built something similar?) and price. Those are both relevant, but they are not the two variables that determine whether the engagement ends with a maintainable system or a black box that breaks and can't be fixed internally.
The five questions below surface the variables that actually matter. They are designed to be asked directly in a discovery or qualification call, and the answers are usually more revealing than any portfolio review.
The n8n and Retool ecosystems have grown fast. So has the number of people who have spent a weekend building workflows and now offer automation services.
The difference between a real automation engineer and a generalist who learned n8n last year is not always visible in the portfolio. Both might have built a CRM integration or a notification workflow. The difference shows up in three areas that don't appear in a case study:
How they scope. A real automation engineer starts with a workflow audit and a written scope document before writing any automation logic. A generalist often starts building and figures out the scope along the way.
How they handle failure. Every automation encounters edge cases that weren't in the original scope. A real automation engineer designs explicit fallback paths for expected failure modes. A generalist often builds the happy path and handles exceptions reactively.
What they leave behind. A real automation engineer hands off documentation that a non-technical person can use to diagnose and escalate issues. A generalist hands off a working system with no runbook and no maintenance path.
The five questions below expose which type you are talking to.
What a strong answer looks like:
The engineer describes a structured pre-build process: a workflow audit, a written scope document that defines the trigger, inputs, outputs, and exception paths, and a review of the target data sources before any build work begins. They may call it by different names (discovery, scoping, requirements gathering) but the substance is the same: they do not start building until the scope is written down and agreed upon.
Watch out for:
"We start with a discovery call and then begin building." Discovery is not scoping. If the first deliverable is automation code rather than a written scope document, the build will encounter edge cases that weren't anticipated and the scope will expand during the build rather than before it.
What a strong answer looks like:
The engineer describes a specific approach to exception handling: explicit fallback paths in the automation logic, alert routing when an exception occurs (so the internal owner knows immediately), and a documented list of known exception types and how they are handled. They can give a concrete example from a previous project of an edge case that surfaced after go-live and how it was resolved.
Watch out for:
"We handle those as they come up." This means edge cases are handled reactively, in production, without a documented path. The automation will break and someone will have to figure out why. That someone is usually you.
What a strong answer looks like:
The engineer can describe specific documentation deliverables: a runbook that explains what the automation does, what triggers it, what data it touches, and what to check when something goes wrong. A trigger log or error log structure. Notes on the exception paths and what action each one requires. Documentation written for the person who will maintain it, not for a developer who already knows how n8n works.
Watch out for:
"We document as we go" or a vague reference to "technical documentation." Ask to see an example of documentation from a previous project. If they can't produce one, or if the example is written at a developer level with no operational context, the documentation at handoff will not be usable by your internal team.
What a strong answer looks like:
Either yes, with specifics (which ERP, what the API looked like, what complications they encountered), or an honest answer about what they would need to learn: which endpoints they would need to test, whether there are known rate limits or authentication complexities for that ERP, and how they would account for that in the build timeline.
Watch out for:
A vague "yes, we have ERP experience" without specifics. ERP APIs vary significantly. A developer who has connected to Salesforce is not the same as one who has connected to Epicor, Syteline, or a legacy system with no REST API and a SOAP interface. Ask for the specific system. If they have not connected to your ERP before, that is not disqualifying, but it should affect the scope estimate and the build timeline.
What a strong answer looks like:
A defined response process: a specific SLA for acknowledging a production issue, a monitoring setup that alerts them before you have to, an escalation path if the issue can't be resolved quickly, and a clear handoff plan if the engagement ends. They can describe how a production incident has been handled in a previous engagement.
Watch out for:
"Reach out and we will fix it." That is not a process. It is a promise with no defined response time, no monitoring, and no escalation path. Automations in production will break. How the break gets handled determines whether it is a 30-minute fix or a week-long outage.
Run all five in your next vendor conversation. Take notes on the answers. Do not evaluate the responses on confidence or polish. Evaluate them on specificity.
A strong automation engineer will give you specific answers with concrete examples. A generalist will give you confident generalities with no operational detail behind them. The difference is usually clear by question three.
The questions also surface something else: how the engineer thinks about the engagement. One who starts with scope, builds in exception handling, and plans for handoff is thinking about your long-term outcome. One who starts building and handles everything reactively is thinking about their build, not your system.
Before your next vendor call, write down your answers to these five questions as you would want an ideal partner to answer them. Then run the conversation and compare.
The gaps between what you need and what you hear are the evaluation data.
See how ShopFab answers each of these five questions before you book any other calls
.png)
