Almost every company now has a generative AI demo. A chatbot answers questions from the policy handbook. A copilot drafts customer emails. An assistant summarizes loan files in seconds. In the boardroom, it looks finished.
Then it meets real users. The chatbot cites a policy that was retired last year. The email copilot leaks one customer’s details into another’s draft. The monthly API bill arrives at five times the forecast. The engineer who built the demo says the model “just needs a better prompt.”
This is the gap between a prompt tinkerer and a production engineer. Both can make a large language model (LLM) respond. Only one can make it respond accurately, safely and affordably, thousands of times a day, and prove it with data.
For fintech, insurance and other regulated businesses, that gap matters even more. A wrong answer isn’t just embarrassing. It can be a compliance incident. So when you’re hiring generative AI developers, a résumé that says “GenAI” tells you very little. These seven skills tell you a lot more.
7 skills that separate production engineers from prompt tinkerers
1. They build the evaluation before the feature
A tinkerer checks a few outputs, likes what they see and ships. A production engineer starts by building a test set of real questions with known good answers. Then they measure accuracy, faithfulness to sources and hallucination rate on every change.
This is the clearest signal of seniority. Model output is probabilistic, so “it worked when I tried it” proves nothing. Strong candidates can name the metrics they tracked and explain how they used automated scoring alongside human review.
What to ask: “How did you know your last AI feature was good enough to launch?”
2. They treat prompts as code
To a tinkerer, a prompt is a clever paragraph that lives in a notebook. To a production engineer, it’s a versioned asset. It sits in source control, runs against the evaluation set before release and can be rolled back when a change breaks something.
They also ask for structured outputs, such as JSON with a defined schema, so downstream code can validate every response instead of parsing free text and hoping.
What to ask: “What happens in your workflow when someone changes a system prompt?”
3. They can tell retrieval failures from generation failures
Most enterprise AI assistants use retrieval-augmented generation (RAG): the system finds relevant documents and passes them to the model with the question. When an answer is wrong, the cause is usually one of two things. Either the right document was never retrieved, or the model ignored or misread it.
Tinkerers rewrite the prompt either way. Production engineers check the retrieved context first, then fix the real cause. That might mean chunking, embeddings, hybrid search, re-ranking or better source data.
What to ask: “Our chatbot cites the wrong policy. Walk me through how you’d debug it.”
4. They design for failure
Models time out. Providers have outages. Outputs sometimes come back malformed. Production engineers plan for all of it with retries, timeouts, fallback models, output validation and a clear path to a human when confidence is low.
In a payments or lending workflow, “the AI was down” can’t mean “the customer was stuck.” Look for engineers who talk about graceful degradation as naturally as they talk about prompts.
What to ask: “What does your system do when the model returns something it can’t use?”
5. They think in cost per task, not cost per token
A demo that costs a few cents per query can become a serious line item at production volume. Strong engineers measure cost and latency per completed task. They bring them down through caching, routing simple requests to smaller models, trimming context and streaming responses.
The best candidates give you numbers: “We cut cost per resolved ticket by 60% by routing classification to a smaller model.” Vague answers suggest they never ran anything at scale.
What to ask: “Tell me about a time you made an LLM feature cheaper or faster.”
6. They defend the system, not just the prompt
Prompt injection, where malicious input tricks a model into ignoring its instructions, tops the OWASP Top 10 for LLM Applications. Sensitive data leakage and excessive agent permissions aren’t far behind.
Production engineers don’t rely on “please don’t reveal confidential data” in a prompt. They mask personal data before it reaches the model, give tools least-privilege access, validate outputs before acting on them and keep sensitive content out of logs. In regulated industries, they also know the audit trail compliance teams will ask for.
What to ask: “How would you stop a user from making our assistant reveal another customer’s data?”
7. They know when not to use an LLM
The most underrated skill is restraint. Not every problem needs a model, and very few need an autonomous agent. A rules engine, a search box or a fixed workflow is often cheaper, faster and easier to audit.
Tinkerers reach for the newest framework. Production engineers start with the simplest thing that meets the requirement and add complexity only when data shows it’s needed.
What to ask: “Tell me about an AI feature you decided not to build.”
How to test for these skills in your hiring process
A standard coding interview won’t surface any of this. Three changes to your process will.
- Replace the take-home demo with a take-home diagnosis. Give candidates a small, deliberately flawed RAG app and a handful of wrong answers. Ask them to find the causes and propose fixes. You’ll learn more in two hours than a polished demo ever shows.
- Ask for a production story, not a portfolio. Have them walk through one feature that real users touched. Probe what broke after launch, how they found out and what they measured.
- Run a trade-off conversation. Present a business problem, such as automating claims intake or answering customer questions about card fees. See whether they ask about accuracy targets, data sensitivity, volume and budget before they mention a model.
Red flags are easy to spot once you know them. Watch for candidates with only notebooks and weekend projects, no answer on evaluation, framework syntax without system design, or promises that AI will be “100% accurate.”
Build or borrow the talent
Engineers who have all seven skills are scarce, and they are in demand at every company racing to move AI from pilot to production. Many US mid-market firms don’t try to hire them all in-house. They pair a small internal AI lead with an experienced external team that has already shipped and run these systems.
If you go that route, apply the same bar. Ask the partner for evaluation reports from past projects, their security approach for regulated data and a named engineer you can interview. Firms such as JumpGrowth, a Dallas–Fort Worth software and AI consultancy, let companies hire generative AI developers who are vetted on exactly these production skills. They can work as an extension of an existing team or as a dedicated pod.
Whichever path you choose, remember the core lesson. Getting a model to talk is easy. Hire the people who can make it trustworthy.
Frequently asked questions
What is the most important skill when hiring generative AI developers? Evaluation. Many developers can get a model to respond, but far fewer can measure and improve answer quality in a systematic way.
Do generative AI developers need a machine learning degree? Not usually. Most production work builds on existing models through APIs and RAG, which depends more on software engineering. Deep ML expertise matters mainly for fine-tuning or custom model training.
How can a non-technical manager assess a generative AI developer? Ask them to walk through a shipped feature, explain how they measured quality and describe what went wrong. Then pair that conversation with a technical review by a trusted engineer or partner.
What are the warning signs of a prompt tinkerer? Only demo projects, no evaluation method, no awareness of cost or security, and a habit of fixing every problem by rewriting the prompt.
