Questions to ask before hiring an AI agency
Most checklists list the questions. This one says what a good answer and a bad answer sound like, and gives the answers we would give ourselves.
The questions that matter before hiring an AI agency are about what happens after the demo: who builds the system, whose accounts it runs in, how you will know it works, and what happens when it gets something wrong. Capability is easy to show in a meeting. Ownership and reliability only show up months later, which is why you ask about them first.
Below are the nine questions we would ask any agency, including us. For each one, what a good answer sounds like and what should worry you.
Can you show me something running in production?
A good answer is a live system you can look at, with a number attached: what it replaced, how long it took, what it saves or earns. A bad answer is a demo video, a slide of client logos, or a prototype that was never used by real customers. Demos are easy to build and prove very little about whether a team can keep something working.
Our own rule is that every case study leads with a verified number, such as a page load going from 3.4 seconds to 0.8, or a catalog going live in two days instead of six weeks. If an agency cannot put a number on its past work, ask why.
Who will actually build it?
A good answer names the people, and they are in the room or on the call. A bad answer is 'our team', followed by a senior person selling the project and someone you have never met delivering it. Subcontracting is not automatically a problem, but you should know it is happening and who is responsible when something goes wrong.
Whose accounts will it run in?
This question saves the most pain later. An AI system depends on accounts: the model provider's API keys, the automation platform, the code repository, the hosting, the database, and the domain. A good answer is that all of them are created in your name or transferred to you, with the agency given access rather than ownership. A bad answer is that the agency will 'manage all that', which in practice means you cannot leave without rebuilding.
Our position is that the code and the knowledge are yours from the start, and ownership is agreed on day one. We either run the system or train your team to, but it lives in your accounts either way.
How will we know it is working?
A good answer describes how the system is measured: a set of test cases the output is checked against, monitoring that tracks errors and response quality, and alerts that tell someone when it breaks. A bad answer is that it worked well in testing. Everything works in testing. The distance between a demo and a system that survives real use is most of the job, and it is the gap we wrote about here.
What will it cost to run, not just to build?
AI systems have running costs that a build quote leaves out. Model usage grows with traffic, hosting and databases run every hour of the day, and the data behind the system needs maintenance to stay current. A good answer gives you an estimate for each of those at the volume you expect. A bad answer quotes only the build. We broke those running costs down in what a production RAG system actually costs to run.
What happens when the AI gets something wrong?
It will. A good answer explains how mistakes are contained: what the system is allowed to do without a person approving it, how it behaves when it is unsure, what gets logged so a mistake can be traced, and how a customer reaches a human. A bad answer is that the model does not really make mistakes, which tells you the agency has not run one in production for long.
How will you handle our data?
A good answer says where your data goes, which providers see it, whether any of it can be used for training, how long it is kept, and which country it sits in. That last point matters more in the GCC and Southeast Asia than many agencies realise, because data protection laws there regulate moving personal data abroad. A bad answer is vague reassurance. We cover the basics in whether it is safe to put company data into ChatGPT.
What happens after launch?
A good answer is specific about the months after go-live: who watches the system, how quickly problems get a response, what maintenance is included, and what ongoing support costs if you want it. A bad answer treats launch as the finish line. AI systems need attention after launch because models get updated, data changes, and usage grows, so an agency with no plan for month three is handing you a problem with a delay on it.
Our own engagements end with the system in production and a clear choice: we keep running it, or your team takes it over with documentation and a proper handover. That choice is part of how we scope all of our services, from the data layer up.
What would you tell us not to build?
This is the question that separates advisers from vendors. A good agency talks you out of things: an agent where a simple workflow would do, a chatbot before the data behind it is clean, a custom model where an off-the-shelf one is fine. A bad answer is enthusiasm for everything. If an agency agrees with every idea you bring, it is selling, not advising. Our own default is the simplest tool that does the job, as we set out in AI agent or a simple workflow.
Reading the answers
No agency will answer every question perfectly, and you should be wary of one that seems to. Listen for specifics: names, numbers, account arrangements, and examples of things that went wrong and how they were fixed. Vague answers to the ownership and data questions are the ones to walk away from. If you would like to try these questions on us, put them to us directly.
Questions
How do I know if an AI agency is any good?
Ask to see a system running in production with a measured result, not a demo. Then ask whose accounts it runs in and how they monitor it. Specific answers to those questions are a better signal than a polished pitch.
Who should own the code and accounts when an agency builds an AI system?
You should. API keys, automation platforms, code repositories, hosting, and domains should be in your name, with the agency given access. Otherwise you cannot change agencies without rebuilding.
What is a red flag when hiring an AI agency?
An agency that only shows demos, cannot say who will build the system, keeps everything in its own accounts, or agrees with every idea you bring. Each suggests the system will be hard to trust or hard to leave.
Who should build your Shopify store depends less on the hourly rate than on who owns and runs it after launch. A plain comparison, written by an agency.
For AI work the choice turns less on the rate than on what happens when the person who built the system is unavailable. A comparison for smaller businesses.
The per-question price is the small part. Where the real cost of a RAG system lives, and how to control it.