What I Learned the Day I Got Fired From a $40k AI Engagement

This story is not flattering. I tell it on intro calls because the people I want to work with appreciate it and the people I don't want to work with don't.

I got fired from a $40k engagement after 9 weeks. The CEO ended the call by saying "Josh, you built something genuinely impressive that we cannot use." He paid me in full. I deserved the firing.

Here's what happened.

The engagement

A 60-person professional services firm hired me to build "an AI knowledge management system." Their problem (in their own words) was that experienced staff were leaving and their tribal knowledge was leaving with them.

The CEO had read about RAG systems. He wanted one. He wanted to be able to ask "how did we handle the Smith engagement scope creep" and get the answer.

I scoped a RAG system. Vector database. Embeddings of every internal document. Slack and email integration. Chat interface. Quarterly retraining. The works.

I built it. It worked. It worked beautifully on demos. You could ask it questions and it would surface the right documents and synthesize answers. The CEO loved the demo.

Why it failed

Nobody used it.

Not because they couldn't. Because it didn't solve their actual problem.

The actual problem was that staff didn't know they had a knowledge problem until they hit one. When they hit one, they walked over to a senior person's desk and asked. The senior person answered in 30 seconds. The conversation was the value, not the answer.

When I gave them a chat interface, I gave them another tool to learn, another tab to keep open, another habit to build. For a question that already had a 30-second answer at the desk next to them, why would they use it?

I had built a search engine for a problem that wasn't search. The problem was that the senior people were leaving and the junior people would no longer have a desk to walk to.

What I should have built

A capture system, not a retrieval system.

The right product would have been a thing the senior people used during their normal workflow that quietly captured their explanations as they explained. A Slack channel where junior people asked, senior people answered, and the system indexed the answers. A meeting transcription system that pulled out the "this is how we handle X" moments. A weekly prompt to senior staff: "what did you teach someone this week, give me 90 seconds."

Capture FIRST. Retrieval LATER, once the corpus is big enough to make retrieval valuable.

I built the second half. The first half was empty.

What I missed in the kickoff

The CEO told me what he wanted ("a RAG system"). He did not tell me what the underlying business problem was in his words. I didn't ask hard enough.

I should have spent the first two weeks shadowing junior staff. Watching when they hit a question they couldn't answer. Asking them where they went next. The answer would have been "to Linda's desk." Linda would have been the unit of analysis, not the documents.

Instead I read every internal document I could get my hands on and built around that. I made the documents the corpus when the actual corpus was the conversations.

The kickoff red flag I missed: when I asked the CEO "what does success look like at the end of this engagement," he said "the AI system is live and being used." That's a deliverable. Not a business outcome. A real answer would have been "junior staff can answer client questions that previously required a senior" or "we successfully onboarded the three new hires we're making in Q3 without losing senior productivity to mentoring."

I should have pushed for that. I didn't. Because I'd already scoped what I wanted to build and was already excited to build it.

What got me fired

Week 8 demo, the CEO brought in two senior managers and three junior staff to try the system live.

The junior staff asked the system three questions. Two of the three answers were technically correct and useless. The third was wrong.

The wrong one was a question about an internal billing convention. The system answered confidently using a 2019 document that had since been superseded. The 2024 update was in a separate Slack thread that the system hadn't indexed. The junior staff didn't know to question the answer. The senior manager said "no that's wrong" and the room got quiet.

The senior manager then said the line I think about a lot. "If I have to be standing here every time someone uses this, what is it for."

I had no good answer.

The firing

The CEO called me three days later. He was kind. He said the work was high quality. He said the architecture was sound. He said the system would not solve their problem and they were not going to pay to maintain something they wouldn't use.

He paid me through week 9 (the original SOW was 12 weeks) and ended the engagement.

I tried to argue for two more weeks to add a "human verification" layer. He said no. He was right.

What I do differently now

Three changes in how I run engagements.

1. The first deliverable in any engagement is a one-page "current state document" that I write after shadowing for a week. The client signs off on it before I write a line of code. The doc has to describe the actual workflow, the actual people, the actual decisions. If I can't write that doc, I can't scope the work.

2. The success metric has to be a business outcome, never a deliverable. "System is live" is not success. "Senior staff time on training drops 50%" is success. If the client won't agree to a measurable business outcome, the engagement doesn't start.

3. I now propose a 2-week paid discovery before any build. The discovery delivers the current-state doc and a recommendation. The recommendation may be "don't build anything, hire a knowledge manager instead." Sometimes it is. The client pays for the discovery either way.

Why I tell this story

The clients who don't want to hear this story are the ones who already know what they want built. They want a vendor. I'm a bad vendor.

The clients who lean in when I tell this story are the ones who suspect they don't know exactly what they want built. They want a partner. I'm a decent partner.

This story screens for fit. It also keeps me honest. Every engagement I take on starts with the question "am I scoping the actual problem, or am I scoping the thing I want to build."

The $40k tuition was expensive. I'm still paying it off in the form of being a more careful scoper. Worth every dollar.