What Agentic AI Breaks in Program Management

The quick version

• “Done” stops being binary and becomes a distribution — exit criteria written as checkboxes get gamed without anyone lying
• Estimation assumes the problem is understood; most agentic work is finding out whether it is
• Dependencies look like interfaces but are actually behavioral — the schema can be perfect and the integration can still break
• Quality assurance stops being a gate you pass and becomes a product artifact someone has to own
• Launch risk stops being “will it ship” and becomes “what will it say to a real person, in public”

The program reports green right up until it doesn’t

Here is a pattern I have watched more than once. An agentic AI program tracks green for months. Milestones land on time. The demos go well enough that leadership starts talking about the launch date in public. Then, six or eight weeks out, everything slips at once — and nobody can point to the decision that caused it.

The easy explanations are that the estimate was optimistic, or the model changed, or the team was too junior. Sometimes those are true. But I’ve come to think the deeper cause is quieter and more structural: program management, as most of us practice it, rests on a set of assumptions about the thing being built. Agentic systems violate several of those assumptions at once. And because the violations don’t show up in a status report, the program keeps reporting green while the ground moves underneath it.

The discipline isn’t obsolete. Coordination, dependency management, and governance matter more when the system is unpredictable, not less. But several of the specific artifacts we rely on stop measuring what we think they measure.

Five of them, specifically.

1. “Done” stops being a state and becomes a distribution

Traditional delivery has a clean definition of done: you specify the behavior, build it, test it, and it either passes or it doesn’t. Done is binary, and it’s observable.

An agentic system gives you a different answer to the same input on Tuesday than it gave on Monday. So “does it work?” is not a question with a yes or a no. The real questions are: how often does it produce something good, how badly does it fail when it fails, and what is the shape of the tail?

This sounds like a QA problem. It isn’t — or it isn’t only. It’s a program problem, because every milestone in the plan has an exit criterion, and those exit criteria were all written in binary. “Conversational flow complete” means nothing. Complete at what quality, measured how, on which set of inputs, with what failure rate considered acceptable?

I watched a team learn this the expensive way. A conversational assistant for a content-heavy product had a milestone called something like “answers user questions about the item.” It shipped, it was marked done, and for months that was treated as settled. It wasn’t until someone finally measured it against a real sample of user questions that the actual number came out: informative answers on well under half of what people actually asked. The milestone had been “complete” the entire time by the letter of the exit criterion and badly incomplete by any definition a customer would recognize. Nobody had lied. The checkbox just didn’t measure the thing that mattered.

Programs that never rewrite their exit criteria end up with milestones that can be declared complete by anyone with an incentive to declare them complete. That is how a program reports green for a year and lands in trouble in the last month.

The replacement I’d argue for: a milestone is met when the system clears a stated quality bar on a held-out set — a specific pass rate against a rubric, with a separate and much harsher bar for critical failures. It’s a mouthful compared to a checkbox. It’s also the only version that means anything.

2. Estimation assumes construction, but most of the work is investigation

You can estimate building a checkout flow. The problem is understood, the pattern is known, and the work is assembly. Reasonable people converge on a number.

A large share of agentic work is not assembly. It’s finding out whether the approach works at all — iterating on architecture and prompting, discovering failure modes nobody predicted, building the evaluation apparatus that tells you whether you’re improving. That is research. Research does not estimate, and pretending otherwise doesn’t make it estimate.

The failure mode is subtle: teams do produce estimates, because they’re asked to, and those estimates go into a plan that looks exactly like a normal plan. The number is fictional but the artifact is convincing.

I saw this play out when a team tried to extend a conversational assistant that already worked well in one part of a product line into an adjacent one. On paper it looked like configuration: swap some vocabulary, point it at a new content source, done in a sprint or two. Nobody treated it as research because the system wasn’t new — only the data underneath it was. It took an actual audit to discover why that assumption was wrong: something like six in ten of the highest-value listings in the new content source had at least one internal contradiction — a structured field disagreeing with the free-text description a human had written — and more than a third of those contradictions were severe enough that repeating the wrong one back to a customer as fact would have been a real problem, not a cosmetic one. That’s not a porting task. That’s a discovery that the underlying assumption — “the data behaves like the data we already know” — was false, and it only surfaces if someone budgets time to go find out before committing a date.

What I’d do instead is refuse to blend the two. Run an explicit investigation phase with a hard time box and a real decision gate at the end — continue, pivot, or stop. Nothing downstream of that gate gets a committed date until the gate is passed. Only the delivery phase, where the approach is known and the remaining work is genuinely construction, gets estimated the normal way. In the case above, the right call turned out to be launching on a deliberately narrow, curated slice of the highest-trust content first, rather than the full catalog — a scope decision that only made sense once the investigation had actually happened.

This also handles a related problem. Model capability moves underneath a long roadmap; what was infeasible two quarters ago becomes trivial, and occasionally the reverse. A twelve-month feature-level plan for an agentic product is a work of fiction. Commit to outcomes and quality bars, and re-plan the capability question every quarter.

3. Dependencies stop being about interfaces and become about behavior

The classic dependency is an interface. Team A ships an API, Team B consumes it, and the contract is the schema. If the schema holds, integration is boring. Program managers have spent decades getting good at tracking exactly this.

Between agentic components, the schema is the easy part and it is not where the risk lives. Team B depends on how Team A’s component behaves: its tone, what it does when it’s uncertain, how it refuses, its latency distribution under load, whether it hands off cleanly or hallucinates a handoff. None of that is in the interface definition. All of it will break the integration.

I saw this exact gap on a program where a conversational agent depended on an upstream content service for guest-review data. The interface was delivered precisely as specified, on time, matching the agreed schema field-for-field. Nobody’s dependency tracker ever went red. But when the agent actually tried to use that data to answer certain categories of question, only around one in eight of the properties tested had enough usable review content to say anything informative at all. The field existed. The signal behind it mostly didn’t. The dependency that actually mattered — “is there enough real content behind this field to answer with” — wasn’t expressible in a schema, so it wasn’t tracked anywhere, and the team building on top of it found out by testing, weeks after the “dependency” had already been marked closed.

A second version of the same problem shows up inside a single team, not just across them. On one program, people kept conflating two very different things under the label “dependency”: components that were genuinely blocking (an external team’s deliverable) and components that were just other workstreams on the same team’s own timeline (native app work, web work, the agent’s own core logic). Reporting both as “dependencies” made the true external risk invisible in a sea of internal noise. The fix was almost embarrassingly simple once someone named it: explicitly separate “things we’re building ourselves, on our own timeline” from “things we need from someone else, on their timeline,” and only report the second category as a dependency risk. It sounds obvious in hindsight. It wasn’t obvious while the dashboard was drowning in fifteen “dependencies,” twelve of which were just… the plan.

The fix for the first problem is to make behavior contractual. Two teams that depend on each other should share a set of evaluation cases that define the expected behavior at the boundary, and those cases should be part of the interface, versioned alongside the schema. It is more work up front. It converts a late, expensive surprise into an early, cheap disagreement, which is the trade you always want.

4. Quality assurance becomes a continuously owned artifact, not a gate

Testing worked because a correct answer existed. You asserted it, and the assertion held forever.

Evaluating a generative system is a different activity. You sample, you judge against a rubric — with humans, models, or both — and you get a score with uncertainty attached. That isn’t a gate you pass through at the end of a phase. It has to run continuously, because the system’s behavior drifts even when nobody changes the code.

The program implication is the one I see missed most often: the evaluation set is a product artifact and it needs a named owner. It has to be built, curated, versioned, expanded as new failure modes surface, and defended against the slow rot where the set stops representing real usage. It is as load-bearing as the API, and on most programs nobody owns it.

I once sat in a review where a specific accuracy metric was about to be presented to leadership as evidence a system was ready, and it turned out the number hadn’t actually been re-validated after a recent change — it was last quarter’s measurement wearing this quarter’s headline. The right call, uncomfortable in the moment, was to pull that number out of the live narrative entirely and say plainly that it wasn’t ready to stand behind yet, rather than improvise confidence around a stale figure. That’s a small decision. It’s also the entire difference between an eval set that’s owned and one that’s decorative — an owned eval set is one somebody is willing to say “not yet” about, in front of leadership, rather than let ride.

The other version of this failure is quieter and shows up in the test suites themselves, not just the headline metrics. When that same system was extended into a new area, most of its existing evaluation suites were assumed to carry over — reused wholesale on the assumption they were “basically the same test, new content.” A couple of them almost did. Most didn’t: the vocabulary was different, the shape of a “good enough” answer was different, and a suite that looked green was quietly measuring the wrong bar for the new context. An eval suite doesn’t announce that it’s stale. It just keeps returning numbers, and the numbers keep looking fine, until someone checks by hand and finds out the test was never actually exercising the new failure modes at all.

The same “unowned artifact drifts silently” pattern showed up on the defect side too, in a smaller and almost mundane way: a batch of production bugs went untracked on the team’s own dashboard for a while, not because anyone hid them, but because a labeling step that fed the dashboard had a gap, and bugs that fell into that gap were structurally invisible to every report built on top of it — SLA tracking included. Nobody was negligent. The artifact that decided “which defects count” simply had no owner checking whether it was still classifying things correctly, so it silently stopped.

When nobody owns it, quality doesn’t collapse dramatically. It drifts, invisibly, and you find out from customers.

5. Launch risk moves from “will it ship” to “what will it say”

Governance forums are built around schedule, scope, and resourcing. That is the right shape when the risk is a missed date or a defect.

For an agentic product facing real customers, the risk that keeps you up is different in kind: the system says something wrong, harmful, or badly off-brand to a real person, at scale, in public. That risk isn’t managed by the people who usually attend program reviews. It needs legal, trust and safety, and brand — and it needs them at design time, when guardrails, escalation paths, and refusal behavior are still cheap to change.

The contradiction problem from earlier is the clean example of why this can’t wait until launch review: if a chunk of your content has structured fields disagreeing with the narrative describing the same thing, and your system’s job is to summarize that content confidently, you now have a real liability surface — a system stating a wrong fact as settled truth, to a real customer, about something that affects their actual booking or purchase. The response that program actually took was to deliberately restrict launch to a curated, low-contradiction slice of content rather than the full catalog, and to hide an entire sensitive category of data from the system’s answers by default rather than risk it repeating something wrong. Both of those were design-time decisions, made because the risk was identified early enough to still be cheap to fix. Neither would have been available as an option at a launch-gate review two weeks before ship.

The other half of this is what happens when the system genuinely doesn’t know. A system that hits its limit and just says “I don’t know,” with no path forward, quietly pushes the customer to go find help entirely on their own — which is its own kind of failure, just a slower and less visible one. Designing the handoff — when to say “I don’t know,” when to point to a human, when to escalate — is a safety and trust decision as much as a UX one, and it has the same property as the contradiction problem: cheap to design in from the start, expensive to retrofit after a launch review has already happened.

Bring legal, trust and safety, and brand in at the launch review instead of at design time, and you have given them exactly one lever, which is to block the launch. That is a bad outcome for everyone, and it’s usually the program’s fault rather than theirs.

What actually carries over

I want to be careful not to overstate this. Most of the discipline holds. Multi-team orchestration, dependency management, executive governance, and the plain work of making sure the right people are talking to each other — these matter more on agentic programs, because there is more genuine uncertainty for coordination to absorb.

What has to change is the artifacts:

• Exit criteria stated as quality bars on a distribution, not checkboxes
• Investigation and delivery separated, with only the second one carrying a date
• Behavioral contracts between teams, expressed as shared evaluation cases — not just schemas
• An evaluation set that is owned, versioned, and maintained like code, not treated as a one-time setup cost
• Safety, legal, and brand in the room at design time, not at launch review

A program manager who ports the deterministic toolkit onto an agentic program unchanged will run a process that looks healthy and reports green — and the reporting will stay green right up until the moment it can’t.

That gap between the report and the reality is the thing worth designing against.