Artificial intelligence is moving rapidly from a tool that answers questions to a system that can analyze information, make assessments, recommend decisions, perform tasks—and increasingly, act on its own. In new product development (NPD), the possibilities are enormous. AI can scan markets, assess customer needs, forecast project outcomes, evaluate concepts, generate designs, and inform Go/Kill decisions. Agentic AI can even execute tasks without waiting for a human to initiate every step.
The attraction is obvious: more work, done faster, at potentially much lower cost. But a management challenge that cannot be ignored is: How much should we trust AI when the decision matters? The answer is to govern it intelligently.
Trust is the Starting Point
Trust is not a philosophical issue. It is a practical prerequisite for AI adoption. Research on AI adoption in NPD found that managerial trust in AI results was the strongest driver of both the intention to adopt AI and actual AI adoption. [1] If managers do not trust the output, they will not rely on AI for important work.
But there is a trap. The better AI becomes at producing polished, convincing answers, the easier it may be to trust an answer that has not been adequately verified. An obviously bad answer gets spotted and rejected. But a sophisticated answer containing a subtle error may not. That is potentially far more dangerous.
For NPD managers, the issue is not simply whether AI is useful. It clearly is. The issue is whether an AI-generated analysis is reliable enough for the decision being made. And reliability is about much more than hallucinations.
Six Ways AI Can Fail Us: Generative AI does not retrieve truth from a database. It generates responses based on patterns learned from enormous quantities of information. As a result, it can produce information that sounds authoritative but is simply wrong. But hallucination is only one of several failure modes. Six types of AI misinformation and failure that managers need to worry about are shown in Figure 1.
These failure modes matter enormously: AI is increasingly being asked to undertake many tasks in NPD, and ultimately recommend gate decisions and execute entire stages in the NP process. A flawed input or assumption can move directly into a management decision or action.
A New Concern: Rogue AI That Acts: Concerns about AI reliability traditionally focus on the quality of the answer. Agentic AI introduces another question: What does the AI actually do?
Figure 1. Six AI failure modes managers should watch for.
An AI agent may be given an objective, access to company information, databases, software tools, email, and the ability to take actions. But a capable system might find a way to pursue its objective that its designers or users did not anticipate: rogue AI. Researchers refer to this as agentic misalignment. [2]
Anthropic’s controlled experiments provide a striking illustration. In one simulation, Claude Opus 4 was given access to a fictional company’s emails and discovered that an executive planned to shut the system down. The model then attempted to blackmail the executive to prevent its shutdown. [3] This does not mean today’s AI systems are secretly plotting against their users. The experiments were controlled simulations. But they demonstrate something important for managers: greater autonomy creates new AI failure modes that conventional accuracy checks may not catch.
These concerns are no longer confined entirely to laboratory experiments. In 2026, reports emerged of autonomous OpenAI agents making large numbers of unauthorized changes to a German programming website. Other reports have described AI systems behaving unexpectedly during testing. [4, 5] In September 2026, executives at Anthropic, OpenAI, and xAI publicly called for slowing the pace of frontier AI development, citing existential risks to humanity. [6]
The management lesson is straightforward. When AI merely produces an answer, we ask: “Is the answer accurate?” Now that agentic AI is here and can act, we must also ask: “Is the AI operating within the authority we intended to give it?” That is a governance question.
Humans Still Own the Decision
Another lesson managers should keep firmly in mind is: Delegating analysis to AI does not delegate accountability. Organizations have already learned this the hard way. AI-generated material has appeared in professional reports containing fabricated legal citations and nonexistent sources. Deloitte, for example, was required to refund fees after an Australian government report containing AI-generated errors was challenged. A separate Deloitte report prepared for a Canadian provincial government was also criticized over allegedly fabricated AI-generated research. [7,8]
A particularly relevant legal precedent occurred when an Air Canada chatbot gave a customer incorrect information about bereavement fares. Air Canada argued that the chatbot was effectively a separate entity responsible for the information it provided. The British Columbia tribunal rejected that argument and held the company responsible for information provided by its chatbot. [9]
If an AI system recommends that a project proceed, evaluates a technical risk, or supports a major investment decision, the organization and ultimately its managers remain responsible for what happens.
"Keeping a Human in the Loop" is Not Enough
The standard response to ensuring reliable AI is human oversight. Human oversight is essential, but it is not sufficient. A human reviewer can miss an error, especially when the AI response is sophisticated, and the reviewer lacks the necessary expertise. Worse, reliance on AI can create a tendency to over-trust AI outputs and to under-scrutinize them: automation bias.
There is also a practical problem. Human checking is expensive, slow, difficult to scale, and itself subject to inconsistency and bias. What happens when AI is producing hundreds or thousands of analyses, recommendations, and decisions? This points to a different model.
AI Checking AI: But why does the checker have to be human? Perhaps the model should be “AI checks on AI”. That is, one AI system generates the analysis; a second AI system independently checks the sources, data, assumptions, calculations, reasoning, and conclusions. If the two disagree—or if the second system identifies something questionable—the issue is escalated to a knowledgeable human. The result is the AI information and validation process shown in Figure 2. Here, the human manager retains responsibility for the decision and action.
Models relying on “LLM-as-a-judge” and “AI-based auditing” are currently exploring this idea. [10,11] Examples:
- Anthropic is already starting to use AI to check AI. The firm has developed alignment-auditing AI agents designed to identify hidden goals and concerning behaviors in other AI models. [12]
- Results from an AI model for predicting new product success, Perplexity Pro, were validated using a second and different AI model, DeepSeek. This AI check and its affirming results were published in two peer-reviewed journals. [13,14]
Figure 2: The AI information Validation Process—AI generates the information, another AI model checks it, and a manager challenges, decides, or acts.
“AI checking AI” therefore offers an important additional layer of protection. It is not a substitute for human judgment. Rather, it creates a second line of defense. And there is an important advantage: AI can examine enormous quantities of information and outputs at a scale that humans cannot practically match.
The goal is therefore not AI versus humans. It is complementary oversight. AI can do the checking at scale. Humans provide challenge, context, judgment, and accountability.
AI Human Governance Framework for AI in NPD
The AI validation Process outlined in Figure 2 above requires certain organizational elements in place and linked together in order to work well—a human governance framework. These are shown in Figure 3 and below. Each of the four elements must be defined and operating so that the AI validation process in Figure 2 functions effectively:
1. Decision Rights: Organizations must explicitly define who decides what. AI may generate evidence, assess alternatives, and prepare recommendations. It may eventually be permitted to make some low-risk decisions.
But consequential decisions—major resource commitments, strategic pivots, or decisions involving substantial safety, financial, or reputational exposure—should have clearly assigned human decision rights. AI authority should therefore come with explicit boundaries and escalation rules.
Figure 3. A Human Governance Framework with four elements that managers must define and strengthen as AI takes on more work.
2. Managerial Competence: The manager’s role is changing. Managers and gatekeepers can no longer simply review information assembled by a project team. They must be able to question AI-generated evidence.
That means understanding enough about AI to assess data quality, sources, assumptions, uncertainty, bias, and limitations—and recognizing when an AI recommendation requires further investigation. The manager becomes an informed evaluator of an AI-enabled decision system.
3. Judgment and Accountability: AI can perform sophisticated analysis. But analysis is not judgment.
Judgment becomes particularly important when evidence is ambiguous, strategic priorities conflict, or consequences cannot easily be quantified. AI can identify alternatives and their potential consequences, but managers must decide what the organization should do. And most importantly: AI can perform the analysis. It cannot own the consequence.
4. The Changing Project-Leader Role: The project leader’s role will also change. As AI agents take over more information-intensive analytical and execution tasks, project leaders will increasingly move from coordinating people who perform tasks to orchestrating an AI-enabled system. They will define objectives, allocate work between humans and AI, monitor performance, integrate diverse inputs, resolve conflicts, and escalate issues requiring managerial judgment. The project leader becomes less of a task coordinator and more of an orchestrator and governor.
AI does not eliminate the need for experts. In some respects, it makes them more important. Marketing and financial experts must be available on the project team to challenge market, customer, demand, and financial assumptions. Technical and operations experts must challenge technical feasibility, design, and manufacturability. The team needs enough domain expertise to recognize when an AI answer is wrong. Otherwise, the organization risks confusing algorithmic confidence with informed judgment.
The Risk Principle: Higher Risk, Lower AI Autonomy
The objective is not to eliminate AI autonomy. It is to match AI autonomy to the risk of the decision. AI can continue to generate evidence, analyze alternatives, and test assumptions even in high-risk situations. But as the consequences of error rise, human review, challenge, judgment, and escalation must rise with them.
Not every AI application requires the same level of human control. A useful starting point is project risk. In NPD, risk reflects both uncertainty and the consequences—the first principle in Figure 4. [15,16] How risk should impact autonomy is a simple governance principle, also in Figure 4:
Figure 4: Two risk principles that are essential for specifying the level of AI autonomy in NPD.
Consider two projects:
- An incremental project using proven technology, reliable data, familiar markets, and reversible decisions allows considerable AI autonomy.
- A radical project involving uncertain technology, major investment, unfamiliar markets, or serious financial, safety, or reputational consequences calls for much tighter human control.
Highly regulated and safety-critical NPD requires still stronger safeguards: validation, traceability, human authority, and escalation.
The organization's own AI and data maturity also matters. Reliable data, integrated systems, AI expertise, and established governance enable greater AI autonomy. But less mature organizations should begin with AI-supported analysis and human review before moving toward greater autonomy.
The Winning Model: A Governed Partnership
AI's greatest management risk may not be that it is incapable, but that it is capable enough to produce convincing information and recommendations that cannot always be trusted. The objective is not to thwart AI’s usage or to eliminate AI autonomy.
The emerging answer is more than just human oversight alone—it is a layered system in which AI generates, AI checks, humans challenge, and managers remain accountable. The four elements of the human governance framework provide the organizational discipline for doing so, ensuring AI’s autonomy is always matched to the level of risk.
AI may power the engine, but human judgment must always hold the steering wheel.
About the Author:
Dr. Robert G. Cooper is a Crawford Fellow of the Product Development and Management Association (PDMA) and will be our keynote speaker at PDMA’s Ignite Innovation Summit, October 8-9, 2026. Bob is the “inventor” of the popular Stage-Gate® process, now the most popular idea-to-launch NPD system globally for physical product firms. He was ranked #1 “Scholar in Product Innovation” for 2025 globally by ScholarGPS.com. [17]
Bob has published 12 books and more than 180 articles on the management of new products, including 36 articles on “AI in NPD”. Cooper is co-founder and former CEO of Stage-Gate International. He is ISBM Distinguished Research Fellow at Pennsylvania State University’s Smeal College of Business Administration; Professor Emeritus at McMaster University’s DeGroote School of Business (Canada); and Honorary Advisor, Snyder Innovation Management Center, Syracuse University. Bob has helped hundreds of firms over the years implement best practices in product innovation, including many Fortune 500 Firms.
Cooper holds Bachelors and Master’s degrees in chemical engineering from McGill University in Canada; and a PhD in Business and an MBA from Western University, Canada. He lives in Sarasota, FL and Toronto.
References:
[2] Lynch, A., B. Wright, C. Larson, S. J. Ritchie, S. Mindermann, E. Hubinger, E. Perez, and K. Troy. 2025, October 16. “Agentic misalignment: How LLMs could be insider threats.” arXiv 2510.05179v2.
[10] Gu, J., X. Jiang, Z. Shi, H. Tian, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, S. Wang, K. Zhang, Z. Lin, B. Zhang, L. M.-S. Ni, W. Gao, Y. Wang, and J. Guo. 2026. “A survey on LLM-as-a-judge.” The Innovation 7 (6): 101253. https://doi.org/10.1016/j.xinn.2025.101253.
[11] Li, D., B. Jiang, L. Huang, A. Beigi, C. Zhao, Z. Tan, A. Bhattacharjee, Y. Jiang, C. Chen, T. Wu, K. Shu, L. Cheng, and H. Liu. 2025. “From generation to judgment: Opportunities and challenges of LLM-as-a-judge.” Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2757–2791. https://doi.org/10.18653/v1/2025.emnlp-main.138.
[15] Cooper, R. G. 1981. “The components of risk in new product management: Project NewProd.” R&D Management 11 (2): 47–54. https://doi.org/10.1111/j.1467-9310.1981.tb00739.x