Knowledge Management and AI: Revisiting the Need to Apply KM Practices and Technology to Enterprise AI Success

Knowledge Management and AI: Revisiting the Need to Apply KM Practices and Technology to Enterprise AI Success

(coverage image by OpenAI from a prompt written by the author)

The APQC’s 2025 Process and Knowledge Management Conference

At APQC’s 2025 Process and Knowledge Management Conference, the future wasn’t just a theme—the future asked hard questions. APQC went all in on space and astronomical metaphors inspired by their proximity to the NASA Johnson Space Center. My session focused on the emerging, often ignored (so far) intersection between KM and AI. This wasn’t a vision talk. It was a provocation aimed at KM professionals navigating a landscape shaped by guardrails no one understands, models no one can find, and prompts treated like proprietary spells with organizations that have not yet made the connection between AI’s knowledge assets, the lifecycles of those assets, including the communities that create them, and the metadata that should make those assets discoverable.

KM’s AI Challenge

Big AI all too often shills magic. My presentation covered the practical considerations that organizations must address to move from experimentation to production and innovation. I was talking to KM leaders about the need and value of KM to AI implementations. AI does seem to perform magical manipulations of words, images, and audio, but applying it to enterprises requires not only getting internal data ready for knowledge graphs or RAG repositories, it also requires ensuring that the knowledge infrastructure that hones, manages and controls the model and its associated mechanisms aligns with strategic goals and tactical realities.

As AI continues to rewire knowledge work, KM must evolve—not just to keep up but to bring order, accountability, and insight to the systems now making decisions on our behalf. KM must operate as a connective function across compliance, data science, engineering, and operations—not as a siloed team solving AI governance alone. AI will force knowledge management to evolve from knowledge curation and discovery to managing what might best be called “meta-knowledge” management―extending KM to include the management of systems that produce, retrieve, and shape knowledge.

Guardrail Management: Making the Invisible Rules Workable

Commercial tool guardrails are rules without much transparency. The effective use of LLMs requires making these boundaries explicit—documenting intent, cataloging behavioral outcomes, and creating traceable change histories. AI vendors may have those processes in place, but like the guardrails themselves, they are opaque to most users. Any organization adopting an internally hosted LLM must manage guardrails with the same precision and thoroughness that they do other strategic enterprise knowledge. Without KM disciplines wrapped around them, guardrails risk drifting into incoherence or irrelevance. KM should also support communities of practice for those who test, refine, and operationalize guardrails across implementations.

At their core, guardrails are policy embedded in code—coded logic that enforces what an AI system can or cannot say, do, or suggest. But unlike written policies, these are often invisible to the people affected by them. Users don’t see what’s being filtered, redirected, or suppressed. And without context, even well-meaning constraints can cause confusion. For instance, a safety-oriented model might refuse to answer a question about a medication side effect, citing “inappropriate content,” when in fact the inquiry was reasonable and necessary. Without documentation or visibility into the guardrail logic, users are left guessing.

KM can address this by treating guardrails as a form of knowledge architecture—developing change logs, test plans, and rationales for each rule or behavioral constraint. Guardrail management should include:

• Intent Documentation: Every guardrail should have a statement of purpose. Why was it implemented? What risk does it mitigate? What value does it preserve?
• Behavioral Logging and Testing: Guardrails should be tested like any other rule-based system. KM should help catalog where they apply, where they misfire, and how they evolve in response to user feedback.
• Version Control and Governance: Changes to guardrails must be tracked. When an AI stops answering certain queries or begins responding differently, KM should be able to trace that back to a change in logic or policy. This is especially important for regulated industries or applications involving legal liability.
• User Education and Contextual Help: When a guardrail prevents a response, users should understand why. KM can support inline help, reference documentation, and escalation paths that make the constraint transparent rather than frustrating.
• Localization and Equity Audits: Some guardrails may introduce bias by over-censoring certain language or cultural expressions. KM can support audits that assess whether guardrails are equitable, respectful, and context-sensitive across different regions or user groups.
• Community of Practice Facilitation: Guardrails are not static—they should evolve based on real-world usage. KM can convene practitioners across the organization to share experiences, propose adjustments, and build a shared understanding of where AI boundaries should be drawn.

As organizations build and deploy their own LLMs, or heavily customize commercial ones, guardrail management becomes a strategic function. Done poorly, it breeds distrust and dysfunction. Done well, it becomes a living framework for ethical, aligned, and context-aware AI. KM’s job is to take these embedded policies out of the shadows and make them part of the organization’s accountable, shared knowledge ecosystem..

LLM Discoverability

The current chaos in model metadata and naming conventions is a classic KM problem. Models in public repositories lack consistent labeling and meaningful metadata. Repositories need to adopt KM practices and tools such as taxonomies, controlled vocabularies, and agreed-upon metadata fields. We should be able to answer: What does the model do? Who built it? How has it been evaluated?

Finding the “right” model on Hugging Face, for instance, can feel like navigating a library where the books have no Dewey Decimal System, titles are written in five languages, and half the authors used pseudonyms. Let’s say you’re looking for a model trained specifically for summarizing healthcare research in plain language. You might search “healthcare summarization,” “medical LLM,” or “research simplification”—but those terms are inconsistently applied, if they appear at all.

A real-world example: when searching for a summarization model fine-tuned on biomedical literature, you may get everything from a BERT-based tokenizer to a multilingual translation model with “health” in the description. The model card might not tell you when it was last trained, whether it includes PubMed abstracts, or how it performs compared to similar models. Worse, benchmarks are often self-reported, undocumented, or linked to expired GitHub repos.

This lack of standard metadata, inconsistent naming, and absence of enforced taxonomy makes reuse difficult and trust even harder. From a KM standpoint, it’s a metadata governance failure—one that prevents organizations from confidently selecting tools that align with their needs. Discoverability, without clarity and classification, becomes little more than educated guesswork, which is why I often turn to communities to ask questions such as, “which is the best model on Hugging Face for transcribing text from audio?”

Context Model Management: The Hidden Architecture of AI Performance

Context models are the scaffolding behind every meaningful AI interaction. They define what the AI system pays attention to, how much information it retains, what it prioritizes, and what it’s allowed to forget. In generative systems, context isn’t just technical—it’s editorial. It’s a deliberate shaping of the AI’s worldview within the limits of its token window, architecture, and design goals.

At a technical level, context models involve parameters like depth (how far back the model looks), focus (what topics or domains it’s tuned to), and configuration (how the context window is populated and refreshed). But these are also knowledge management issues. KM needs to ask: Who decided what goes into the context? Are these editorial decisions documented? Are they tested against actual use cases? Are they inclusive of diverse knowledge sources, or are they shaped by convenience and default?

Poor context model design leads to brittleness. A model trained to assist in healthcare might forget critical compliance language if the context window fills up with redundant conversation. Or it may drift off-topic if prompts introduce terms outside its configured domain. In agent-based systems, weak context models can result in agents repeating tasks, dropping steps in a workflow, or making recommendations that don’t align with current organizational knowledge.

KM’s role here is to make the invisible visible. That includes documenting assumptions, establishing context lifecycle management, and ensuring that context models evolve alongside organizational priorities. Context isn’t static—it should shift as strategies change, as language evolves, as new data becomes relevant. Without KM at the helm, organizations risk deploying AI systems with outdated or misaligned perspectives, undermining both trust and effectiveness.

Just as metadata governs findability, context models govern coherence. Managing context well isn’t a technical luxury—it’s a strategic necessity.

Prompt Collaboration and Sharing

Prompts are emerging as new knowledge artifacts—reusable, improvable, and highly contextual. KM practices like version control, annotation, and curation are essential to ensure prompt quality. Collaboration platforms for prompts will need everything from user tagging to effectiveness scoring—KM again serves as the backbone.

Several platforms have emerged to address the challenges of prompt management:

TeamAI’s Custom Prompt Libraries: TeamAI provides a centralized repository for prompts, enabling teams to build, share, and scale prompt libraries tailored to specific departmental needs, such as marketing, HR, or customer support .

PromptHub: The PromptHub platform offers a comprehensive suite for prompt management, including version control, testing, and deployment. It supports Git-based versioning and allows teams to organize prompts with metadata, facilitating efficient collaboration across various departments. Heidi Health’s medical knowledge team utilized PromptHub to manage clinical prompts, resulting in streamlined workflows and enhanced collaboration.  

PromptDrive: Designed for seamless integration with models like ChatGPT, Claude, and Gemini, PromptDrive enabled teams to save, share, and iterate on prompts within a unified workspace. Features like folder organization, tagging, and a Chrome extension were designed to enhance accessibility and usability. PromptDrive has recently shut down, but its features remain relevant to current prompt-sharing needs.

Microsoft Copilot Prompt Gallery: Integrated within Microsoft Teams and Outlook, the Copilot Prompt Gallery allows users to save and share custom prompts across their organization. It streamlines the process of accessing and utilizing prompts within familiar workflows .

LLM Model Management and Retirement

LLM Model Management and Retirement: Governing a Shifting Knowledge Base

KM governs obsolescence. When a model retires, KM ensures continuity—tracking what knowledge was encoded, how usage patterns shifted, and how newer models change outcomes―or require new conversational approaches. This isn’t about managing what the models know, but what we know about the models.

Managing the lifecycle of LLMs is more complex than sunset policies or change logs. These are not static knowledge assets—they actively shape how knowledge is interpreted, retrieved, and applied. When an enterprise shifts from GPT-3.5 to GPT-4, or from one vendor’s foundation model to another’s, it isn’t just adopting new features—it’s introducing a new epistemological frame. Language models “know” different things, respond differently to the same inputs, and behave unpredictably across edge cases.

From a KM standpoint, this demands a governance framework that includes:

• Model Version Cataloging: Every system or product embedding a model must document the version in use, the source (vendor or open model), and any fine-tuning applied. This catalog becomes essential for replicability, auditing, and debugging.
• Interaction Provenance: Outputs generated by LLMs should carry metadata noting the model version and configuration at the time of generation. Like a footnote in a research paper, this provides context to the content—especially if decisions are made based on the output.
• Usage Pattern Monitoring: KM teams should analyze how users interact with models over time—what questions are asked, what workflows are augmented, and how responses shift across versions. This insight helps assess whether newer models are delivering measurable improvements or just different answers.
• Change Impact Analysis: Before swapping models or versions, organizations need testing environments to evaluate differences in response quality, hallucination frequency, domain expertise, and guardrail behaviors. Changes should be reviewed like any other critical system upgrade.
• Communication Protocols: Users need to know when the model has changed. Whether through release notes, dashboards, or internal newsletters, organizations should treat LLM updates like software updates that require awareness, training, and sometimes behavioral shifts.
• Retirement and Archive Strategy: Older models may need to be retired, but their fingerprints remain—in decisions, content, and outputs. Organizations should archive model versions and prompts so that historical analyses and audits can be grounded in context. Just as organizations archive former policies and procedures, they should archive LLM configurations and interactions.
• Vendor Dependency Mapping: For commercial models, KM should track contractual terms, model access policies, API limits, pricing structures, and change control provisions. This knowledge is essential when commercial shifts (e.g., price hikes, deprecations, dissolution) force migration or negotiation.

The challenge is that LLMs blur the line between infrastructure and collaborator. They are both system and surrogate—a backend API and a front-line knowledge worker. That duality makes version management not just a technical task, but a strategic KM concern. The models may not “retire” with fanfare, but if organizations don’t manage their transitions, they risk inconsistency, misalignment, and loss of institutional memory. KM must own this transition—not as a technical artifact, but as a knowledge event with real operational consequences.

New Knowledge or Lack Thereof

Generative AI brings us face-to-face with an uncomfortable reality: these systems don’t know in the traditional sense. They reflect past information, often filtered through opaque training processes, without mechanisms to validate, update, or even admit what they don’t know. As organizations evolve—adopting new processes, launching new products, changing policy or structure—AI systems lag. They continue to speak with the confidence of authority, even when their knowledge is outdated, incomplete, or simply wrong.

This gap is more than inconvenient. It’s dangerous in contexts where timeliness and accuracy are critical—engineering specs, legal interpretations, compliance guidelines. KM needs to step in with practices that track knowledge readinesswithin AI systems. That means assessing latency from event to ingestion, creating update protocols tied to business events, and supporting AI observability that lets teams know what content models rely on—and what they may be missing.

Just as important is managing how uncertainty is communicated. Current systems rarely admit when they don’t know something. KM can develop frameworks that help AI communicate boundaries, cite sources, and escalate ambiguous queries. It also needs to train humans to recognize the limits of machine output—to approach generative content as a first draft, not a final answer.

In this new environment, KM becomes a dynamic steward of the organizational learning loop. Its job is to ensure that emerging insights—from the field, from feedback, from analysis—don’t just sit in a database or Miro board, but become part of the systems that power decision-making. The cost of ignoring the “not yet known” is high, and AI will not compensate for our inattention.

RAG and Knowledge Graph Configurations

Knowledge graphs are curated epistemologies. RAG is a dynamic content delivery system. KM ensures that both remain trustworthy. That means auditing sources, aligning structures with strategic knowledge domains, and continuously evaluating inclusion criteria. Otherwise, these systems become black boxes of questionable insight.

Knowledge graphs offer a controlled, structured representation of organizational knowledge. They make relationships explicit, define hierarchies, and enable reasoning over concepts. But they’re only as good as their governance. Taxonomies drift. Ontologies get stale. Nodes multiply without oversight. KM must treat knowledge graphs not as one-time engineering projects, but as living assets—requiring updates, validation, and alignment with evolving strategy. If the product team introduces new features or the org enters a new market, the knowledge graph should reflect that—ideally before the first customer question arises.

RAG (Retrieval-Augmented Generation) complicates things. Unlike a knowledge graph, which is curated and structured, RAG reaches out into the wild—pulling content from indexed documents, databases, or even live systems to ground responses. It promises recency and specificity, but at a cost: the sources it pulls from may be outdated, biased, incomplete, or poorly written. Worse, the retrieval logic might not be visible to end users, creating the illusion of authority without transparency.

KM must play a central role in managing RAG configurations. That includes:

• Source Selection and Vetting: Not every document should be eligible for RAG indexing. KM should define inclusion criteria, tag documents with content quality scores, and ensure that expired or superseded content is excluded or deprecated.
• Content Structure Optimization: RAG works better with well-formatted, clean, and semantically rich documents. KM teams should help structure knowledge assets with headings, summaries, and metadata that improve retrieval relevance.
• Audit Trails and Attribution: Every RAG-generated response should be traceable. KM needs to ensure that source attribution is possible, visible, and accurate—so users can trust, verify, or challenge what the system says.
• Bias and Coverage Analysis: KM must assess whether certain perspectives are overrepresented or excluded. For example, are customer service documents dominating retrieval while legal policies are underrepresented? Is there linguistic or cultural bias in the retrieved sources?
• Alignment with Knowledge Graphs: In well-integrated systems, RAG shouldn’t exist in isolation. It should reference the concepts and relationships in the knowledge graph, filling in dynamic content while staying consistent with curated definitions and taxonomies. KM bridges this divide, ensuring the two systems don’t contradict each other or create fragmentation.

Without this kind of oversight, RAG and knowledge graphs risk becoming knowledge silos—one too rigid, the other too loose. KM’s responsibility is to orchestrate the relationship between fixed knowledge structures and adaptive content generation, preserving integrity without sacrificing agility. These are not just technical configurations. They are epistemological decisions that determine how organizations reason, explain, and act.

Knowledge Management and AI: Recognizing the need for applying KM to AI

The integration of AI into enterprise knowledge work is not a plug-and-play moment—it is a transformation. One that exposes gaps, amplifies inconsistencies, and forces organizations to re-evaluate how knowledge is created, stored, retrieved, and trusted. As this shift unfolds, knowledge management must step into a new role—not as a passive repository builder, but as the operational core of AI governance, design, and use.

AI systems are not neutral tools. They make assumptions, automate decisions, and increasingly shape the strategic narrative of the organization. Prompt libraries become internal language standards. Guardrails encode institutional policy. Context windows define relevance. Knowledge graphs shape what is understood and what is ignored. These are not engineering details. They are strategic business decisions—and they require the full weight of KM disciplines to manage them responsibly.

Each AI component—from prompt engineering to agent orchestration—sits atop a knowledge foundation. When that foundation is weak, outcomes become unpredictable, inconsistent, or misaligned. KM brings the methods, language, and frameworks to interrogate these systems: Where does knowledge come from? How is it structured? Who decides what’s included? What happens when knowledge changes?

Organizations that succeed with AI will not be the ones that chase the latest model or jump on every feature release. They will be the ones that treat AI as a knowledge problem—one that requires lifecycle thinking, cross-functional ownership and shared accountability. KM provides the connective tissue between technical execution and strategic purpose.

This is not just a new frontier for KM—it’s a proving ground. If KM can rise to the challenge of AI, it will solidify its place not as a support function but as a critical discipline for navigating uncertainty, building trust, and sustaining organizational intelligence in a rapidly shifting world.

For more serious insights on AI, click here.

Did you enjoy this post on Knowledge management and AI? If so, like it, subscribe, leave a comment or just share it!