Your AI is operating with half a brain. Here’s the missing piece

There is a growing disconnect between AI’s promise and the ability of enterprise AI projects to deliver tangible productivity and ROI gains. Recent research from MIT shows a staggering 95% failure rate for GenAI pilot programmes. The timeline for meaningful progress isn’t encouraging either. Andrej Karpathy, OpenAI Co-Founder, recently suggested that truly agentic AI systems might not arrive for another decade. What’s driving this gap? Today’s LLMs prioritise plausibility over precision – they’re optimised for persuasive responses, not necessarily correct ones.

None of this should come as a surprise. Enterprise systems don’t operate in a vacuum. They are bound by demanding parameters – security protocols, privacy regulations and compliance mandates alongside a host of other governance frameworks. Without strong guarantees around accuracy, explainability, security, governance and privacy, even the most promising AI deployments are unlikely to progress.

However, these stalled efforts, while frustrating, are actually doing exactly what they should. In fact, they indicate a positive environment in which organisational safeguards are functioning as they should. One of the most common failure points sits beneath the model layer entirely with a fragmented data foundation. Most AI systems are expected to reason without the necessary context. As a result, they’re often working with outdated information that lacks training on an organisation’s specific domain expertise. What’s more, most AI systems still operate without meaningful context. LLMs and vector stores have little understanding of user intent, access constraints or explainability.

Despite this, the encouraging news is that each of these challenges has a solution: an AI context layer based on a knowledge graph.

Understanding the limitations of LLMs

Yann LeCun, former Chief AI Scientist, Meta, says that LLMs will continue to hallucinate until they embody the four characteristics of intelligent behaviour:

1. Understanding: Capacity to understand the physical world (to which I would add digital).
2. Persistent memory: Ability to remember and retrieve things.
3. Reasoning: Ability to reason.
4. Planning: Ability to plan.

His assertion is that LLMs fail to meet these characteristics in anything more than a very primitive way. As LeCun puts it: “If you expect a system to become intelligent without having the possibility of doing these [four] things, you’re making a mistake.”

Consider the human brain as a comparison. Right-brain behaviour is often characterised as creative and impulsive – brimming with brilliant ideas, lacking in self-reflection and sometimes including ideas that a sane person would never act on. The right brain is great at coming up with new ideas, but typically lacks understanding, persistent memory, reasoning or planning – much like AI systems today.

The left-brain, in contrast, is associated with detailed understanding, logical reasoning, fact-based memory – the capabilities that tell your brain when a crazy idea is closer to a hallucination than a business plan. A knowledge graph can serve as the left brain in an AI system. It represents the types of connections, past experiences and most important relationships that help present the LLM with the best choices given knowledge of the past.

We can build on this analogy further:

• Right brain – LLMs (and vectors) exist outside discrete and understandable facts that can be directly communicated or explained to humans or even to other machines. LLMs are statistically-inferred and opaque word prediction engines whose behaviour – as seemingly amazing as it can be – is entirely based on statistics around word frequency and proximity. Like the proverbial right brain, LLMs are impulsive, inscrutable, not entirely predictable and mostly right, but sometimes spectacularly wrong. This core part of the AI stack behaves in ways that are mostly functional, sometimes dysfunctional and always opaque.

• Left brain — Knowledge graphs store precise details about the facts most relevant to decisions. Knowledge graphs also capture the essential relationships between these same facts. Much like their LLM neural network counterparts, the structures used to store and process data mimics the mechanisms inside the brain.

Data is stored in ways that are understandable by humans, but can also be executed upon by machines.

They structure knowledge in a way that lends itself to gating via data access controls, providing AI with a sorely-missing capacity for discernment.

While you can’t ask them random questions using arbitrary language constructs in the way you can an LLM, a knowledge graph can provide rich context to it, so that it can make a better decision.

Moreover, graph databases can provide exact answers to complex high-stakes questions, complementing LLMs’ creative abilities with optionality for exact, deterministic answers.

Let’s not forget that some questions do still have exact answers!

Much as the brain’s two hemispheres work together to produce more robust outputs, the explicit knowledge and connections available in a knowledge graph can help LLMs provide better answers. They do so by providing rich, specific context as input. This context can include more nuances about the objects, relationships and rules involved in any given question. It can also include weights resulting from context-based computations (commonly known as ‘graph algorithms’), which use the emergent shape of the network of knowledge to improve results.

Two common examples are:

• PageRank, which originated with Google as a better way of ranking relevant results and is often a better way to rank vector results.
• Graph neural networks (GNNs), which numerically describe the way data is shaped, and can be used for topological similarity (such as, does this person’s behaviour look more like a high-value customer or a fraudster).

When does your AI need both hemispheres?

This framework offers a straightforward way to help choose the right architecture for an AI system. If the stakes are low where a probabilistic answer is good enough, and where context, explainability and the ability to gate results based on access controls aren’t important, then a more right-brain solution comprising LLMs and a vector database will do just fine.

If the stakes are high, however, there’s a good chance you’ll need a knowledge graph to get your application across the prototype-to-production chasm.

Consider AI applications along the following spectrum:

• At one end lies pure creative tasks with a human in the loop. You have writer’s block and don’t know where to start. You need a creative partner to help you get an idea off the ground. Or you have a language-specific task like summarising meeting notes. These scenarios tap directly into what LLM’s do best, which is language and creativity. Here, the occasional hallucination is far less of an issue – and in some cases arguably a feature.

• The opposite end of the spectrum houses business-critical applications where there is virtually no room for error. These systems, agentic or otherwise, are responsible for the core business operations. Normally when the value of a good decision is high, the cost of a poor decision is even higher. In the best case, a misstep hurts the bottom line and at worst, it damages reputation and brand, violates regulations, compromises system security or even threatens health and human safety.

Decisions at this level demand elevated AI accuracy standards. Requirements extend beyond precision, encompassing auditable and provable results to ensure trust among stakeholder and regulatory alike.

Customer service copilot application is at the heart of the spectrum. The stakes here remain moderately high, but having humans in the loop provides a crucial buffer. When professionals apply their judgement and common-sense checks, the pressure on AI accuracy and explainability eases somewhat. Quality responses and current context still matter greatly, yet there’s room for occasional imperfection.

Connecting this back to the analogy of the brain: simple creative problems can thrive working with only a right brain. However, as the stakes climb, left-brain capabilities become essential. We sometimes joke about humans operating on autopilot, but the reality is that humans function with two hemispheres engaged – and so should your AI systems.