Beyond One LLM: Tracking Knowledge Changes with Neo4j

Session track: Data Intelligence

Session time: 03:30 am - 04:00 am UTC November 12

Session description:

# Abstract LLMs, prompts, and data sources are changing faster than ever. Many organizations now use cloud LLMs, local models, and RAG pipelines together to balance cost, privacy, performance, and vendor independence. However, most AI systems still treat model outputs as isolated responses. As models evolve, it becomes difficult to understand what changed, why it changed, and which knowledge should be preserved and reused. In this session, Koji Annoura will introduce Knowledge Graph Diff, a graph-based approach for managing knowledge as a versioned asset in Neo4j. Inspired by Git, the approach will apply concepts such as snapshots, diffs, branching, validation, and merging to knowledge extracted from LLM outputs. Koji will demonstrate how claims, evidence, concepts, relationships, and perspectives can be represented as an evolving knowledge graph. The session will show how Neo4j can compare graph snapshots, identify added, removed, modified, or conflicting knowledge, and preserve provenance and alternative interpretations across multiple LLMs and RAG workflows. Cloud and local LLMs will be able to contribute knowledge without creating dependency on a single model or vendor. Unlike traditional GraphRAG or agent memory systems that focus primarily on retrieval and memory, this session will focus on knowledge diff: understanding how knowledge changes over time, how perspectives differ, and how feedback can improve a knowledge layer. You will learn how to model versioned knowledge in Neo4j, write Cypher queries to compare graph snapshots, and design a reusable knowledge layer above RAG pipelines and LLMs. You will also learn how graph databases can help organizations retain, compare, and improve knowledge as AI systems continue to evolve. The goal is not to find one perfect answer, but to understand how knowledge changes, differs, and improves over time.