Tracing the LLM Behaviour: Debugging and Evolving Agent Workflows with Neo4j

Session Track: AI Engineering

Session Time: 08:30 am - 09:00 am November 6

Session description

Modern LLM apps don’t just answer prompts—they reason, plan, and act across complex workflows. As these systems grow in capability, they also grow in opacity. When something goes wrong (the agent loops, misfires a tool, or takes an unexpected detour), developers are left in the dark. In this talk, we introduce a powerful method for tracing, explaining, and evolving LLM agent behavior using Neo4j. You’ll learn how to: - Model agent execution as a graph of memory hops, tool calls, intermediate thoughts, and transitions - Use Cypher queries to trace failures, detect reasoning loops, and isolate ambiguous decision paths - Apply graph metrics to evaluate agent complexity and fragility - Visualize and replay agent reasoning for debugging and auditability We’ll demonstrate how Neo4j transforms opaque runs into explainable, inspectable agent flows, enabling developers to ship smarter, safer, and more trustworthy GenAI systems.