Most teams treat evals as a flat spreadsheet: a prompt version, a judge score, a pass or fail. That works until you ship your tenth prompt change of the quarter, your escalation-accuracy judge drops three points, and nobody can tell you why.
We run roughly 30 LLM-as-judge evals in production. Each prompt change is a node. Each judge is a node. Each example is a node. Each outcome is an edge. The thing we wish we'd had on day one was a graph that lets us walk that web backward: "show me every prompt change in the last 14 days that touched the routing logic, the judges that regressed, and the examples that flipped."
This talk is about modeling eval lineage as a queryable graph. We'll cover the schema (prompts, judges, golden examples, runs, outcomes, regressions), the multi-hop queries that turn three-day debugging sessions into three-query answers, how the graph reshaped our CI gating on prompt PRs, and the parts of your existing eval stack you can keep flat.
If you're running more than 10 judges and "which prompt change broke this one?" still takes a human a day to answer, you're paying the flat-storage tax.
Audience Takeaways:
1. Why the spreadsheet model of evals doesn't survive past roughly 10 judges and a few months of velocity.
2. A minimum eval-lineage schema: prompts, judges, examples, runs, outcomes, regressions, and the edges that make debugging cheap.
3. The Cypher (or equivalent) queries that turn "what broke this judge?" into a one-shot answer.
4. How to wire the graph into CI so prompt PRs block or certify automatically based on lineage, not just the latest run.
5. What you don't need a graph for: the parts of your eval stack that are fine as flat storage.
We run roughly 30 production LLM-as-judge evals in a regulated AI setting, and the eval-in-CI gate this talk is built on came directly out of two live outages where offline evals were green and prod was dark anyway. The graph model is what finally made the debugging story scale.
Speaker
Rashi Agrawal
Head of AI, Hinge Health
Rashi Agrawal is the Head of Agentic AI at Hinge Health, where she leads the strategic engineering of high-stakes AI systems within complex regulatory landscapes. By expertly navigating the rigorous requirements of clinical safety, HIPAA, and global regulations, she ensures that disruptive technology remains both secure and compliant. Focused on the pioneer side of generative technology, Rashi is architecting "state-of-the-art" frameworks for Agentic AI that move beyond simple automation to solve critical member problems and drive dramatic business growth.
Formerly the Head of AI at GoodLeap, a leading FinTech in Green Energy, Rashi spearheaded enterprise-wide AI transformation initiatives that optimized everything from complex loan processing to customer engagement while managing a complex cross functional global portfolio. With a unique blend of holistic vision and deep technical expertise, she has a proven track record of developing intelligent decision-making systems and risk assessment platforms that deliver measurable business value through Generative and Agentic AI.
Earlier in her career, Rashi led engineering teams at Yahoo, where she transformed early-stage technical challenges into massive growth engines for their multi billion-dollar advertising business. Her approach is defined by a commitment to technical innovation grounded in strategic business transformation, ensuring that AI serves as a competitive moat rather than just a technical layer.
Beyond the office, Rashi is a global explorer who has traveled to over 45 countries, bringing a diverse, international perspective to her leadership. She is an active thought leader in the Engineering Leadership and AI community. An Indian immigrant with a Master’s in Computer Science from San Jose State University, an alumna of Stanford Graduate School of Business Executive Education program, she is also the founder of Women In Tech AI (WIT AI), a community dedicated to empowering and elevating women leaders in the artificial intelligence space.
