Doing your homework with Neo4j & AI

Building graphs from unstructured documents

Tools are usually (well, at least sometimes) very good at doing what they are designed to do. It is hard to argue that a screwdriver isn’t good at tightening loose screws (at least if you use the correct screwdriver). But sometimes you find innovative ways to use tools for something other than what they were designed for. It turns out that screwdrivers are also very good for opening paint cans.

In the same spirit I will use one of the latest tools from Neo4j for something other than what it was intended for. And isn’t this really the beauty of software, that it is so versatile (maybe even more so than a screwdriver)?

The use case

I have two teenage kids whose favourite task is not studying. Does anyone else recognise that? As parents we have to get involved, both to nag (and nag and nag) and also to help by quizzing them on their homework or tutor them on their math problems (soon discovering that ninth-grade maths is way beyond what you remember from school). Wouldn’t it be nice if AI could help out with some of this? Well, I guess they already can get help with math problems (even though LLMs are surprisingly bad at even the simplest arithmetic, like addition); and I don’t think we are at the point where we can trust AI to do the nagging for us.

I must admit (don’t let my kids see this) that when I was a kid I really hated studying and wasn’t a major fan of school in general. But I did like computers. So I wrote a program on my Amiga to help me with my English vocabulary, and that turned out to be a big help.

Testing vocabulary is very binary though, and easy for a computer to do (even in the ‘80s), but to have the computer quiz you on material for a history exam is a lot harder. But with the help of AI it should be possible. You could do this by just sending the entire text they are studying as the prompt and instructing it to form questions from that. But this is a rather unreliable approach and the LLM tends to be very narrow in the questions it formulates.

Let’s instead take a more structured approach using Neo4j and a new tool that was just released. This all works in Aura Free, so no need to spend any money to do this. For the last part (evaluating the responses) I will present two options, one that you can do without cost, and one where you need an API key for OpenAI, Bedrock, Vertex AI or similar.

The homework documents

For this to work you need the homework they were meant to study as a PDF (and it needs to be as a text-based PDF, not a PDF with scanned images). In this digital age it seems quite likely that they already have it in digital form and you can just save it as a PDF. If, instead, they have it on paper you may need to scan it or take a picture of it and then use OCR software to convert it to text. If you don’t have access to OCR software you could also just send the scans to your LLM* of choice and have it convert them to text, that is something it is actually rather good at.

For our example we’ll pretend we have a history assignment regarding the Swedish-Danish war of 1657–1658, where the Swedish king marched the bulk of his field army across the frozen Danish straits to defeat the Danes, and we have the study material as a PDF:

https://github.com/ChristofferBergman/homework-helper/blob/c4fdf30de28f233ee96ced7ea1eb1f439338182c/MarchAcrossTheBelts.pdf

This file is just a dump of the Wikipedia page on the topic, but it could of course be an OCR’d scan of the relevant chapters of the textbook from school.

Document Intelligence

The new Neo4j tool I mentioned is Document Intelligence. This takes unstructured documents (like a PDF)* and creates a Neo4j graph from that, adhering to a graph model you provide. The tool looks at the document(s) and proposes a model from that, which you can modify to produce the graph you want.

The purpose is to be able to load large quantities of unstructured data (like the Epstein files maybe?) and be able to do analysis on that. It could be graph analysis like what was done on the Panama Papers (pure graph analytics), but it is primarily intended for use in GraphRAG applications.

But we will not use it in any such way; we will bend it for our purpose. We will open a paint can instead of tightening a screw.

Let’s do this step by step to import the homework document into a graph in the way we want.

If you don’t already have Aura, click on this link and register a free account. You also need to create an instance, but this can be an Aura Free instance (this is one of the options you will see when you click Create instance).

Now click on Document Intelligence in the toolbar and then click the Create graph model button.

Drag the PDF over to the Drag & Drop area and the PDF will appear there. Normally the next step would be to click the Generate model button, but we will not do that, instead we will define our model manually.

Click the + Node label button in the lower-right corner. Give the label the name Question and give it one property called question (string) and set that as the unique property of the node.

Now hover with the mouse on the edge of the Question node until you see a + icon next to the mouse pointer. Then drag from there so that you create a new node with a relationship from the Question. Give this new node the name Answer, with property answer as a string. Finally click on the relationship and give it the type ANSWER. Once that is done you should have three entities (two nodes and one relationship) and a green check mark on each of them.

And when you have that you can click the Run import button and select the instance you want to import to (the free instance created earlier). You may have to supply the password for this instance. You should have gotten that when the instance was created.

You will now see that the import is running. This can take several minutes, so be patient.

Switch to the Query tool in the toolbar to the left and connect to your instance. Since we aren’t really using the tool for what it was designed for, and we had no clear guidance that we wanted one answer per question, we may want to do some cleanup. Just run this query to remove questions (and corresponding answers) where there isn’t exactly one answer per question:

CYPHER 25
MATCH (q:Question)
WHERE COUNT { (q)-[:ANSWER]->() } <> 1
CALL(q) {
  MATCH (q)-[:ANSWER]->(a:Answer)
  DETACH DELETE a
}
DETACH DELETE q;

Now launch this query to view the graph created:

MATCH (a)-[b]->(c) RETURN *

You will see the graph and you will see that it contains a bunch of question-and-answer pairs. You also see that they are all connected to the “Chunk” (the part of the original document) that they belong to (we will use these later). All the chunks are connected to the source document (or documents in case you had multiple PDFs).

Homework trainer

Now that we have the questions, answers and chunks as a graph it is easy enough to fetch some random questions with Cypher and ask for the answer to the question.

But here comes the real challenge. In a vocabulary test you need to enter one word and it has to be correctly spelled, so you can just do a simple string comparison. But the questions here are much more complex and harder, and evaluating whether an answer is correct isn’t as straightforward.

Free and simple option: Aura agents

The simplest way to solve this is with another Aura tool: Agents. You will find Agents in the toolbar a couple of rows above the Document Intelligence, under Data Services.

If it is your first time using Aura Agents you will see a message that you need to enable Aura Agents to get started. Click Enable Aura Agent, and then make sure that both the Generative AI assistance and the Aura Agent check boxes are checked.

Now go back to Agents and select Create from scratch.

Give your agent a name, e.g. Homework assistant, and a description, e.g. Tests you on your homework. The important part is the prompt for the agent. I wrote it like this, which worked well:

You are a homework test assistant, and your goal is to ask questions on the kids’ homework to evaluate how well they know it. To your help you have a graph with questions and answers, and your task is to assess how well their answer matches the answer from the graph. Either classify the answer as correct or incorrect, and if it is correct grade it between 1 and 3. Give it 1 if it answers the bare minimum and 3 if it is well developed and covers all parts of the answer.
Start by fetching a question, together with a textbook answer and some more context on the subject. Give the question to the user and wait for their next prompt, which is their answer. Evaluate the answer, present the result back to the user, and then ask for the next question.

Select Internal as the access option (this is the free option).

Now we need to add the tools for our agent. The tools are what give the agent access to our graph in different ways. Scroll down to the bottom of the agent creation frame. There is already a Natural Language to Cypher Tool added, but we don’t want that so either disable or delete it. We only want one tool, and that we’ll define ourselves. Click Add tool and select Cypher Template.

In the tool dialog, give the tool a name, e.g. getQuestion, and a description that is sufficient for the agent to know how to use the tool. I gave it this description:

Fetches a random question, together with a textbook answer and some more context on the subject.

We don’t need any parameters, but use this Cypher query for the tool:

MATCH (q:Question)
WITH q ORDER BY rand() LIMIT 1
MATCH (q)-[:ANSWER]->(a:Answer)-[:__NODE_TO_CHUNK__]->(c:__Chunk__)
RETURN q.question AS Question, a.answer AS TextBookAnswer, c.text AS MoreContext

Now click Add tool, and then Create agent.

The agent is created now and we can select it in the list on the left and start to chat with it. Our agent is a reversal of how agents normally work. Normally we ask questions that the agent answers, but here we want it to ask us the questions. We gave it those instructions in the prompt, so hopefully it understands what to do. But we need to kick it off, so start by writing Start exam in the prompt.

And sure enough, it starts firing away with questions. Sometimes it asks the next question immediately, as we instructed it, but often it asks us if we want another question, so we need to add a Yes please (I always try to be polite to LLMs to prepare for the day when they take over) between each question. But other than that it works well.

Advanced option: A small quiz application

The solution with Aura agents works well, but since they currently can’t write to the graph we have some limitations. Every time it asks a question it draws a random question, so we can get the same question several times, and we don’t know when we have been asked all of them. Also, it can’t keep a total score for us. To solve this we need to make it just slightly more advanced, and we also need to get an API key for OpenAI, and that will incur some cost (very small though).

Here I introduce a new property called alreadyAsked to track what questions that have already been asked. In my routine to ask all questions I start by clearing this state:

MATCH (q:Question)
REMOVE q.alreadyAsked

Note that alreadyAsked is a global property, so this approach wouldn’t work for a multi-user application, but the intention here is for a single user.

And now I call this query and present the question returned to the user, asking them to give their reply. And I repeat this until this returns no rows:

MATCH (q:Question) WHERE q.alreadyAsked IS NULL
WITH q ORDER BY rand() LIMIT 1
SET q.alreadyAsked = TRUE
RETURN q.question AS question

When the user has given their answer I pass the question as the $question parameter, the user’s answer as $answer, and the OpenAI API key as $apiKey. And then I call this query to evaluate how accurate the answer is:

CYPHER 25
MATCH (q:Question {question: $question})-[:ANSWER]->(a:Answer)-[:__NODE_TO_CHUNK__]->(c:__Chunk__)
CALL(a, c) {
  WHEN a.answer = $answer THEN
  RETURN 3 AS score, "Spot on" AS evaluation, a.answer AS correct
  ELSE
  WITH ai.text.structuredCompletion(
    "Make an evaluation of how good this answer is to this question. " +
    "Return 0 if the answer is deemed to be incorrect. " +
    "If it is correct give it a score between 1 and 3 depending on how good it was. " +
    "Give it 1 if it answers the bare minimum and 3 if it is well developed and covers all parts the answer.\n" +
    "This is the question: " + $question + "\n" +
    "This is the answer provided (to be graded): " + $answer + "\n" +
    "This is the correct answer: " + a.answer + "\n" +
    "Here is more context on the subject: " + c.text,
    {
      type: 'object',
      properties: {
        score: {
          type: 'integer',
          description: '0 if answer is incorrect, and 1-3 for how good the answer is if it is correct.',
          minimum: 0,
          maximum: 3
        },
        evaluation: {
          type: 'string',
          description: 'An explanation of why that score was given.'
        }
      },
      required: ['score', 'evaluation'],
      additionalProperties: false
    },
    "OpenAI",
    {token: $apiKey, model: "gpt-5.2"}) AS result
    RETURN result.score AS score, result.evaluation AS evaluation, a.answer AS correct
}
RETURN score, evaluation, correct

If you want more information on ai.text.structuredCompletion, you can check the documentation and also my earlier blog.

I have written a small Java routine that runs the queries above and gives you a score at the end. You can find that here on the GitHub repo for this blog.

A test run

Here is a test run of the Java routine with the Cypher queries above and the example homework about the March Across the Belts. It yielded 40 questions, so to shorten it I have removed many of the questions here (mainly those I answered incorrectly and am too embarrassed to show).

Question 1
What happened on 31 January to Swedish troops?
They marched out onto the ice of the Little Belt
Incorrect answer!
Correct answer: Swedish troops captured Nyborg without a fight, along with official Otte Krag and several senior Danish officers.
Evaluation: The provided answer does not match what happened on 31 January in the given context. The correct event is that Swedish troops captured Nyborg without a fight and took Otte Krag and several senior Danish officers captive. Saying they “marched out onto the ice of the Little Belt” refers to a different action in the campaign and does not address the specific 31 January event asked about.

Question 2
Who was delegated the preparation of a detailed plan for the crossing?
Wrangel
Correct answer! Good job!
Score: 2
Correct answer: Carl Gustaf Wrangel was delegated the preparation of a detailed plan for the crossing.
Evaluation: The answer is correct: the preparation of the detailed plan was delegated to Carl Gustaf Wrangel. Although it only gives the surname (“Wrangel”) rather than the full name, it unambiguously identifies the correct person and matches the expected answer.

Question 3
When did Charles X Gustav hold a military conference in Kiel?
1657
Incorrect answer!
Correct answer: On 9 January 1658, Charles X Gustav held a military conference in Kiel.
Evaluation: The question asks for the specific date of the military conference in Kiel. The correct answer is 9 January 1658. The provided answer “1657” is the wrong year and does not include the date, so it is incorrect.

…

Question 40
What was the main objective of the Swedes after the assault on Frederiksodde?
To cut off the Danes
Incorrect answer!
Correct answer: The Swedes set an assault on Copenhagen as their main objective.
Evaluation: The provided answer does not match the correct objective. After the assault on Frederiksodde, the Swedish main objective was to assault Copenhagen to force Denmark to capitulate. “To cut off the Danes” is too vague and describes a different aim than the specific stated objective.

Exam over
You got 11 of 40
Your average score (1–3) of the correct answers was: 1.55

Well, that’s not a result to be proud of, but at least I got 11 correct answers out of 40. However, an average score of 1.55 probably won’t earn me an A. I guess I’ll go back to computer science…

// TODO: Replace example with a subject that doesn’t make me look like a fool before publishing this blog…

* Please consider copyright regulations before uploading PDFs to an LLM

---

Doing your homework with Neo4j & AI was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.