How to import Wikidata into Neo4j

Before we can import RDF data into Neo4j, we first need to create a Graph Config to instruct Neosemantics on how to store the data. For this, we’ll need to call the n10s.graphconfig.init procedure with a map of parameters. As with the Get Started Tutorial, we will use the mapping procedures in n10s to automatically rename the vocabulary terms used in the data retrieved from Wikidata when it is persisted into our Neo4j database.

Depending on the Graph Config, Neosemantics will treat multivalued properties in different ways (multivalued properties are a common feature in multilingual datasets like Wikidata, where we find multiple values for names, one for each available language: United Kingdom, Regno Unito, Royaume Uni, etc). By default, Neosemantics will assume properties are single valued and each new value for the same property on the same node will overwrite the previous value, but in some cases it may make sense to keep all the values. We can apply a configuration setting to store an array of these values in Neo4j.

As we did in the tutorial, we need to ensure that there is a unique constraint on the URI property for nodes with a label of :Resource

In order to import the data into Neo4j, we will first need write a query to retrieve a list of triples from Wikidata. We will use SPARQL CONSTRUCT and SPARQL DESCRIBE for that. Our final RDF query will return information as triples of subject, predicate and object which use terms from several vocabularies/schemas. Both unique identifiers for things and vocabulary elements may be confusing at first because they are all numeric codes, but luckily the Wikidata Query Service comes with auto-completion and we can also browse the triples on Wikidata.org.

Wikipedia (mostly for human consumption) is linked to Wikidata (structured data for programatic use). If you navigate to a Wikipedia Entry, you will also see a Wikidata item link on the left hand navigation. This can also be opened using a keyboard shortcut: Ctrl+Option+G on a Mac.

Take for example the United Kingdom wikipedia entry, this has a reference of Q145 on Wikidata. The Wikidata entry displays a human readable table of all triples stored against the item, including names, population by year and life expectancy.

We can try the programatic access using the Wikidata Query Service to return all known tripples for the element using a DESCRIBE SPARQL query.

In total there are over 800,000 triples, so we should probably be more specific about the data that we are specifically interested in.

Under the Statements header, we can see that United Kingdom is lised as an instance of (wdt:P31) a Commonwealth realm, an island nation and most importantly a sovereign state - Q3624078. We can use this triple pattern (something being an instance of a sovereign state) to identify all countries and bind them to a ?country variable.

This query shows that the first subject returned by the query, wd:Q757 has labels in many languages.

The combination of configuration settings from the n10s.graphconfig.init call earlier will ensure that:

To demonstrate the ability to import a relationship from a predicate, we can add the continent property (wdt:P30) to the query. This property in RDF links a country to its continent. Then we also add a property, the country’s population (wdt:P1082). Wikidata has certain properties mesured at different points in time. Population is one of them. For the sake of brevity, we will not explain the intricacies of Wikidata’s model but we’ll just add filters to our query so that it only returns population counts after 2010 and also restrict the languages of each label to English, Arabic, Russian and Chinese. For more details, check Wikidata’s documentation and examples.

Here is the updated query:

Then we can use the information from the WHERE clause to construct our triples ready for ingestion into Neo4j.

We’ve seen that SPARQL SELECT queries return tabular results, but we want to get RDF data instead. To instruct the Wikidata query service to return triples we can replace the SELECT section of the query with a CONSTRUCT clause. The CONSTRUCT section defines how the data retrieved in the WHERE clause should be returned. So we can use this section to rename certain terms or even restructure the information as we wish. The output of a SPARQL CONSTRUCT query is a stream of subject, predicate and object triples which together represent an RDF graph.

To preview what the data will look like in Neo4j, we can use the n10s.rdf.preview.fetch procedure. In the Getting Started guide, we used a static URI but we can query Wikidata’s APIs programatically by sending a GET request the following URL:

As the URL requires an encoded version of the query, we can use the APOC apoc.text.urlencode function to encode the SPARQL query above. The endpoint also requires that we send an Accept header with the content type that we wish to consume, in this case Turtle.

Be sure to install the APOC plugin and restart Neo4j before proceeding any further.

If we run the query using the n10s.rdf.stream.fetch procedure, we can see the list of triples along with some additional metadata like the datatype and the language tag.

The fetch method is useful when we want to preview in neo4j the triples returned by an RDF source, in this case our SPARQL query on Wikidata, but also if we want to process them with cypher instead of delegating the import to neosemsntics.

We can also use the n10s.rdf.preview.fetch procedure in Neo4j Browser to preview the data as a graph.

The query will return a set of nodes connected together with relationships as defined in the CONSTRUCT portion of the query.

Once you are happy with the preview, you can run the n10s.rdf.import.fetch procedure with the same parameters.

Removing the limit should load and parse over 32,000 triples.

The config that we have provided ensures that the countryName property for each Country node is an array of values representing the country’s name in a specific language. If we take a look at the array, each item is a string which starts containing the value, an @ symbol and then the language.

Neosemantics provides a number of helper functions for extracting information from multilingual data:

In this guide we have learned how to:

If you have experienced any issues during this tutorial you may find the solution on the Troubleshooting page.