MicroPython and Neo4j

Stuart Green, Consulting Engineer at Neo4j Jan 16 4 mins read

Many microcontrollers (ESP8266, ESP32, and RP2, for example) are capable of running a bare metal optimized implementation of Python 3 (MicroPython). This includes a reduced subset of the standard Python library.

These small embedded systems frequently include WiFi hardware for external network connectivity (IoT applications). This article demonstrates that with the new Neo4j Query API, a microcontroller can be a client to a Neo4j graph database server (cloud-based Neo4j AuraDB or local self-managed).

Hardware and Software

In this example, the microcontroller used is a Raspberry Pi Pico W with MicroPython v1.24.1 installed — get the uf2 file.

Due to the limitations of MicroPython and the memory restrictions of the hardware (264KB RAM on Pico W), we are unable to use the Neo4j Python Driver and are dependent on HTTP(S) connectivity for client connection to the Neo4j database server.

Network Connectivity

Establish a local WiFi connection for the microcontroller (substitute appropriate values for Your_WiFi_SSID and Your_WiFi_Password):

import network

ssid = 'Your_WiFi_SSID'
password = 'Your_WiFi_Password'

wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.config(pm = 0xa11140)  # Disable power-save mode
wlan.connect(ssid, password)

max_wait = 10
while max_wait > 0:
  if wlan.status() < 0 or wlan.status() >= 3:
    break
  max_wait -= 1
  print('waiting for connection...')
  time.sleep(1)

if wlan.status() != 3:
  raise RuntimeError('network connection failed')
else:
  print('connected')
  print(wlan.ifconfig())

We assume the network has a default route to connect to the Neo4j database server.

Requests Library

In order to interact with the Neo4j Query API, the microcontroller needs to make HTTP(S) requests. In MicroPython, this is implemented via the urequests module (a subset of the popular Python requests package). However, upon testing, this was found to be unsuitable as it did not support the HTTP/1.1 protocol required by the Neo4j Query API. An alternative was found in mrequests (the mrequests.py file is simply copied onto the microcontroller file system).

Call Neo4j Query API

Once installed, mrequests can be used to call the Neo4j Query API:

import mrequests      
import json

url = f"https://NEO4J_SERVER_HOSTNAME/db/NEO4J_DATABASE_NAME/query/v2"

headers = {b'Content-Type': b'application/json'}
auth = (NEO4J_USERNAME, NEO4J_PASSWORD)
json_query = json.dumps({'statement': CYPHER_QUERY})

The URL of the Neo4j Query API contains two components that define the NEO4J_SERVER_HOSTNAME and NEO4J_DATABASE_NAME. The authentication tuple, auth, is formed from NEO4J_USERNAME and NEO4J_PASSWORD. These capitalized values need to be substituted with appropriate values from your Neo4j DB setup (check out Querying Your Neo4j Aura Database Via HTTPS (Again) for details about creating a free Neo4j Aura instance).

CYPHER_QUERY defines the graph database query and is converted to JSON as json_query.

Perform a query and collect the response, r:

r = mrequests.post(url, data=json_query, auth=auth, headers=headers)

Transform the JSON response to a Python dictionary:

data = r.json()

Query Procedure

The snippets can be combined to create a simple Cypher query function that returns the results as a dictionary:

def neo4j_query(cypher, db):
    url = f"https://{db['server']}/db/{db['database']}/query/v2"
    headers = {b'Content-Type': b'application/json'}
    r = mrequests.post(url,
                       data=json.dumps({'statement': cypher}),
                       auth=(db['username'], db['password']),
                       headers=headers)
    return r.json()['data']

We create a dictionary (neo4j) to hold the Neo4j settings and use this with the Cypher query string (CYPHER_QUERY) to call the neo4j_query function:

neo4j = {'server': NEO4J_SERVER_HOSTNAME,
         'database': NEO4J_DATABASE_NAME,
         'username': NEO4J_USERNAME,
         'password': NEO4J_PASSWORD}

data = neo4j_query(CYPHER_QUERY, neo4j)

The returned dictionary (data) contains two keys:

values: a list of lists (rows) with the query results
fields: a list of column names (headers)

Considerations

Please note that the results of the Neo4j query could easily consume all the memory of the microcontroller, and the query should be restricted to only return primitive types (e.g., node or relationship properties). There is also a lack of error handling for Cypher query response or with the HTTPS connection.

Example

We can use the Neo4j demo server and run a query against the FinCEN dataset. The query returns the top 10 highest transaction volumes:

neo4j_settings = {'server': "demo.neo4jlabs.com:7473",
                  'database': "fincen",
                  'username': "fincen",
                  'password': "fincen"}

cypher = """
MATCH (e:Entity)--(f:Filing)
WITH e, round(sum(f.amount)) as total
WITH e, total ORDER BY total DESC LIMIT 10
OPTIONAL MATCH (e)-[:COUNTRY]-(c:Country)
RETURN e.name, c.name, total
"""

data = neo4j_query(cypher, neo4j_settings)

print("Fields: ", data['fields'])
print("Values: ", data['values'])

The resulting output:

Fields:  ['e.name', 'c.name', 'total']
Values:  [['The Bank of New York Mellon Corp.', None, 2.363868e+10], ['Credit Suisse AG', 'Switzerland', 1.124013e+10], ['Rosbank', 'Russia', 1.075286e+10], ['Deutsche Bank AG', 'Hong Kong', 7.852493e+09], ['Deutsche Bank AG', 'Czech Republic', 7.852493e+09], ['Deutsche Bank AG', 'Germany', 7.852493e+09], ['Deutsche Bank AG', 'United Kingdom', 7.852493e+09], ['Deutsche Bank AG', 'United States', 7.852493e+09], ['Deutsche Bank AG', 'India', 7.852493e+09], ['Deutsche Bank AG', 'Netherlands', 7.852493e+09], ['Deutsche Bank AG', 'Singapore', 7.852493e+09], ['Sberbank of Russia', 'Russia', 5.084848e+09], ['Ojsc Jscb International Financial Club', 'Russia', 3.98219e+09], ['DBS Bank Ltd', 'Singapore', 3.789794e+09], ['ING Netherland NV', 'Netherlands', 3.724419e+09], ['VTB Bank', 'Russia', 3.504217e+09], ['UniCredit Bank, Cjsc', 'Russia', 3.442193e+09]]

Summary

We’ve demonstrated how a microcontroller running MicroPython can act as a client to a Neo4j database server via the Neo4j Query API. This enables access to graph database technology directly down to the embedded device/IoT level.

There are other features of the Query API that could also be useful, such as query parameterization, explicit transactions, user impersonation, and token-based authentication). You can learn more in the Query API documentation. You can also learn how to Use Neo4j Query API With R on Aura and about Moving to the Neo4j Query API.

MicroPython and Neo4j was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

cyphergraph-databasemicrocontrollersmicropythonNeo4j