Exploring LinkedIn in Neo4j

Ever since I have been working for Neo, we have been trying to give our audience as many powerful examples of places where graph databases could really shine. And one of the obvious places has always been: social networks. That’s why I’ve written a post about facebook, and why many other graphistas facebook and others to explain what could be done. But while Facebook is probably the best-known social network, the one I use professionally the most is: LinkedIn. Some call it the creepiest network, but the fact of the matter is that professional network is, and has always been, a very useful way to get and stay in contact with other people from other organisations. And guess what: they do some fantastic stuff with their own, custom-developed graphs. One of these things isInMaps – a fantastic visualisation and colour coded analysis of your professional network. That’s where this blogpost got its inspiration from. _3zvr0ySd6DRbl-qIu6wdPVJ-e0nsazuO6pdZR-DDAvh2blydmIfA3OxOopTaqnUqpe9wA27hKt7E650y7HSKWvFrxE35puYYmhcTakOrajTKBVt7K9iPoudHg The thing is: the InMap above is a “static” picture of your network. You can’t really *do* anything with it. You can’t browse through it. You can’t query it. So there began my quest for a way to get the data out of the InMap, and into Neo4j. Something that I expected to take days or weeks – but from the first google search to publishing this post was literally just 2-3 hours of work. It’s dead easy. Well I should qualify that. You of course have to have somepleace to start – a place that can help you get the data from LinkedIn, into a format that I could work with to import into neo4j. So after 5 minutes of googling, I came across this Dataiku blog post by Thomas Cabrol that had written a couple of simple python scripts to get me going.

Step 1: access linkedin API

Thomas’ scripts run against the LinkedIn developer API. This API requires you to authenticate, and therefore you actually need to register an application (in this case: the python scripts) in your configuration. Easy to do: just go to LinkedIn’s developer site and just register an app. ToQ-Uy9wAtWkvJnVjuXVTuzbFh_6OjB_337inGSY_ahsIsZbyUMuYqNieyRW4vczQD8uIgKi1z5Fye_KZEeNSM0ymeK6XUaDMoYvyV0GZeYEKIpGdosxKbUf2g The important thing here is that this process will generate a number of OAuth keys, tokens and secrets that you will need to insert into your scripts. Easy peasy – but you need to do so before proceeding. RmpghnXrCHAOcwK3_BAh1gQfwnS6lwOhXpn8cDevg8d5XNYEmBv5W8ME85gP-fV9jkbQ8WF-86eWXnvoI935LDKIkUUzTrMJ8kBGTN5zD6oKGOnYS7TuRWx6wA Thomas’ work was based on 3 scripts (one to get the OAuth credentials, one to extract the data from LinkedIn, and one to clean up some duplicates), but for the purpose of this blog post (and because of some changes in LinkedIn’s OAuth policies) you really only need 1 script. I have therefore had to fork Thomas’ script and have put my version over here. Note that you will need to insert your own credentials and name for this script to make sense. ProTip: make sure that you have installed oauth2 and urlparse, the imported modules – otherwise the python script won’t work. Once you have the script, just open up a Terminal, run the python script (python linkedin-query.py) – and then wait a couple of minutes. If all goes well, it will generate a linked.csv file that holds all the connections (name pairs) that you need – in your 1st degree network. That means:
  • connections from yourself to people in your network
  • connections from people in your network, to other people in your network.
Exactly what InMaps shows. In my particular case, that meant 1516 nodes (I admit, I am a proficient user of LinkedIn ;)), and 5345 connections/relationships. Important note: LinkedIn actually limits the amount of API calls that you can make to their servers. You can read more about it on their developer pages. For my network, it meant that I could only run the python script once per day. So beware!

Step 2 is the easy part: importing the .csv into Neo4j

Now all we need to do is get the CSV file into neo4j – and as some of you probably know, there’s more than one way to do that. Whichever way you choose, you would always need to have a simple data-model, and in this case, it could not be simpler: MR-vbGgs3TPqGJKKfJD4ajyyAAFYXUGf4fQ_JRhHcy8ECItOmvEUyvbN9arjMFbhCm0hUs0cBZR6b7ZlLzXZNvrtb9PN2jqx05RoqBjgvzTST74wq27EBC3azA Because this dataset is still quite small, I chose the spreadsheet way to import the data. All I had to do was import the csv into a spreadsheet, dedupe some of the nodes, and create the cypher statements to inject the data into Neo4j. BUHi0C6ZrOzeWULVWKlMEUL4lrE1wDvyDJg3et60IEus0lEgQ1xPk4uAXQhkHfwhPoHn_qJp7JaqQaLcrkMsfwktfWCJNRxsoBLttKTWhVFwF8PBp_fawNFdRA Make sure to configure the Neo4j auto-index for the name property for this to work.(Set  node_auto_indexing=trueand  in conf/neo4j.properties. The resulting statements look like this: create ({name:’Rik Van Bruggen’}); To create the nodes. And: start         n=node:node_auto_index(name=’Tareq Abedrabbo’),         m=node:node_auto_index(name=’Yuri Bukhan’)  create unique n-[:CONNECTED_TO]->m; to create the relationships. Again: easy. Start the Neo4j server. Take the spreadsheet, copy/paste the cypher queries into the console that connected to the server (bin/neo4j-shell) and you are all set. If you want to skip all that you could also just download the imported neo4j graph database and take a look at my professional network – but to be honest it’s a lot more fun if you do it with your own network. You can then just browse to the webadmin console, and there we have our interactive InMap equivalent. No color codes (yet) – but plenty of interactivity to go around. X-soPxXxoS-S93QPlClCAqN47GlHLGPvyrXVEthGRPlrEI6wKYkjrq81jRwxWiRnRn4WgTRHdcRSXRaFrTuTesWQaaUIQvoK2alow-suCKdbKL8Wf-sFjExAsQ

Step 3: Querying the Interactive InMap

The nice thing of having this dataset into Neo4j now, is of course that we can interact with it. I personally found it very nice and cool – as is almost always the case with graphs – to “take a walk on the data”. Just grab a node in the webadmin, select some of its connections, and just interactively browse from node to node – and find out stuff about your LinkedIn network that you may not have known before. And then of course, you can also “programmatically” interact with the data – and Cypher is the prime tool for that. Here’s a couple of queries that I made up – but I am sure that you can make up some more. If you’re into it, you can also put your own interactive visualization in front of the Neo4j graph database, something like Max de Marzi’s Neovigator would be interesting. If you have any questions regarding use-cases for Neo4j or how to use Neo4j in your project, don’t hesitate to contact me. Want to learn more about graph databases? Click below to get your free copy of O’Reilly’s Graph Databases ebook and discover how to use graph technologies for your application today. Download My Ebook



Robert Alexander says:

Dear Rik,
I tried your script and it seems to be working but there’s something troubling me.
Here is an example:
In the file produced by your script I can grep my friend Giorgio 37 times of which of 27 as the second name and 10 as the first name in the pairs.
In my linkedin page on Giorgio I see he has 195 connections of which 69 are SHARED.
What is the cause of the discrepancy between 69 and 37 ?
Just by manually examining the data for example I see that we have a common friend Mauro which does show up in the records produced by your script but I see only 10 records with him and NOT one showing both Giorgio and Mauro having a connection.
Any ideas ?
Might I be simply running onto the API limit ?
Thanks so much in advance for helping.

Hey Robert

I do think that’s a typical API problem. Have not tried to replay the script, but I have seen stuff like that happen with API problems in the past.

Good luck 🙂


Robert Alexander says:

Upon analysis of the csv file I have ascertained that only exactly <= 10 shared contacts are retrieved, which applies in both the example I gave in my previous post and all other 700+ contacts I have.

Do you happen to know if this is a limitation of the API you are using or if it's related to any other issue I am unaware of ?


Adrian Churchill says:

LinkedIn data extractor is very helpful for me to connect with different people. I really like it. Wonderful technique of data extraction from LinkedIn. Don’t have words to say about this data extractor tool.
For More : http://www.youtube.com/watch?v=P6iLn6UqiCU

[…] a tree structure. The order in which code is executed in the browser is based upon a stack.  The connections you make with people on Linked-in or Facebook are based on graphs. Data structures penetrate every […]

1 Trackback

Leave a Reply

Your email address will not be published. Required fields are marked *


Upcoming Event


From the CEO

Emil's Blog

Have a Graph Question?

Contact Us

Share your Graph Story?

Email us: content@neotechnology.com

Popular Graph Topics