igraph-help
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [igraph] Graphs too large for Ram


From: Bernie Hogan
Subject: Re: [igraph] Graphs too large for Ram
Date: Mon, 19 Dec 2016 17:28:15 +0000

No! 

[Hi everyone, been ages,] 

neo4j is a stellar product that we use as a backbone to our program NetworkCanvas, but it is woefully inefficient at the scale you're describing. It's basically a database of networks as nodes and edges where edges are a series of bi-directional hashes relating nodes for easy traversal. The cypher language for querying it is neat. I wouldn't say elegant but neat. Neo4j works very nicely with high dimensional data but it is not the most efficient graph database. I might be wrong, but I was given some serious grief lately for using neo4j by some CS people here at Oxford. I told them that for me usability was more important than performance as my current networks are extremely high dimension (multiplex, longitudinal, many atributes) but not very large. They agreed but said I should look a little further afield if I'm working towards more big big data networks. 

In the case mentioned here, you might want to have a look at Zen but it doesn't seem very active: http://zen.networkdynamics.org

Also, Jure Leskovec's SNAP is also geared towards very large networks and can definitely handle the sort you're referring to https://snap.stanford.edu/data/ It's been used to analyze hundreds of millions of accounts and billions of edges on MSN among other things. I haven't used either of these packages in ages though so YMMV. 

Take care, 
Bernie 

Bernie Hogan, PhD
Research Fellow, Oxford Internet Institute
Faculty Fellow, Alan Turing Institute
University of Oxford
http://www.oii.ox.ac.uk/people/hogan 



On Mon, Dec 19, 2016 at 3:29 PM Ruchika Salwan <address@hidden> wrote:
Hey,

Thanks a lot Tamas !! will check it out for sure. You have been a great a help. :)

Best,
Ruchika

On Mon, Dec 19, 2016 at 6:41 PM, Tamas Nepusz <address@hidden> wrote:
I've heard that Neo4J is the de facto standard tool for dealing with graph databases. Never used it though.

T.

On Mon, Dec 19, 2016 at 12:32 PM, Ruchika Salwan <address@hidden> wrote:

Hi,
That's true. I have developed the basic version with Igraph. Can you tell me about any other library that I can use to implement the algorithm for massive graphs

Thanks,
Ruchika


On 15 Dec 2016 18:12, "Tamas Nepusz" <address@hidden> wrote:

I am following this research paper whose findings I have to replicate. And one of their graphs has 5million nodes and 69 million edges. That's the smallest dataset they are using.

igraph has no problems with a graph of that size on a decent machine. (Mine has 8 GB of RAM and an Erdos-Renyi random graph of that size fits easily). Larger graphs can become problematic -- but anyway, working with in-memory graphs and on-disk graphs is radically different, and igraph was designed for the former use-case, so it won't be of any help to you if your graph does not fit into RAM. The problem is that igraph makes assumptions about the cost of certain operations; for instance, it assumes that looking up the neighbors of a vertex can be done in constant time. These assumptions do not hold if the graph is on the disk because the operations get much more costly. So, in that case, you are better off either using another library that stores the graph in a database, or implement your algorithm from scratch.

T.

_______________________________________________
igraph-help mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/igraph-help


_______________________________________________
igraph-help mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/igraph-help



_______________________________________________
igraph-help mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/igraph-help


_______________________________________________
igraph-help mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/igraph-help

reply via email to

[Prev in Thread] Current Thread [Next in Thread]