Re: [igraph] iGraph functions in parallel?

[Magnus Torfason]

Chiming in on that, I would like to add that I've used snow and tried 
out snowfall with R and igraph. snowfall is actually a front-end for 
snow (which again is a front-end for MPI).

I agree that it does not take a great amount of effort to create a 
function that works on graphs in parallel. However, I also think that 
Tamas's point about having to load the graph multiple times into memory 
is a valid and important one. I've used it for working with multiple 
graphs, so it was not a direct issue for me, but each slave of a snow 
cluster is a separate process, and sharing of objects between the memory 
space does not happen automatically - you have to pass them explicitly 
to each of the slaves (and I am not aware of how to achieve that without 
duplication).

Best,
Magnus

On 9/17/2009 8:35 AM, Gábor Csárdi wrote:
> I don't know much about snowfall, but if that does not work for you,
> then the 'snow' package should, it works above MPI or even without it.
> It work on multicore or SMP machines or even on clusters. It is very
> high level, parallel diameter calculation can be programmed in 10
> minutes and 10 lines of code, I would guess.
>
> Best,
> Gabor
>
> On Tue, Sep 15, 2009 at 12:56 PM, Tamas Nepusz <address@hidden> wrote:
> > Hi Juan,
> >
> > igraph does not contain native support for openMP or any other
> > parallelization solution yet. However, an obvious way to distribute the
> > computational load of diameter calculations (if your computer has
> > multiple CPU cores) is to start independent shortest path calculations
> > from each of the nodes and aggregate the results later. For instance, if
> > you have 4 CPU cores, start four R instances, partition the set of
> > vertices into four roughly equal sets and calculate shortest paths
> > individually from each of the vertices s.t. the first R instance
> > processes paths starting from the first quarter of the vertices, the
> > second R instance processes paths starting from the second quarter and
> > so on. Unfortunately this means that you have to load your graph into
> > the memory four times and if it doesn't fit, you will have problems as
> > the OS will keep on swapping all the time.
> >
> > As for snowfall, I haven't tried it and I'm not an R expert, so I won't
> > try to comment on that part. Since most of igraph's code is in C, igraph
> > calls are sort of a black box for R: some R objects go in there and some
> > other R objects come out, but R knows nothing about what happens inside.
> > Therefore, I don't think that snowfall could magically parallelize
> > igraph's diameter() call, but it might be possible that the solution I
> > mentioned above is feasible using snowfall and a _single_ R instance
> > where snowfall will take care of distributing the individual shortest
> > path calls.
> >
> > --
> > Tamas
> >
> > On Tue, Sep 15, 2009 at 12:41:54PM +0200, Juan Manuel Tirado wrote:
> > > Hi everybody,
> > >
> > >    I'm currently working with large graphs (about 250,000 nodes and
> > > 1,500,000 edges). This size seems to be too large for igraph, and some
> > > operations such as diameter() take more than one day (I stopped the
> > > operation before it finished), the same thing with the average path length.
> > >
> > >    My question is does exist any way to execute igraph in parallel (using
> > > openMP for example)? I have found an R package called snowfall (
> > > http://www.imbi.uni-freiburg.de/parallel/) but I'm not sure if it is
> > > possible to run igraph with this package.
> > >
> > >    Regards,
> > >
> > >    Juan
> > > _______________________________________________
> > > igraph-help mailing list
> > > address@hidden
> > > http://lists.nongnu.org/mailman/listinfo/igraph-help
> >
> >
> > _______________________________________________
> > igraph-help mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/igraph-help