Re: [igraph] Re: Re: Callback for long metric calculations.

[Chris Wj]

Tamas, thank you for that explanation. I understand the limitations of the Thread class and I was thinking about using the subprocess or pyprocessing module to have a multiprocessing solution. pyprocessing was made to look and act just like Thread, so I could potentially implement it with that. I'm wondering what kind of issues I am going to run in to if I use the processing.Process class from pyprocessing along with the igraph.set_progress_handler method (process memory copy?). Ideally, I am looking to implement a metrics calculation queue, that can report on individual calculation progress as well. 

If using subprocess, I was thinking to pickle the graph and send it to the worker process. Then, PIPE back periodically the progress and finally return the answer or a new picked graph object that has the calculation stored (will performing calculations change the internal graph structure by storing some of the calculations like a cache?).

These are mainly ideas I have brewing about optimizing making multiple calculations. I'm also looking to slap a web interface on it, which can timeout if I make it wait for calculations to finish.

On Thu, Dec 4, 2008 at 5:50 PM, Tamas Nepusz <address@hidden> wrote:

Dear Chris,

What I want to do is have my python program spawn a thread, run a calculation, and update an external resource with the percentage complete.

If the calculation you do is implemented in Python, the attached source code may provide hints for you. (This is a somewhat slower solution to calculate a graph's diameter -- the builtin Graph.diameter() is faster as it is implemented in C). Note that Python's threads are not equivalent to OS-level lightweight threads, they are "emulated" in the Python layer, and this has some consequences; e.g., if a thread enters a calculation in the C layer (as most igraph functions do), it cannot be interrupted until the execution returns to the Python interpreter. (There are some exceptions, though, but that's not important for us at the moment). Therefore, I wouldn't really use Python's threading module in this case; if there's a long-running calculation, I just simply print the progress to stdout regularly after every k calculation step.

If the calculation is an igraph built-in calculation that supports progress reporting (e.g., betweenness centrality), you can use igraph.set_progress_handler() to attach a Python function to igraph. The function must accept two arguments, the first is a textual description of what igraph is doing right now, the second is a percentage. So, for instance, if you want to monitor a long betweenness centrality calculation, do the following:

import igrpah

def my_progress_handler(msg, percentage):
 print msg, percentage

igraph.set_progress_handler(my_progress_handler)
g = igraph.Graph.Barabasi(100000)
g.betweenness()

Note that not all functions support progress reporting. If you think you found a function in igraph where no progress reporting is done but it would make sense to do so, please let us know.

--
Tamas




_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help