Hi Arthur,
You might find this bit of exploratory code useful:
http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54. It is an example of spawning off dozens of sub-processes and using nanomsg to communicate the data back. We needed to find changed files in gigs of data where the originating process would not be able to directly see the files (the program will be setuid). With this proof of concept code on a 32 processor machine we saw a task that would take hours drop to minutes. No surprise there but it was nice that it worked. It might be handy if this idea was abstracted into an egg with special versions of map, for-each etc. but I don't have time to attempt that now.
Nanomsg is pretty neat but it has quirks. To work reliably all open and closing of ports had to be protected with mutexes which seems very odd. Note: the code was only a proof of concept trial and it probably won't work for you out of the box. It is nice that with nanomsg you simply change the connection URL to switch from in process communication to inter-process to across hosts. We saw no difference between IPC and tcp but we were only handing back and forth tiny amounts of data so that is to be expected.
Matt
-=-