Re: [Chicken-users] Parallel procedures in CHICKEN

[Arthur Maciel]

Hi Matt! Thank you very much for the code! If I have any success I'll report here. Have a Happy New Year! 

Cheers, 

Arthur 

Em 28 de dez de 2016 15:29, "Matt Welland" <address@hidden> escreveu:

Hi Arthur,

You might find this bit of exploratory code useful: http://www.kiatoa.com/cgi-bin/fossils/megatest/artifact/50100144d4ed2b54. It is an example of spawning off dozens of sub-processes and using nanomsg to communicate the data back. We needed to find changed files in gigs of data where the originating process would not be able to directly see the files (the program will be setuid). With this proof of concept code on a 32 processor machine we saw a task that would take hours drop to minutes. No surprise there but it was nice that it worked. It might be handy if this idea was abstracted into an egg with special versions of map, for-each etc. but I don't have time to attempt that now.

Nanomsg is pretty neat but it has quirks. To work reliably all open and closing of ports had to be protected with mutexes which seems very odd. Note: the code was only a proof of concept trial and it probably won't work for you out of the box. It is nice that with nanomsg you simply change the connection URL to switch from in process communication to inter-process to across hosts. We saw no difference between IPC and tcp but we were only handing back and forth tiny amounts of data so that is to be expected.

Matt
-=-

On Wed, Dec 28, 2016 at 8:46 AM, Arthur Maciel <address@hidden> wrote:

Hi Kooda!

Em 24 de dez de 2016 07:00, "Kooda" <address@hidden> escreveu:

On Sat, 24 Dec 2016 02:11:37 -0200
Arthur Maciel <address@hidden> wrote:
> Is there a way to implement map, for-each and other procedures in a
> parallel way so
>
> (use srfi-1)
> (map (lambda (x) (+ x 1)) (iota 1000000)
>
> would automatically split the list into smaller lists according to the
> number of CPU cores and then gather the results back?

I guess you could spawn a process pool and send these processes a thunk
that calculates their part and send back the result. You could use s11n
egg for that, I believe.

I’m not sure it would be faster than the regular functions though.

Do you recommend any specific way to create the pool and especially to  communicate between the processes? 

About the speed, I'll  test and report the results.

Thanks! 

Arthur 

_______________________________________________
Chicken-users mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/chicken-users