Re: Octave Interpreter

[Bipin Mathew]

Hey Guys,

   I have a few questions about MPI and peoples ideas of the scope of this project more generally.

1. ) Is it possible, using MPI, to spawn processes on specific servers? The tutorials I have seen on line appear to indicate that you can specify the number of processes to spawn but not precisely where they will be spawned ( aside from the mpd.hosts file ), can this be set programatically?

2.) How is MPI's error handling and failure resilience? If a node fails mid-computation what happens? Or even pre-computation, is there an online way of knowing which nodes are available, or will MPI just not even attempt to launch on servers it perceives as down?

3.) Are we expecting to support persistence of distributed objects? This is important since "Big Data" is seldom ephemeral. People want to load their tables / data-cubes once. Of course we should also support a mechanism for temporary distributed objects for constructs like ifft(fft(X)) and for ad-hoc analysis / prototyping. Therefore I vote yes, that we should support persistent distributed objects, but just wanted to get other peoples views. This also motivates my thought that slave processes should launch as close to the data as possible a la. Hadoop. 

4.) What are peoples opinions on other transport layer technologies like Google's protocol buffers or Thrift?

Regards,

Bipin

On Mon, Oct 6, 2014 at 9:03 AM, c. <address@hidden> wrote:

On 6 Oct 2014, at 14:39, Stefan Seefeld <address@hidden> wrote:

> That might be a good starting point, though I think ultimately we need
> something more expressive and robust. I'm notably thinking of an
> architecture for managing multiple "engine" processes that run octave
> instances but that read their input not from an interactive tty, but
> some socket. This would allow multiple engines to be orchestrated via an
> MPI Communicator (doing data-parallel computation), or using some other
> paradigm such as task-level parallelism.

I'm not really sure I understand what you are talking about.

I'd just recommend, before you start implement anything new, you
look at options that are already available for parallel computing
and interprocess communication in Octave and Octave Forge, for example:

 http://octave.sourceforge.net/parallel/
 http://octave.sourceforge.net/sockets/
 http://octave.sourceforge.net/mpi

and if those packages do not have the features you are looking for,
you start by implementing those missing features that you personally need.

> I think the ipython project has very useful material for this:
> http://ipython.org/ipython-doc/2/parallel/parallel_intro.html

looks interesting but I'm not really sure how I would use that myself.
Personally I'd rather stick to standard APIs for parallel algorithms like OpenMP
and MPI if possible.

>    Stefan
c.