[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Octave Interpreter
From: |
Stefan Seefeld |
Subject: |
Re: Octave Interpreter |
Date: |
Thu, 02 Oct 2014 08:34:38 -0400 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 |
On 2014-10-01 21:16, Bipin Mathew wrote:
> Hello Manu,
>
> I was actually thinking of working on something along the same
> lines. Googling around, I see a few efforts along these lines:
>
> http://octave.sourceforge.net/parallel/
> http://octave.sourceforge.net/mpi/index.html
>
> While I have not used these personally, looking at the examples, it
> appears one needs to be quite aware of MPI / parallel programming to
> make effective use of these package. It would be nice if all these
> things could be made more transparent to the end user.
>
> Many years ago, I actually worked with a team at MIT Lincoln
> Laboratory, designing this sort of thing for Matlab:
>
> http://www.ll.mit.edu/mission/cybersec/softwaretools/pmatlab/pmatlab.html
>
> The idea was to hide as much of the parallelism as possible. This was
> done by using a special "dmatrix" class to represent your matrix,
> which was, in reality, spread across several servers. I don't know if
> this is still an active project there ( Any of my old Lincoln friends
> can comment? :-) ).
Interesting that you bring this project up, as I was considering
proposing something similar. While I haven't worked at the MIT/LL, I
have collaborated with them.
> Of course technology has moved on since then and there is probably
> alot of new technology we can leverage to make something similar that
> cleaner/faster/smarter ( i.e. more awsome ). For example, metadata for
> where the data was was stored in a static file back in days. I reckon
> we can just query a distributed file system to get this same
> information these days, along with all the benefits of replication /
> performance / resilience.
I believe Jeremy Kepner's original implementation was deliberately using
very simple means to implement the transport protocol (all file-based),
to make it easy for users to get off the ground.
In collaboration with the MIT/LL et al. I have been working on a C++ API
specification (http://portals.omg.org/hpec/content/specifications) as
well as library implementation (http://openvsip.org/) incorporating much
of the concepts of pMatlab's distributed arrays (i.e., using "maps" to
define the distribution of blocks of data across many computing nodes).
I was considering working on scripting frontends for that implementation
in Python and Octave, and I have been watching with interest the recent
work on classdef support in Octave, as I was considering using that to
define the high-level API for Maps, Blocks, and Views.
(In fact, I presented the idea to add Python and MATLAB "frontends" to
VSIPL++ backends for a seamless integration into a high-performance
computing platform at an IEEE conference two years ago:
http://ieee-hpec.org/2012/index_htm_files/Seefeld.pdf, and the ideas
were received quite enthusiastically.)
> All this being said, perhaps someone more familiar with the source
> code could comment on how easy / impossible such a thing would be
> given the current state of Octave? In terms of your direct question of
> how you can build a compiler for Octave, I believe a good place to
> start is to look at the lex/yacc grammar files:
>
> octave/libinterp/parse-tree/lex.ll
> octave/libinterp/parse-tree/oct-parse.yy
I'm honestly not entirely sure I understand the question: What
additional support in terms of language support is needed to provide a
"distributed Octave" ? What I was expecting (though I haven't
investigated much of it yet) is infrastructure work akin to
http://ipython.scipy.org/talks/0903_siamcse09_ipython_dist_bgranger.pdf,
to allow an Octave "frontend" to dispatch computation to multiple "engines".
Would any of this require changes or additions to the language /
interpreter, rather than just additional functions ?
I think a good path towards support for pMatlab in Octave would be
* infrastructure work that allows multiple "octave engines" to be
controlled from one "octave controller" via some scheduler, akin to
ipython's implementation)
* an Octave package that adds an MPI-like API on top of the above
* A new API defining distributed arrays in terms of "maps"
I'd be happy to collaborate on this, in particular as there are
discussions in the OMG HPEC Working Group (http://portals.omg.org/hpec/)
to add new language bindings for Python and MATLAB/Octave to the VSIP
specifications.
Stefan
--
...ich hab' noch einen Koffer in Berlin...
- Octave Interpreter, Manu Kaul, 2014/10/01
- Re: Octave Interpreter, Bipin Mathew, 2014/10/02
- Re: Octave Interpreter,
Stefan Seefeld <=
- Re: Octave Interpreter, Richard Crozier, 2014/10/05
- Re: Octave Interpreter, Stefan Seefeld, 2014/10/05
- Re: Octave Interpreter, c., 2014/10/06
- Re: Octave Interpreter, Stefan Seefeld, 2014/10/06
- Re: Octave Interpreter, c., 2014/10/06
- Re: Octave Interpreter, Stefan Seefeld, 2014/10/06
- Re: Octave Interpreter, c., 2014/10/06
- Re: Octave Interpreter, Bipin Mathew, 2014/10/06
- Re: Octave Interpreter, Stefan Seefeld, 2014/10/06
- Message not available
- Re: Octave Interpreter, Stefan Seefeld, 2014/10/06