Re: [Octave Forge] Possible addition of gradient-descent framework for optim package

[Olaf Till]

This e-mail is possibly a duplicate, sent because I think the previous
sending (a few minutes ago) did not work.

Hi Daniel,

On Fri, Oct 23, 2015 at 07:57:57AM +0200, Daniel Kraft wrote:
> Hi!
> 
> (I sent this already to Olaf some time ago, but I think it makes sense
> to give everyone a possible say in the discussion.)
> 
> I currently include functions for a gradient-descent method for shape
> optimisation in my level-set package.  (You can look at them in the
> development sources, they are not yet in a release -- look at the so_*
> functions.)
> 
> They implement a generic descent method including potentially parallel
> line search.  Problems can be specified generically by defining suitable
> callbacks.  The full descent process can also be instrumented via
> callbacks, so that you can produce plots at each step, save plots for
> selected iterations as PDF, or do whatever you want.  I use that for my
> own research.
> 
> Another highlight is the ability to save the full descent run to a log
> file and replay it later.  This way, you can, for instance, run a
> calculation over night and later investigate each iteration
> interactively, use it to produce plots or do other stuff with it
> (without the need to recalculate).  You can look at the demos for the
> functions if you wish to see it in action.
> 
> Currently, these functions depend on level-set and do only shape
> optimisation.  The points of dependence are, however, only two of three
> single lines of code.  So I think that it would be fairly easy to
> introduce another callback that breaks the coupling, so that the methods
> can be used for all kinds of optimisation problems.
> 
> Do you think that they could be included into the optim package instead
> of level-set if this is done?  I can, of course, provide the full patch.
>  Carne also offered to include them into image when I talked to him at
> OctConf, but we both agree that they do, of course, fit much better into
> optim.
> 
> Yours,
> Daniel

the following reply was made up (but not sent) one month ago, when I
suggested you to involve the mailing list with this.

Though there are also standalone optimizers in 'optim', there are also
frontends, whose interfaces were attempted to be designed as general
as possible. If possible, new algorithms should not be standalone, but
backends to the existing frontends. Your so_... functions would
introduce a different interface, which would be a complication to
'optim' which outweighs, in my opinion, the possible advantages.

As for the algorithms: Gradient descent is not what I'd think
desirable for 'optim'. Line search is e.g. implemented in core Octaves
'sqp' with a few lines of code, also Armijo with backtracking, but
additionally constraints are handled, which would be necessary for a
general algorithm employing line search. I wouldn't think a public
function is needed for line search, since it is typically used as a
component of another algorithm.

You mention the possibility of storing and replaying the course of
optimization. The existing frontends of 'optim' can call user-provided
callbacks, with which it should be possible to implement this. Can't
you interface the existing functions of 'optim' with your package by
providing suitable callbacks, instead of writing separate optimization
code? If some functionality for this should be lacking in 'optim', we
could possibly add it.

Olaf

-- 
public key id EAFE0591, e.g. on x-hkp://pool.sks-keyservers.net

signature.asc
Description: Digital signature