Re: [Gzz] ``async_storm--benja``: Supporting aynchronicity in Storm

[Benja Fallenstein]

Tuomas Lukka wrote:

> On Sun, Nov 24, 2002 at 03:29:26PM +0100, Benja Fallenstein wrote:
>  
>
> > =========================================================
> > ``async_storm--benja``: Supporting aynchronicity in Storm
> > =========================================================
> >    
>
>  
>
> > This PEG defines a pattern to support asynchronicity.
> > A ``JobListener`` is introduced that can be used
> > to inform clients when new data arrives. All Storm lookups
> > take an optional ``JobListener`` parameter.
> > If the requested data is available locally, they
> > simply return it. Otherwise, they return ``null`` and
> > start a new thread to retrieve the data. Once that job is done,
> > the ``JobListener`` is informed; it can at that time
> > re-request the data and expect it to be available.
> >    
>
> "expect it to be available": too strong. 
> We don't really want to guarantee this: e.g. 
> 	get
>         ...
>         finished...
>         ...
>         (long time passes, system is busy, disk almost full,
>          block removed in other thread to make room in cache)
>         ...
>         try to get block..
>
> We shouldn't mark all those blocks nonremovable.

Hmm. Ok. First I was a bit worried about cases where we want blocking 
access (e.g. during startup, when we're waiting for the first pointer to 
be resolved and really cannot do anything else)-- but in these cases, we 
can write:

listener = BlockingJobListener()
while(true) {
block = pool.get(id, listener);
if(block != null) break;
block.wait();
}


> > Issues
> > ======
> >
> > - Should we keep variations of the Storm methods that do not
> > take a ``JobListener``? If so, what should these do
> > if data is not available locally: Implement blocking IO,
> > spawn a lookup job that does not inform any listener
> > when it finishes, or simply return ``null``
> > and do nothing?
> >
> >  RESOLVED: I'm leaning towards not providing these methods
> >  for now, since it's not clear what they'd do.)
> >    
>
> I'd say they should do exactly what get() should do if called
> with a null Listener, and that behaviour *needs* to be defined.

NullPointerException? ;-)

> I'd give the system some leeway here: 
> 	1) if locally stored, return block
>         2) if not locally stored, return null, and three options:
>                 - do nothing
>                 - start looking for it actively
>                 - mark it as "get this later if happen by" - somewhere
>                   between the two above options.

Ok, sounds good.

> > Changes
> > =======
> >
> > I propose a similar interface for Storm, ``JobListener``::
> >
> >   /** Called when new data has arrived. */
> >   void newDataAvailable();
> >
> >   /** Called when the lookup finishes.
> >    *  @param timeout Whether the lookup was finished
> >    *                 because of a time-out. If this
> >    *                 is false, the lookup was finished
> >    *                 because the pool implementation
> >    *                 believes that all relevant data
> >    *                 has been retrieved now.
> >    */
> >   void finished(boolean timeout);
> >
> > The two separate callbacks make it easy to use the same
> > interface for lookups like for xu links, where we update
> > the view each time new information arrives, and
> > lookups like for pointer blocks, where we can only
> > continue when all relevant data is known.
> >    
>
> Do we *want* to do that? Why should the one interface be used for both?
>
> I'd prefer
>
>         abstract class BlockListener {
>                 public void finished(BlockId id) {
>                 }
>                 public void timeout(BlockId id) {
>                 }
>         }
>
>         interface DataListener {
>                 void newDataAvailable();
>         }

a) finished() is not specific to blocks: For most lookups, we want to be 
able to know when they're finished (e.g. pool.getIds()).
b) Listeners should not be abstract classes, since some people like to 
implement them as mix-ins.
c) I think timeout() and finished() will usually be implemented the same 
way, so why have two different methods?

We could have a JobListener and a JobCompletionListener, with the former 
inheriting the latter, if you prefer (because for blocks, 
newDataAvailable() won't be called).

> > The lookup methods in ``StormPool`` would simply get an additional
> > ``JobListener`` parameter::
> >
> >   Block get(BlockId id, JobListener listener);
> >    
>
> Should spec more exactly: if returns null, listener will be called later.

Ok.
-b.

>