Re: edif2 corner condition.

[Hans-Peter Sorge]

    Hi Chris,
    
    I have done some inotify tests to see what happens using different
    editors.
    
    Editing a FUNCTION in apl-ws  will 
     
    1. start edif2 and create a file FUNCTION.apl - this file create
    will directly trigger inotify messages 
    
    inotify_test/ CREATE FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ MODIFY FUNCTION.apl
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    
    based on the events one FX FUNCTION
    is being processed by edif2 - even if two monitors (CREATE and
    MODIFY) are set. 
    I did not dig into this whether edif2 receives combined CREATE,
    MODIFY events as one event.       
    
    
    2. edif2 then starts the editor "emacs FUNCTION.apl" which in turn
    keeps inotify rather busy:
    
    --- start emacs
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ ACCESS FUNCTION.apl
    inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
    inotify_test/ OPEN,ISDIR
    inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
    inotify_test/ OPEN,ISDIR
    inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
    inotify_test/ OPEN,ISDIR
    inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
    inotify_test/ OPEN,ISDIR
    inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
    inotify_test/ OPEN,ISDIR
    inotify_test/ CLOSE_NOWRITE,CLOSE,ISDIR
    
    --- when start editing
    inotify_test/ CREATE .#FUNCTION.apl      
    
    --- save from editor  
    inotify_test/ MODIFY FUNCTION.apl                 -- inotify
    messages do not guarantee a stable order of messages from events
    inotify_test/ OPEN FUNCTION.apl                    -- inotify
    messages do not guarantee a stable order of messages from events
    inotify_test/ MODIFY FUNCTION.apl                  --
    inotify messages do not guarantee a stable order of messages from
    events
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    inotify_test/ DELETE .#FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
    --- again edif2 produces only one FX FUNCTION despite 2xMODIFY
    events  
    
    
    --- leaving the editor -
    no message
    
    
    
    ----- for slickedit ---------
    --- open buffer
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ ACCESS FUNCTION.apl
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    
    --- start editing
    --- no event
    
    --- save buffer
    inotify_test/ OPEN FUNCTION.apl                     -- inotify
    messages do not guarantee a stable order of messages from events
    inotify_test/ MODIFY FUNCTION.apl                   --
    inotify messages do not guarantee a stable order of messages from
    events
    inotify_test/ MODIFY FUNCTION.apl                   --
    inotify messages do not guarantee a stable order of messages from
    events 
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ ACCESS FUNCTION.apl
    inotify_test/ OPEN FUNCTION.apl
    inotify_test/ ACCESS FUNCTION.apl
    inotify_test/ CLOSE_NOWRITE,CLOSE FUNCTION.apl
    inotify_test/ CLOSE_WRITE,CLOSE FUNCTION.apl
    --- leaving the buffer -
    no message
    
    
    * -- inotify messages do not guarantee a stable order of messages
    from events.
    I observed events in different order:
    OPEN, MODIFY, MODIFY or
    MODIFY, OPEN, MODIFY or even
    OPEN, MODIFY, MODIFY, MODIFY.
    More diverse message are being sent by different editors.
    
    
    btw. I have setup emacs config to not create a backup file. 
        Handling of backup FUNCTION.apl~ would be even more verbose.
    
    Coming back to where things go wrong:
    
    in APL WS edit function:
     - 'EDITOR'  edif2 'FUNCTION' 
    
     edif2 starts processing:
     -  A create a file from function
        (- write  file content from function)  
     -   inotify gets triggered - MODIFY 
     -  B read file function
        (- write file content from function)
     -  fx function
    
     - open external EDITOR FUNCTION.apl
       in EDITOR do nothing
       then close EDITOR  
    
    The race is on between A and B. 
    
    If the file content got written before B all is fine.
    If not, the file read happens during the write, by that truncating
    the FUNCTION.
    
    Because of the verbosity of inotify and the editors handling of
    files 
    it seems to be tricky to rely on a specific inotify message or even
    rely 
    on some specific sequence of inotify messages for a consistent file
    processing.  
    
      
    Some possible solutions:
        1- Serialize the events:  stop inotify, file write, file edit,
    start inotify  
          The first FX caused by FUNCTION-write-file/inotify-MODIFIED
    could be prevented  
             However - if MODIFIED creates a message, it seems
        (my feeling is), that MODIFIED does not guaranty, that
    the full FUNCTION content got written by that time.
             It could be that some things are going on at
    the discretion of the editor ( or even the OS) to run unrelated
    processes in any order.  
    
        2- Set a timer to .... ? NO.. NEVER EVER:-)  
      
        3- Make file_name unique. e.g. by adding a full timestamp
    (yyyymmddhhmmssmmm.FUNCTION.apl). Or create a unique path for
    file_name.
          This could even  open an way for experiments - 
            open the same function twice for editing.
               make changes  to instance A of FUNCTION - test in APL
               make changes  to instance B of FUNCTION - test in APL
              decide which is better ...
           I am still not sure, that the race condition could be
    resolved by adding a timestamp.  
    
        4- Add some "intelligent" message filter.
           Lots of work with little hope for success.
    
        5- As edif2 tries to sync two unrelated processes (APL, EDITOR)
    based on events outgoing from either APL or EDITOR at any point in
    time
           there needs to be a an arbiter which has the control. 
           APL -> EDIF2  -- I want to edit FUNCTION
           EDIF2          -- OK I'll set up the environment
                         -- start EDITOR
          EDIF2          -- pick FUNCTION from APL
                      -- create file for editing    
                       -- let EDITOR pickup the file
           But then from here on the same a sync inotify problems
    remain.         
                     
        6- Let the editor "directly" talk to EDIF2.
          Create a save macro that generates a file FUNCTION.svd
           Editor-save writes file FUNCTION.svd
           EDIF2 reacts to inotify FUNCTION.svd only.
           And again - inotify might even fire MODIFIED as soon as the
    file name got established 
                    and the content was not written in total.  
             
        
    vi is even more active on all sorts of backup files, attribute
    changes.
    nano, to the contrary, has the least amount inotify events.
    
    Having gone through 1..6 and the inotify events from different
    editors, finally the following state handling is likely to fix the
    situation:
    
    while receive inotify events { 
        if   (MODIFY FUNCTION.apl) {
             while receive inotify events {
                  if (CLOSE_WRITE,CLOSE FUNCTION.apl  ) {
                    read  file FUNCTION.apl
                    FX FUNCTION
                    break; 
                    }
                 if (CLOSE_NOWRITE,CLOSE FUNCTION.apl)
                    break
                 if (ANY_OTHER_INOTIFY_EVENT) continue  // ignore
                   } 
                  
               }
    
    With all four editors the inotify sequence    
    MODIFY,  ..., CLOSE_WRITE,CLOSE   
    identifies a save file.
    ... denotes inotify events depending on EDITOR's behavior.  
     
    The inotify events CLOSE_NOWRITE,CLOSE so far seems to be a save
    event to escape the loop. 
    THE events happen too when the editors get closed w/o
    editing/saving.
    
    
    Finally:
    
    The tricky part. There is one state loop for possibly more than one
    file being edited.
    
    soooo - Item 5 comes into play again.
    Currently edif2 is state less.
    If the state analyzer is being implemented one might need to
    implement a file status state too.
    
    
    // some very rough ideas .....
    // warning - untested code ...
    // more warning not 100% C++ 
    
    class inotifyFileStateVector( ){
         // initialize for each instance  
         bool modify=false; 
         bool clnwc = false;
          string FN = fn;
         public: 
          inotifyFileStateVector( fn); 
         ~ inotifyFileStateVector( );
         }    
    
    bool 
    inotifyFileStateVector::checkFN(event){
         int eventT = event.type()
          string fn  = evetn.fileName() 
    
         if ( FN != fn ) return (false)
    
          if (eventT == MODIFY) modify=true;
          if ((eventT ==CLOSE_NOWRITE || CLOSE) && modify )
    clnwc=true;
    
          if ( modify &&  clnwc ) {
           modify=false; 
           clnwc = false;
            return(true)
         } 
        return (false); 
    }             
           
    -------------------------------
    
    inotifyFileStateVector  iFSV
    
    // keep instances of function states in a vector
    ...
    // check - only one entry for each FUNCTION_name allowed.
    bool exists=false;
    for (iFSV) {
        if (  FUNCTION_name == iFSV.FN) exists=true; 
        }
    
    if (! exists){
       ......
       // append new instance  
       iFSV.append(new iFSV(FUNCTION_name))
       ...
       } else {
        // ignore new edit request
         // maybe return message to user "Being edited."
        // maybe bring editor window to foreground.
        }  
    ....... 
    
    // check each message against the vector of function states
    while receive inotify events { 
        for (ifsv in iFSV) {
          bool aSaveOccured =  ifsv.checkFN(inotify_event);
           if ( aSaveOccured ) {
            .....
             FX ifsv.FN
            .....  
            }
           }
        } 
    
    ---------------------------------------
           
    Best Regards
    Hans-Peter

Am 30.10.20 um 16:50 schrieb Chris Moller:

> Looks
>       like it might be a concurrency problem--edif2 forks, IIRC, twice,
>       serially, and I never was completely sure that I had the
>       waitpid()s right.  I'll take a look.
>       
>       
>       Chris
>       
>       
>       On 10/30/20 11:22 AM, Hans-Peter Sorge wrote:
>       
> > Hi,
> >         
> >         
> >         working with an external editor under some circumstances some
> >         function content gets destroyed:
> >         
> >         
> >               )wsid
> >         
> >         IS CLEAR WS
> >         
> >         
> >               'libedif2.so' ⎕FX 'edif2'
> >         
> >         edif2
> >         
> >         
> >               'emacs' edif2 'FN1'   - adding two lines, save and exit
> >         
> >               'emacs' edif2 'FN1'   - body OK
> >         
> >               'emacs' edif2 'FN2'   - no entry, just exit
> >         
> >               'emacs' edif2 'FN1'   - last line got lost
> >         
> >         
> >               'emacs' edif2 'FN3'   - no entry, just exit
> >         
> >               'emacs' edif2 'FN1'   - just the first line is left over.
> >         
> >         
> >               .. and the user interface (KDE) is unresponsive for a
> >         moment
> >         
> >         
> >         Best Regards
> >         
> >         Hans-Peter
> >         
> >         
> >       
>       
>       
>