[open-cobol-list] Re: Index Files.

[Roger While]

The intention is good.
The implementation is not.
I have played around with sync'ing before (both for IS
and normal files)

So, some comments.
Let's start with a application program that I have that loads
ca. 750,000 records.
This normally takes a minute or so on one of my big boxen.
With sync'ing, I terminated the run after 2 hours.
It's obvious that we don't want to do this unconditionally.

Also from Berkeley DB -
http://www.sleepycat.com/docs/gsg/C/usingDbt.html#datapersist

Other Cobol's (using standard file handler facilities) also
do not sync I/O (MF, ACU). Note, however, that
"extended file handler" facilities may allow this.


Lets be very clear here that, without file transaction facilities,
there is NO guarentee that file contents are correct if the
run-unit terminates abnormally eg. System crash, program crash.

Following mainly relates to IS I/O (but not exclusively).

What is a concern is when a program does NOT "CLOSE" a file.
(But otherwise terminates normally over a "STOP RUN").
In this case, what we can do is :
(Note 1)Maintain an open file list in libcob/fileio.c
In cob_stop_run, call an I/O exit function that checks the
open file list for non-closed files. If detected, close the file and produce
a warning on stderr.
This is OK for non-transaction file I/O.

This is NOT OK when we eventually get transaction file I/O.
If a program does NOT close the file, it would be assumed that the
program wants a rollback.

Some notes on sync'ing.

System:
Start at low-level -
Disks these days (and for SCSI has been standard for many years) have 
physically
their own cache.
Data Point 1 ) - Are your disks caching data ?
Data Point 2 ) - What are the controller settings (BIOS) ?
                       Controllers, specifically cache/raid, have their 
own cacheing/settings for DP1.
                       Also, Cache policies might be involved (write-back, 
write-through)
Get to Kernel
Data Point 3) - What are the driver settings ?
                       Are we write-(though|back)?
                       eg. megaraid (Perc xx) assumes write-through)

Get to file system
Data Point 4) - What are the mount options for the file system ?
                       (Assumptions made here that affect recovery)

Now, because of the Data Points above, it maybe that whatever you do, you
are going to land up with something you did not expect when a system crash 
occurrred.

Program :
We could, theorectically, set a trap signal on
SIGINT, SIGHUP, SIGQUIT and implement Note 1.


----

A fflush on an opened file pointer (and likewise a DB->sync) will not
have expicitely the desired effect.

I will look at implementing Note 1 above until as such time that
we implement transaction file processing.



> There is a bug in the fileio.c program.  The data does not get synced
> after every write, rewrite, or delete.  It gets left in cashe and  If
> the program terminates abnormally  or does not close the file before
> exiting all or some file writes, rewrites, and deletes can be lost.  I
> fixed the fileio.c program on my system to sync after any write,
> rewrite, or delete.  If anyone is interested in this code let me know.
> This update needs to be in the standard release of this product if you
> desire stable indexed file I-O.