Re: [Bug-tar] major problem with --listed-incremental silently failing

[John R. Vanderpool]

in case anyone is interested in this thread it's been getting stuck in
my mail queue due to using a different address than i am sending from.
warning this msg is long because it has been about 10 replies back
and forth.

thank you much for hanging with me sergey!!!  test with CVS snapshot so
far looks good - still running but i did not see all the non-specified
files get tagged as "is new" during initial scan.

> John R. Vanderpool <address@hidden> wrote:
> 
> > this does not work using tar 1.14 with RHEL4, only the parent dir gets
> > backed up for /ftp/data/2 and it just silently does not backup any of
> > the other files on that fs:
> > 
> > tar -c -f /dev/dlt35 -b 256 -v --one-file-system
> > --listed-incremental=/var/tmp/bck.tmp.incremental --ignore-failed-read
> > --totals ./ /home /usr/local /ftp /vol1 /ftp/data/2 

Sergey Poznyakoff writes...
> 
> The problem is that, up to an including version 1.15.1, the operation 
> of --one-file-system depended on the device number of the current
> working directory, so that all subdirectories with the differring device
> number were omitted. Obviously this behavior is wrong. It will be fixed
> in the next release. 

thanx for the reply.  that's not what i'm seeing (i don't think at
least).  my curr working directory is / - so you're saying it shouldn't
be backing up anything other than / (it is, just skipping /ftp/data/2,
*unless* /ftp/data/2 is moved higher in the list [only tested first in
list] then it works as expected)

another person replied directly and said that --one-file-system and
--listed-incremental are being now touted as mutually exclusive (i hope
not, that is how i have always used it and need it to work).

i think maybe a lot of the confusion comes from this:

  is --one-file-system like "find"'s -xdev (aka -mount)? or as it
  appears by the language used in gtar --help ("stay in local file system
  when creating archive") that it really is more like find's -local !?
  but it does work like -xdev.  gnu tar should i imagine have both of
  these capabilities.

> John R. Vanderpool <address@hidden> wrote:
> 
> > thanx for the reply.  that's not what i'm seeing (i don't think at least).

Sergey Poznyakoff writes...

> Yes, you are right. It is another issue. Please try the following
> patch:
> 
> Index: src/incremen.c
> diff -pur tar-1.15.1/src/incremen.c tar-1.15.1-n/src/incremen.c
>                 directory->children = ALL_CHILDREN;
...

nope, that tries to backup everything now (i.e. i have / as one of the
file systems and it began traversing into file systems not listed on
the command line)
fish said:
> > > nope, that tries to backup everything now (i.e. i have / as one of the
> > > file systems and it began traversing into file systems not listed on
> > > the command line)
> > 
> > When in incremental mode tar must traverse into file systems not listed
> > on the command line, it is normal.

> Sergey Poznyakoff wrote:
>
> I managed to reproduce the situation you were talking about. Indeed, tar
> was traversing too far under the mount points, whereas normally it should
> see only the mount point itself. Here is an updated patch for version
> 1.15.1. Please inform me if it works for you.

thanx for trying to help!

this patch seg faults, here is a trace of:

~/tar -c -f /dev/dlt40 -b 256 -v --one-file-system 
--listed-incremental=/var/tmp/bck.tmp.incremental --ignore-failed-read --totals 
./ /home /usr/local /ftp /vol1 /ftp/data/2 

lstat64("/ftp/.data/002", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
write(2, "/root/tar: ", 11/root/tar: )             = 11
write(2, "/ftp/.data/002: Directory is new", 32/ftp/.data/002: Directory is 
new) = 32
write(2, "\n", 1
)                       = 1
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 11017 detached

> diff -pur tar-1.15.1/src/common.h tar-1.15.1-n/src/common.h
> --- tar-1.15.1/src/common.h   2004-12-21 16:31:58.000000000 +0200
> +++ tar-1.15.1-n/src/common.h 2005-12-13 13:02:30.000000000 +0200
...

Sergey Poznyakoff writes...
> 
> John R. Vanderpool <address@hidden> wrote:
> 
> > this patch seg faults, here is a trace of:
> 
> I should have paid more attention while converting it to 1.15.1! Here is
> the correct version.

this one won't compile (patch went in ok):

incremen.c: In function `scan_directory':
incremen.c:222: error: syntax error before "directory"

i don't see the problem that gcc is complaining about, the parens match
up, and the ! seems ok (this is gcc 3.4.4 i used):

                if (one_file_system_option && device != stat_data.st_dev
                    && !((np = name_scan (name_buffer)) && np->explicit)
line 222 --->     directory->children = NO_CHILDREN;

it is definately this one it is whining about though:
        !((np = name_scan (name_buffer))
> > i don't see the problem that gcc is complaining about, the parens match up
> 
> No, they don't:
> >                 if (one_file_system_option && device != stat_data.st_dev
> >                 && !((np = name_scan (name_buffer)) && np->explicit))
> --- This one is missing ----------------------------------------------^

ok, that compiles, thanx.  but now it is back to scanning all the
directories in every file system, once it sees that a certain level dir
is not requested shouldn't it stop descending into the dir hierarchy?

tar: 
./ftp/.data/002/tovs_l1b/2004/10_2004/TOVS1B/NOAA-15/HIRS2_0410/HIRX_041009: 
Directory is new

even if it doesn't back up these files (not supposed to since not on
list), the additional scanning will take many extra hours on this host
(millions of directories)

and its memory usage has bloated a lot:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28365 root      15   0 1228m 1.2g  612 D  1.3 60.5  11:30.21 tar
                       ^^^^^^^^^^
Sergey Poznyakoff writes...
> 
> John R. Vanderpool <address@hidden> wrote:
> 
> ok, that compiles, thanx.  

that last patch seems to have worked

> but now it is back to scanning all the
> directories in every file system once it sees that a certain level dir
> is not requested shouldn't it stop descending into the dir hierarchy?
> 
> That's right, it should. 

why does it need to do that?  once it determines that a certain level
of dir is not in the requested list it shouldn't have to keep going down
that tree.

due to this behavior, tar now traverses an extra 700,000 dirs on
this system (was 648265, now is 1380063 "dir is new" lines printed),
which takes an extra almost 5 hours to run! (real 678m47.454s vs real
966m34.065s)

Please try this snapshot:
> 
>  ftp://download.gnu.org.ua/pub/alpha/tar/tar-1.15.2-20051214.tar.gz

> > > but now it is back to scanning all the
> > > directories in every file system once it sees that a certain level dir
> > > is not requested shouldn't it stop descending into the dir hierarchy?
> > > 
> > > That's right, it should. 
> > 
> > why does it need to do that?
> 
> What I said means that "Yes, it should stop descending into the dir 
> hierarchy" :^) 

ah, that explains the confusion!

> Are you sure it does not? 

yes

> I have run similar test on my
> box and as far as I can tell, no unrequested directories has been
> recursed into. To make sure we speak about the same thing, here's what
> happens:
> 
> 1. At the beginning, tar scans the requested directory hierarchies.
> 2. When tar happens to descend into a mount point, it does not scan any sub-
> directories under it, *unless* this mount point has also been specified
> in the command line.
>
> Please verify if this is not so on your system.

with just that patch you sent it does descend

> I've forgotten to ask you: have you tried the CVS snapshot I suggested?

i will try that now