Re: [Quilt-dev] Another shell re-write of backup-files.

[Jean Delvare]

Hi Kaz,

On Sunday 20 March 2011 07:43:45 pm Kaz Kylheku wrote:
> On Sat, 19 Mar 2011 13:55:05 +0100, Jean Delvare <address@hidden>
> 
> wrote:
> > And the fact that the linked
> > copies have their timestamp updated as I work is not a problem at
> > all in practice.
> 
> Hi Jan,
> 
> But if a widely-included header file is touched in every
> hard-linked tree, you could be looking at long incremental rebuilds.

This is correct. But then again it isn't a big problem in practice, at 
least for me / us (Suse). For one thing, the reference tree (from which 
others are hard-linked) is typically never used for builds, it's only a 
reference to spawn working trees. For another, the touch only impacts 
other trees when you pop down to the first patch in the series which 
modifies a given file, in several related trees at once. This seems 
rather unlikely to happen. Lastly, my typical work session rarely 
implies building different trees at the exact same time. In fact, I have 
never noticed any unexpected rebuild so far. To be fair, I did not even 
realize it could happen until you pointed it out.

Anyway, we don't have any other choice here. Not touching the files 
after "quilt pop" would lead to incorrect partial rebuilds.

You may argue that using hard-linked trees is a bad idea because of the 
above theoretical issue. Well, if you don't like it, just don't do it, 
and let people who like it, do it.

> The reason it works this way is obviously for speed.
> Quilt restores files using "backup-files -r -t".
> Files are restored by hard linking followed by
> a touch. (In my script I chose to implement -r -t
> as an inefficient copy; a decision I will revert
> for the sake of compatibility and speed.)

For completeness, please note that "hard linking followed by a touch" is 
really a trick for "move followed by a touch" here, when we don't have 
to keep the backup. The "-r -k" combination has a lot in common with "-
r" from a technical perspective, but semantically they are completely 
different. In practice, quilt never "-r -k" to the original working 
tree; it's always to a separate, temporary workplace. You really 
shouldn't think of "-r -k" as "restore and keep the backup" as it is 
advertised, but as "duplicate the backup". It might as well have been 
made a separate action IMHO, for clarity.

> Link + touch means that a "quilt pop -a" never has
> to copy any files (except maybe in some corner cases),
> making it very fast.

Correct.

> I do not believe that it's due to any intent
> to preserve links. That is a naive interpretation
> of what the program does, because such a requirement
> would be totally nonsensical. Rather, it's all about
> speed.

And I don't share your analysis, sorry. The link preservation is by 
intent here, not by accident. The code could be simplified if it were 
not the case. And, as I explained before, the use case for hard-linked 
trees is in heavy use at the company were quilt was developed 
originally. This can't be a coincidence, can it?

> "quilt pop" performs a just-in-case link breaking
> pass in the case when not all the patches are popped.
> This evidently covers some corner cases when it is
> suspected that there may be a link between the backup
> directory at the current patch level, and some
> of the working files.

This is so "evident" that your explanation is wrong ;) It's all about 
the original files, not their backups. Their backups may well have hard 
links, we even preserved them on purpose originally.

Honestly, I have no idea what case is supposed to be handled by the 
just-in-case link breaking (other than "user shot him/herself in the 
foot on purpose".) It looks totally paranoid to me, and I'd get rid of 
it. But I can only imagine that I'm missing a case were it would 
actually have saved someone's work. This decision if beyond the scope of 
the C-to-bash conversion anyway.

> When all patches are popped,
> there is no backup directory, so this suspicion doesn't
> exist.

That's correct.

> Note that this just-in-case breaking is done
> in the backup directory, not in the tree:
> 
>   # code from pop
>   patch="$(top_patch)"
>   if [ -z "$patch" ]
>   then
>     printf $"No patches applied\n"
>   else
>     # Ensure that the files in the topmost patch have a link count
>     # of one: This will automatically be the case in the usual
>     # situations, but we don't want to risk file corruption in weird
>     # corner cases such as files added to a patch but not modified.
>     $QUILT_LIB/backup-files -L -s -B $QUILT_PC/$patch/ -
>     printf $"Now at patch %s\n" "$(print_patch $patch)"
>   fi
> 
> The command
> 
>   backup-files -L -s -B $QUILT_PC/$patch/ -
> 
> means "walk the backup directory $QUILT_PC/$patch,
> and replace any file that has a link count > 1
> with a clone of that file."

No, sorry, it doesn't mean that. See for yourself (with the original C 
implementation of backup-files, which is the reference):

# ln AUTHORS AUTHORS2
# ls -l AUTHORS AUTHORS2
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 AUTHORS
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 AUTHORS2
# lib/backup-files -B backup/ -b -L AUTHORS
Copying AUTHORS
# ls -l AUTHORS AUTHORS2 backup/AUTHORS
-rw-r--r-- 1 jdelvare users 1110 2011-03-21 11:29 AUTHORS
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 AUTHORS2
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 backup/AUTHORS
# ln AUTHORS AUTHORS3
# ls -l AUTHORS AUTHORS2 AUTHORS3 backup/AUTHORS
-rw-r--r-- 2 jdelvare users 1110 2011-03-21 11:29 AUTHORS
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 AUTHORS2
-rw-r--r-- 2 jdelvare users 1110 2011-03-21 11:29 AUTHORS3
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 backup/AUTHORS

At this point, both the original and the backup files have a hard-linked 
copy.

# lib/backup-files -B backup/ -L -s -
# ls -l AUTHORS AUTHORS2 AUTHORS3 backup/AUTHORS
-rw-r--r-- 1 jdelvare users 1110 2011-03-21 11:30 AUTHORS
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 AUTHORS2
-rw-r--r-- 1 jdelvare users 1110 2011-03-21 11:29 AUTHORS3
-rw-r--r-- 2 jdelvare users 1110 2009-09-09 10:18 backup/AUTHORS

As you can see, backup-files -L broke the link of the original file, and 
left the backup file untouched.

I told you twice already, but I can write it a third time: backup-files 
semantics are not trivial.

-- 
Jean Delvare
Suse L3