Re: [Gnu-arch-users] Re: tla file-locking (good idea or bad idea?)

[Jeremy Shaw]

Hello,

Using branches has been suggested a few times, and I have discussed it
with some people, but I have never really come to satisfactory way of
doing it. But maybe I have some sort of mental block from too many
years of CVS.

The following is a more verbose description of what we are trying to
do -- which might help explain why we currently use file-locking.

 We prefer that all debian packages are built in the autobuilder
before going into the os. This is beneficial for several reasons:

(1) If the autobuilder builds the source, then we can be reasonably
    sure we can do it again later. In the past we had problems where
    people would check source into cvs that did not actually build
    when you checked it out (for example, cvs import might have
    accidently ignored a directory named acore, or the developer might
    have run autoconf by hand before typing dpkg-buildpackage, etc).

(2) We can be sure we actually have the source. If you have the source
    only on your laptop, build a buggy deb which gets into the OS, and
    then go home early, it makes it harder for someone else to fix
    it. Or perhaps, you just forget to check in your changes.

(3) If you rebuild a library, packages that depend on that library
    might also need to be rebuilt. The autobuilder can detect this and
    automatically rebuild any affected packages.

(4) Who knows what is installed on your laptop. We don't need packages
    in our OS that are linked against the NVidia driver just because
    you happen to have an nvidia card.

In debian, the version of a package is determined by the topmost entry
in the debian/changelog. This means that you need to have that entry
correct before you run dpkg-buildpackage (the debian command for
turning source into a binary package). The autobuilder has rules for
how to automatically generate new debian/changelog entries. So
normally, we let the autobuilder modify the debian changelog, and we
don't touch it at all.

However, sometimes it is neccessary to make a non-autogenerated debian
changelog entry. If you make such a change, the autobuilder will see
that you explicitly set the debian/changelog entry, and use your hand
crafted entry instead of autogenerating one. (The following builds
will be autogenerated again -- but using the new version number as the
starting point).

So, in brief, these are our basic requirements:

(1) the autobuilder must change its local copy of the debian/changelog
    before running dpkg-buildpackage (which can take hours to run).

(2) If the autobuilder completes successfully then it should commit
    its debian changelog.

(3) If the autobuilder fails to compile the package, it sends an error
    message and does NOT commit the changelog.

(4) The developer must be able to edit the debian/changelog in a
    manner that affects the autobuilder.

(5) The autobuilder must always be able to commit successfully if it
    completes the build.

(6) A developer should be allowed to commit changes to a package while
    the package is autobuilding, provided the commit will not affect
    the ability of the autobuilder to commit the debian/changelog.


Now, cleary (3) is the sticky point. If we commited a new changelog
even when the build failed, then we could simply do the commit at the
beginning, instead of waiting to see if the build failed or
not. However, our experience tells us this is not very desirable.

So, we have come up with 5 possible solutions:

(1) File Locking -- (current solution)

In cvs, if we simply lock the debian/changelog while a package is
being autobuilt, then the developer can commit any file they want,
except debian/changelog. This works great, because its very rare to
actually want to change the debian/changelog by hand. But much more
common to change a package while its being autobuilt.

The downside is, if you WANT to commit a debian/changelog, you have to
wait until the build is done, and then remember to do it.

(2) Make the autobuilder smarter about adding debian/changelog entries

Another possibility is to make the autobuilder smarter about adding
debian changelog entries. When it builds a package, it will update the
debian/changelog and commit it immediately. If the build is
successful, then everything is great.

If the build fails, the autobuilder will remember this. The next time
it attempts to build that package, it will look at the last entry in
the debian/changelog, and see if its from the previous build that
failed. If so, it will just reuse that existing entry. If the entry is
different, then it must be because someone explicitly changed the
entry, and so it will use that entry instead. [note: I am glossing
over the details here].

This solution would still occansionally result in their being a
debian/changelog entry for which no .deb was ever built, but that is
acceptable due to the rarity of it.

(3) implement a third party file locking mechanism

Basically what tom lord suggested. Have some third party program the
takes care of locking the debian/changelog and call it from the tla
hooks.

(4) Have the autobuilder build on its own branch

I am still exactly not clear on how this would work. I think maybe
there is a developers branch and an autobuilder branch. Before the
autobuilder builds, it tries to merge everything from the developers
branch, into its local branch. If there is a conflict, it dies and
sends an email?

If a developer wants to update the debian/changelog, then they have to
merge the debian/changelog from the autobuilder branch into the
developer branch, then edit and commit (in the developers branch). Of
course, if they doing this while the autobuilder is building, they
won't know they caused a problem until the next time the autobuilder
tries to compile that package?

This seems less desirable than the file-locking solution because you
still have to wait until the autobuilder is done with the package
before updating the changelog, but you don't get immediatly rejected.

Also, its valid (though somewhat frowned upon) to build the package by
hand and upload it. I am certain people will forget to merge the
debian/changelog from the autobuilder branch to the developer branch
before building. That won't cause any problems, except lost time. But
lost time still sucks.

(5) Something complicated

Allow some mechanism where if you commit a new changelog while the
package is being autobuilt -- then you have to provide two versions,
one that works when the build suceeds, and one that works when the
build fails. I think darcs has something similar to this, but not
quite.

In any case, we are currently looking at doing (2). Currently the
autobuilder is designed to access the source repository through an
abstraction layer. This would allow us to do things like point to
debian's sarge repository and say "rebuild this" (provide we write a
plugin for the abstraction layer). So, (2) would make the autobuilder
more generalized instead of more specialized. We still have to think
about it for a while.

However, I am interested in how to solve this problem with branches,
because I think I might be missing something important. Or maybe I do
understand already, but just don't like the trade-offs?

Jeremy Shaw.

At Wed, 21 Jan 2004 21:54:42 +0000,
Stig Brautaset wrote:
> 
> Hi,
> 
> On Jan 20 2004, Jeremy wrote:
> > I would like to add file-locking to tla. 
> ...
> > Here's why this could be useful -- We have an autobuilder that can
> > check out a project, build the .debs, and upload the packages and
> > source to our local 'debian' repository. When the autobuilder checks
> > out a package, it automatically generates a new debian/changelog entry
> > in its local copy before it starts the build. If the build is
> > successful, it will commit the debian/changelog. If the build fails,
> > then it will send an email, and leave the debian/changelog alone.
> 
> How about the autobuilder committing to its own archive/branch?
> 
> Stig
> -- 
> brautaset.org
> 
> 
> _______________________________________________
> Gnu-arch-users mailing list
> address@hidden
> http://mail.gnu.org/mailman/listinfo/gnu-arch-users
> 
> GNU arch home page:
> http://savannah.gnu.org/projects/gnu-arch/