[Gnu-arch-users] Re: give us a hand with arch

[Andrea Arcangeli]

On Fri, Sep 26, 2003 at 12:10:44PM +0200, Pau Aliagas wrote:
> You now have the latest and greatest in your mailbox.

thanks ;)

> As I posted in a previous email, you'd have branches for the different 
> features, not for every commit. And you'd move those changes from the 

correct.

> feature branch (pure version) to your aa branch. Rejects can be treated 
> manually or, as you seemto do, intelligently with some scripting. The 
> impotant thing is doing the development in the right branch, so that you 
> know atutomatically what type of patches you'll merge from that branch.

yes.

however these pure versions have to be one on top of the other somehow,
or as you say they will generate rejects. I don't want to maintain two
versions: one against mainline and one not. Mainline is a moving target
so those patches often rejects anyways, and I don't want to do
maintainace twice all the time.

I simply order the chain in function of what will get merged first, so I
also my risk of having to solve rejects by hand is lower, over time
while the code gets merged into mainline.

Assuming there would be no dependency across patches everything is quite
clear now.

But the dependency bit and the need of ordering (i.e. my "re-tag" idea)
isn't totally clear yet.

Note that I don't want to maintain an -aa tree at all. I ideally want my
-aa tree to be only a new branch deleted and regenerated every time and
always tagging the last branch in the chain. Basically the -aa tree
would have only the base-0 patchset and that would be a tag only. No
code in it.

I want arch to checkout in order through the whole chain.

And this way I retain the development comments etc..

So I can diff an older tree with a newer tree, and it will show me not
only which branches are affected (i.e. with old-style "patches") but it
will tell me exactly what has changed in detail as well, possibly in
multiple steps.

So conceptually it looks quite cool and it would retain lots of more
information than my current 'patches' do.

Note also that often the prism-merging wouldn't work for me, often when
I fix a bug I run quickly into a depdendency, so it's much safer that I
fix the bug directly in the "pure" o1-sched branch, than in the final
"working" tree if I've to fix a bug in the o1 scheduler. Otherwise I
will risk a reject and having to write the fix twice when I apply it
from "working" to "pure". Note: this is exactly what I'm already doing
automated, I normally fix bugs in their own 'branch' to avoid
duplicating work.

> > Especially I still need the "virtual tag" (again s/hook/tag/), into a
> > unpacked tree. Otherwise when I checkout I need to unpack 200M. That's
> > way overkill, and caching wouldn't be useful either. 
> > 
> > I mean, I must be able to do a:
> > 
> >     tla get linux--aa--2.4
> > 
> > and to get out the tree in seconds, I don't like to wait 200M to be
> > written on disk before I get it. That would slowdown me too much. As
> > said I'm currently down to a dozen of seconds by using /dev/shm with my
> > current scripts to fixup a reject (that in this case means merging from
> > a branch with a reject in between).
> 
> You shoud always have a working dir for aa, I don't see why you should 
> get it and delete it. And you can have working dirs fr the feature 
> branches (pure branches if you feel like).

Having always a working dir for aa is fine. However I'm not going to
have a working dir for all the branches. That would be overkill, even
with hardlinks, the directory metadata space would be a too huge waste.
So I still definitely must avoid the tar xzf during checkout, but I
believe I was wrong suggesting the "virtual tag", a simple cache of the
repository copied via cp -al does the trick. And I understand this is
how it's already working.

So in short, all I need is to 'tla cachrev linux--marcelo--2.4' and a
'tla get' that preserves the hardlinks, so I can extract the 'pure'
trees quickly to work on them. Really I'm unsure yet if arch caches via
tar.gz or via an unpacked tree (didn't try to run this command yet). If
it's caching via tarballs, then I will need the library hooks with shell
script etc...

But if the hooks are only meant to make the checkout of a tree faster
without passing throug the patching (not to preserve the hardlinks),
then I don't care in the short term. I believe we need to improve the
applying of a 'patchset' anyways and gzip could be the worst offender at
the moment. Also I'm unsure if it's unpacking the patchset in memory via
libz + an embedded tar, or if it's unpacking somewhere in a tmp
directory, in the latter case that should better go in /dev/shm or it'll
slowdown into the fs code. But those are implementation details, I'm
caring most about the design right now.

NOTE: I will work on the 'aa' branch only for new complex feature I feel
won't be merged in mainline quick. For the others I will create a branch
at the top of the chain (so it won't risk to reject when it's replied
into the mainline branch).

> > > Naming patches is not possible, but you have summary + keywords + log 
> > > associated to each one. Much more expressive than a simple name.
> > 
> > Yes. Though if the patch names would be named that would be a natural
> > API to implement a command that extracts all the patchsets into a
> > directory, plus a .ordering file that lists the order where they should
> > be applied. That would be basically the explort from arch to 'patch'.
> >
> > Then you can do:
> > 
> >     for i in `cat .ordering`; do patch -p1 < $i; done
> > 
> > and reach the tree too.
> 
> Usually you should apply the patches in the commit order, not to worry
> about interdependencies. So why not get patch-n to patch-n+i separatedly
> and be done with it? no need for ordering, order will be provided by the
> patch number. Once you have extracted the patches, you can renme them and
> do whatever if thats's how you you to send out patches, that's orthogonal
> to arch.

Those patches are the 'branches'. I call them patches but they're
branches not the patchsets in each branch ;)

so they will have ordering dependency. Each branch has to be tagged from
another branch.

> 3. no need to ta get each time

since I'll normally fix bugs in the "pure" branch directly, I feel I
need a quick 'tla get' that only does cp -al + patch -p1, like I do
today. Just there will be lots of more patches since arch will be
granular. I can't keep all pure branches unpacked at the same time, that
will take too much space, and around 90% of those pure branches never
ever need changes. They only get eventually merged into mainline.

> > > > Also I can't upload 200m every time, so I need to tell arch to tag
> ....
> > so basically you're saying that I won't need to regenerate the tree some
> > hundred times like I'm doing right now ;), and so in turn I won't care
> > anymore about the speed of the checkout. maybe I'm biased about the
> > absolute need of quick clones not involving an 200M unpack, after having
> > cloned trees thousand and thousand of times ;)
> 
> You'll replay changes, or star-merge, but you don't need to tla get often, 

see above.

> only when you need a project which you don't have tree for. Once you have 

and this project is one of the hundreds of pure branches.

> the tree, even with local uncommied changes, importing the missing patches 
> is done like this:
> cd your-aa-tree
> # have a look at what's missing
> tla what-smissing -s
> tla replay
> # or have a look at whats missing from a related project
> tla whats-missing linux--linus-2.4
> tla star-merge linux--linus-2.4

yes, I believe the "tla star-merge linux--linus-2.4" maybe lovely, since
it's exactly what I'm doing by hand by extracting the patchsets by hand
from cvsps first and backing it out from mainline, every time I get a
reject.

For the non conflicting parts of mainline evolution I don't care if it's
my old patchsets applied on top of new mainline, or if it's new mainline
changeset on top of my old -aa. But when I get conflicts I definitely
want the mainline changeset to reject when applied, not the other way
around. And the granularity of cvsps plus the above sounds like it could
make make my life much easier by automatic and logging everything that
goes on with my tree, so I can look back if something went wrong.

> You see, your working tree is always up to date effortlessly.

Yes.

> -tla star-merge to synchronise trees

ok

> -tla get-patch to cherry-picj patches from other archives

ok.

Let's assume I'm Marcelo, how can he ask arch to get all patches in the
branch sched-o1, and not all the patches in the previous branches
(assuming the sched-o1 branch isn't the first one that tag directly on
top of marcelo?)

basically I would need it to tell, pick all patchsets (patch-1/2/3
etc..) in this very branch but ignore everything that happened before
base-0. This is what I normally I would expect him to do. Right now this
command is very easy for him: it consists in wget
ftp://ftp.kernel.org/pub/people/andrea/..../o1-sched-patch; patch -p1 <
o1-sched-patch.

Then if the patch doesn't apply he will fix the rejects.

> -tla replay to update a working dir to the latestspecfied version in the 
>  repo

does tla replay also fallback in the tagged branches in between "marcelo"
and "o1-sched" right? I was thinking at a local-replay.

Also what's the difference between 'tla replay and tla update'?

> Have a look a the tutorial that is much better than my explanations.

I read the one in the tar.gz package in the files section, but it wasn't
exactly clear the difference between update/replay, however if it's
explained in the new src.rpm just avoid to answer the above lame
question ;).

> You've got a few answers on this one.

and I'll answer it separately too.

> You also got more qualified answers on this one: cscvs and tla-tools are 
> advanced tools to dela woth CVS import/export automatically. You can get 
> them from the repos:
> 
> Register thes archives and tla get them:
> tla register-archive address@hidden \
>     http://quackerhead.com/~duff/{archive}/address@hidden
> tla get address@hidden/cscvs--experimental--1.1 target-dir
> 
> tla register-archive address@hidden \
>     http://arch.linuxguru.net/~miles/address@hidden
> tla get address@hidden/tla-tools--devo--0 target-dir

thanks for the url. I wonder why they weren't anywhere in the webpage. I
also wonder where's the url of arch itself, the cvs was empty, and I
found no arch urls.

> And you can create local versions, publish a mirror and ask them to pick 
> up your changes :)

I hope they already work fairly well ;) The cvs2arch night hack already
worked fine for a repository.

> > > You can export changsets to patches directly doing tla get-patch. That'd 
> > > do the trick :)
> > 
> > oh that's the fundamental command I couldn't find ;)
> 
> It looks like you'll be using it a lot.

:)

actually I thought I needed it so much because I was thinking in the
wrong dimension. really I don't care about the single patchsets, they're
great to keep logs of the evolution of the code but I won't check them
so often.

what I need is a get-local-branch, I want everything from base-0 to the
last patchset in each branch. But it must not pass the 'tag'.

I can probably diff two pure versions in order too instead but the
'get-local-branch' concept doesn't sound too bad either.

Infact the get-local-branch that shows an unified diff is very important
especially for who has to merge the patch. In the kernel we just can't
sync the tree and be done with it. We've to look everything before
synching. There are exceptions, like when somebody maintains a totally
isolated subsystem, but for what I deal with I definitely can't merge it
blindly, that's out of question. That's also why working with unified
diffs is the preffered for me so far.

examining the tree and reading the unified diffs like cvsps -g is a
fundamental operation for me and I guess for most other kernel developers.

> > if the arch exports works well enough, after we've something like cvsps
> > for arch, I may prefer to shrink it to one line only too. We'll see. the
> > 2.4 version is the most challenging from my point of view, since my 2.4
> > tree is quite huge and hard to maintain.
> 
> Try these tools instead of working from scratch.

I will try ;) thanks.

I hope it's not a too-corner case usage. At least I feel like if it can
cover my needs, then I doubt it can't cover any other much simpler
usage, i.e. like the way they use bitkeeper in 2.[56] ;) So far I always
thought it was nearly impossible for me use a revision control system for
my 2.4 tree. I even planned to write a sort of "patch" management
system by evolving my scripts, so that people could also pick patches
and that auto upgrades on new releases and tells you when something
rejects so you fixup and it keeps the log. And my idea  close to
what arch is already (but I was thinking at each branch to be a single
patchset, I'm biased). Reading the docs arch is presented more as a
regular but more powerful revision control system, but if you see it as
a patch managemnet system, then you can exploit best the features like
the fact a 'tag' branch is zerocost and its patchsets won't be different
from a normal patch, in the sense their size won't depend on how many
files are in the tree etc...

Andrea - If you prefer relying on open source software, check these links:
            rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
            http://www.cobite.com/cvsps/
            svn://svn.kernel.org/linux-2.[46]/trunk