[Gnu-arch-users] Re: give us a hand with arch

[Pau Aliagas]

On Sat, 27 Sep 2003, Andrea Arcangeli wrote:

> On Fri, Sep 26, 2003 at 12:10:44PM +0200, Pau Aliagas wrote:

> however these pure versions have to be one on top of the other somehow,
> or as you say they will generate rejects. I don't want to maintain two
> versions: one against mainline and one not. Mainline is a moving target
> so those patches often rejects anyways, and I don't want to do
> maintainace twice all the time.

Ok, derive your branches from the version you expect to merge to amb from.

> I simply order the chain in function of what will get merged first, so I
> also my risk of having to solve rejects by hand is lower, over time
> while the code gets merged into mainline.

You can specify which version--patch-level you derive from. More 
granularity is impossible as this is the maximum.

> Assuming there would be no dependency across patches everything is quite
> clear now.
> 
> But the dependency bit and the need of ordering (i.e. my "re-tag" idea)
> isn't totally clear yet.

No need of re-tag. Think exactly what you mantain and build the proper 
tree of versions.

> Note that I don't want to maintain an -aa tree at all. I ideally want my
> -aa tree to be only a new branch deleted and regenerated every time and
> always tagging the last branch in the chain. Basically the -aa tree
> would have only the base-0 patchset and that would be a tag only. No
> code in it.
> 
> I want arch to checkout in order through the whole chain.
> 
> And this way I retain the development comments etc..

Read below. You could have or not an aa tree, you obviously need a bunch 
of versions ready to merge with main trees.

> So I can diff an older tree with a newer tree, and it will show me not
> only which branches are affected (i.e. with old-style "patches") but it
> will tell me exactly what has changed in detail as well, possibly in
> multiple steps.
> 
> So conceptually it looks quite cool and it would retain lots of more
> information than my current 'patches' do.

You'd keep al the information in patch logs in every tree.

> Note also that often the prism-merging wouldn't work for me

The prism merging is fine when you hack and you want tom move a 
consolidated patchset to a pure version. If you do it directly in the pure 
version, that's fine too as you save a few steps. I just borrowed Tom's 
proposal as a way of sorting out a few patches put them in a patchset and 
moving them to the "pure version".

> often when I fix a bug I run quickly into a depdendency, so it's much
> safer that I fix the bug directly in the "pure" o1-sched branch, than in
> the final "working" tree if I've to fix a bug in the o1 scheduler.

It's fine to do it in the o1-sched tree, you save steps. Use the prism 
merging technique when you hack in a tree and then yu realise it should go 
to another ;)

> Otherwise I will risk a reject and having to write the fix twice when I
> apply it from "working" to "pure". Note: this is exactly what I'm
> already doing automated, I normally fix bugs in their own 'branch' to
> avoid duplicating work.

It's fine as you say.

What I see is that you have two needs:

-have a bunch of versions ready to merge with external trees (mainly 
 Marcelo, but could be Andrew's, Linus' or whoever's). Let's call these 
 "border versions"

-have a few of versions for internal merging/testing while developing, but 
 to export patchsets to the "border versions" when ready. Let's call these
 "internal versions".

(I'll try to call them versions instead of branches to be in synch wot 
arch terminology).

You don't need anything special to work as you want. You only have to
create the versions you expect to merge some day, the "border versions",
derived from the trees where you want to merge them. Let's put an example:

-you have the o1-sched and vm-bug-squashing versions
-you expect these 2 versions (and 25 more) to merge someday with Marcelo 
-you want them ready to export (via patch, tla get-patchset, tla 
 star-merge or whatever), free of conflicts

Procedure
---------

Derive from the version where you expect to merge back and forth
$ tla tag linux-kernel--marcelo--2.4 linux-kernel--aa-o1-sched

Create the working tree (fell free to use hardlinks as explained in 
previous mail)
$ tla get linux-kernel--aa-o1-sched linux-kernel--aa-o1-sched
$ cd linux-kernel--aa-o1-sched
Patch wildly for a few days.
>From time to time you want to synchronise with Marcelo, so you update your 
workibg tree with his latest patches:
$ tla whats-missing -s linux-kernel--marcelo--2.4
----list of patches + summary -----

You have 3 options, at least: update, replay or star-merge
$ tla replay linux-kernel--marcelo--2.4
$ tla star-merge linux-kernel--marcelo--2.4
$ tla update linux-kernel--marcelo--2.4

I think the smartest one is tla star-merge that would do the best choice.

Update would undo your patches up to latest Marcelo's version in your 
tree, apply Marcelo's patches (no conflict) and re-apply your patches 
(maybe conflicts to solve patch by patch, maybe not).

Replay is maybe the worst as it gives precedence toyour patchs, applying
the missing Marcelo's patches on the top of yours.

You'll sure have conflicts when Marcelo merges your patches and you try 
to re-apply them on your tree (unless he uses arch, then you'd see it's 
one of your patches and would skip it). You can solve the differences and 
re-run star-merge/update and then it will go on from next patch.

So the graph for the "border branches" would be:

                        border branches
                        ---------------

               | < ~~~~ >> feature a  <---> |
               |                            | marcelo
       working | < ~~~~ >> feature b  <---> |
               | ....                       
               | < ~~~~ >> feature c   <--->| linus

You can add an aa tree that you'd star-merge with all the pure versions,
if you needed it. it's only a matter of star-merging with the respective 
versions.

> Having always a working dir for aa is fine. However I'm not going to
> have a working dir for all the branches. That would be overkill, even
> with hardlinks, the directory metadata space would be a too huge waste.
> So I still definitely must avoid the tar xzf during checkout, but I
> believe I was wrong suggesting the "virtual tag", a simple cache of the
> repository copied via cp -al does the trick. And I understand this is
> how it's already working.
> 
> So in short, all I need is to 'tla cachrev linux--marcelo--2.4' and a
> 'tla get' that preserves the hardlinks, so I can extract the 'pure'
> trees quickly to work on them. Really I'm unsure yet if arch caches via
> tar.gz or via an unpacked tree (didn't try to run this command yet). If
> it's caching via tarballs, then I will need the library hooks with shell
> script etc...

You can have n extracted tree using hardlinks using cp -al, as explained
in the previous mail. When we have tla get --hardlinks you'l do it
directly. The only drawback will as you keep on patching, brealing the
hard links, but that doesn't seem a showstopper.

> But if the hooks are only meant to make the checkout of a tree faster
> without passing throug the patching (not to preserve the hardlinks),
> then I don't care in the short term. I believe we need to improve the
> applying of a 'patchset' anyways and gzip could be the worst offender at
> the moment. Also I'm unsure if it's unpacking the patchset in memory via
> libz + an embedded tar, or if it's unpacking somewhere in a tmp
> directory, in the latter case that should better go in /dev/shm or it'll
> slowdown into the fs code. But those are implementation details, I'm
> caring most about the design right now.

As Tom said, let's take one by one. Let's start working and later on see 
if there are any flaws or is just your eagerness to use it RSN.

> NOTE: I will work on the 'aa' branch only for new complex feature I feel
> won't be merged in mainline quick. For the others I will create a branch
> at the top of the chain (so it won't risk to reject when it's replied
> into the mainline branch).

You'll create special version for each new feature and not pack them all 
in aa directly, so that it's easier to export. You can combine them all in 
aa effortlessly afterwards. You couldevenhave different aa's, specialised 
in several things, including patches from the necessary branches. Imagine 
aa-s390. aa-new-genvm...

> > > > Naming patches is not possible, but you have summary + keywords + log 
> > > > associated to each one. Much more expressive than a simple name.
> > > 
> > > Yes. Though if the patch names would be named that would be a natural
> > > API to implement a command that extracts all the patchsets into a
> > > directory, plus a .ordering file that lists the order where they should
> > > be applied. That would be basically the explort from arch to 'patch'.
> > >
> > > Then you can do:
> > > 
> > >   for i in `cat .ordering`; do patch -p1 < $i; done
> > > 
> > > and reach the tree too.
> > 
> > Usually you should apply the patches in the commit order, not to worry
> > about interdependencies. So why not get patch-n to patch-n+i separatedly
> > and be done with it? no need for ordering, order will be provided by the
> > patch number. Once you have extracted the patches, you can renme them and
> > do whatever if thats's how you you to send out patches, that's orthogonal
> > to arch.
> 
> Those patches are the 'branches'. I call them patches but they're
> branches not the patchsets in each branch ;)
> 
> so they will have ordering dependency. Each branch has to be tagged from
> another branch.

Yes, but let's call them versions to stick to the arch conventions 
cat--branch--version--revision

> > 3. no need to ta get each time
> 
> since I'll normally fix bugs in the "pure" branch directly, I feel I
> need a quick 'tla get' that only does cp -al + patch -p1, like I do
> today. Just there will be lots of more patches since arch will be
> granular. I can't keep all pure branches unpacked at the same time, that
> will take too much space, and around 90% of those pure branches never
> ever need changes. They only get eventually merged into mainline.

Already answered.

> > > > > Also I can't upload 200m every time, so I need to tell arch to tag
> > ....
> > > so basically you're saying that I won't need to regenerate the tree some
> > > hundred times like I'm doing right now ;), and so in turn I won't care
> > > anymore about the speed of the checkout. maybe I'm biased about the
> > > absolute need of quick clones not involving an 200M unpack, after having
> > > cloned trees thousand and thousand of times ;)
> > 
> > You'll replay changes, or star-merge, but you don't need to tla get often, 
> 
> see above.
> 
> > only when you need a project which you don't have tree for. Once you have 
> 
> and this project is one of the hundreds of pure branches.

Idem.

> > the tree, even with local uncommied changes, importing the missing patches 
> > is done like this:
> > cd your-aa-tree
> > # have a look at what's missing
> > tla what-smissing -s
> > tla replay
> > # or have a look at whats missing from a related project
> > tla whats-missing linux--linus-2.4
> > tla star-merge linux--linus-2.4
> 
> yes, I believe the "tla star-merge linux--linus-2.4" maybe lovely, since
> it's exactly what I'm doing by hand by extracting the patchsets by hand
> from cvsps first and backing it out from mainline, every time I get a
> reject.

Hopefully we can have a linux-kernel--linus--2.6 and a  
linux-kernel--marcel--2.4 built directly from cvs to have the same 
granularity than bk users. Now we have a linux archive from 0.01 but the 
granularity is relase, pre and rc. For the oldkernels that's fine, but I 
suppose that for the new ones is too little as conflicts would be huge. 
OTH, if you have the bk changesets, conflicts will be small, localised and 
easy to solve.

> For the non conflicting parts of mainline evolution I don't care if it's
> my old patchsets applied on top of new mainline, or if it's new mainline
> changeset on top of my old -aa. But when I get conflicts I definitely
> want the mainline changeset to reject when applied, not the other way
> around. And the granularity of cvsps plus the above sounds like it could
> make make my life much easier by automatic and logging everything that
> goes on with my tree, so I can look back if something went wrong.

Yes, I think so, but nobosdy has yet imported the CVS linux trees into 
arch. I'm eager to do it but will not have free time in the forseeble 
future.

The precedenco of patches is determined on the command you use:
-replay -> apply missing patches on the top of the current tree, that is. 
 local patches take precedence.
-update -> undo my patches up to the atest merge, apply missing patches 
 and re-appply my patches. That is external patches have precedence.
-star-merge: not sure how it would do it.

> Let's assume I'm Marcelo, how can he ask arch to get all patches in the
> branch sched-o1, and not all the patches in the previous branches
> (assuming the sched-o1 branch isn't the first one that tag directly on
> top of marcelo?)
> 
> basically I would need it to tell, pick all patchsets (patch-1/2/3
> etc..) in this very branch but ignore everything that happened before
> base-0. This is what I normally I would expect him to do. Right now this
> command is very easy for him: it consists in wget
> ftp://ftp.kernel.org/pub/people/andrea/..../o1-sched-patch; patch -p1 <
> o1-sched-patch.

It could be done the same if you exported the patches, but it looks like 
the stone age when you are used to arch :b
Moreover, when you rsynched with his non-arch tree, youd have to fix te 
conflicts arised of importing your own patches again.

If he used arch, he coud so several things:
-the worst: cherry-pick patches
 It's bad because it's painful compared to the other. It would be more or 
 less what you do now, but not having to export the patches.
-the optimal: he could star-merge with your tree
 If he's confident on you and you have a completely pure branch for each 
 feature, he could (and definately should) star-merge with you. He could 
 then review the changes and commit. This way, he'd have your patch-logs 
 and you'd BOTH completely avoid ALL merging conflicts. You'd star-merge 
 one from the other wildly!

> Then if the patch doesn't apply he will fix the rejects.
> 
> > -tla replay to update a working dir to the latestspecfied version in the 
> >  repo
> 
> does tla replay also fallback in the tagged branches in between "marcelo"
> and "o1-sched" right? I was thinking at a local-replay.

With the new topology I propose you wouldn't have middle branches, so you 
wouldn't need t star-merge in chain.

> Also what's the difference between 'tla replay and tla update'?

Seen.

> > Have a look a the tutorial that is much better than my explanations.
> 
> I read the one in the tar.gz package in the files section, but it wasn't
> exactly clear the difference between update/replay, however if it's
> explained in the new src.rpm just avoid to answer the above lame
> question ;).

Read the docs in the current source, the others are obsolete.

> > You've got a few answers on this one.
> 
> and I'll answer it separately too.
> 
> > You also got more qualified answers on this one: cscvs and tla-tools are 
> > advanced tools to dela woth CVS import/export automatically. You can get 
> > them from the repos:
> > 
> > Register thes archives and tla get them:
> > tla register-archive address@hidden \
> >     http://quackerhead.com/~duff/{archive}/address@hidden
> > tla get address@hidden/cscvs--experimental--1.1 target-dir
> > 
> > tla register-archive address@hidden \
> >     http://arch.linuxguru.net/~miles/address@hidden
> > tla get address@hidden/tla-tools--devo--0 target-dir
> 
> thanks for the url. I wonder why they weren't anywhere in the webpage. I
> also wonder where's the url of arch itself, the cvs was empty, and I
> found no arch urls.

I register archives as I see them on the list and they look useful. The 
wiki is going under a revision, so that might be it.

> > And you can create local versions, publish a mirror and ask them to pick 
> > up your changes :)
> 
> I hope they already work fairly well ;) The cvs2arch night hack already
> worked fine for a repository.

The oter tools look impressive from what I've read in the list. And Miles 
has a few tools to help in the explicit tagging of sources. You'd sure 
need them.

> > > > You can export changsets to patches directly doing tla get-patch. 
> > > > That'd 
> > > > do the trick :)
> > > 
> > > oh that's the fundamental command I couldn't find ;)
> > 
> > It looks like you'll be using it a lot.
> 
> :)
> 
> actually I thought I needed it so much because I was thinking in the
> wrong dimension. really I don't care about the single patchsets, they're
> great to keep logs of the evolution of the code but I won't check them
> so often.

Better work with replay/update/star-merge/prims merging; use get-patch if 
you need to export patches.

> what I need is a get-local-branch, I want everything from base-0 to the
> last patchset in each branch. But it must not pass the 'tag'.

tla get cat--branch--versions--rev dest_dir

> I can probably diff two pure versions in order too instead but the
> 'get-local-branch' concept doesn't sound too bad either.
> 
> Infact the get-local-branch that shows an unified diff is very important
> especially for who has to merge the patch. In the kernel we just can't
> sync the tree and be done with it. We've to look everything before
> synching. There are exceptions, like when somebody maintains a totally
> isolated subsystem, but for what I deal with I definitely can't merge it
> blindly, that's out of question. That's also why working with unified
> diffs is the preffered for me so far.
> 
> examining the tree and reading the unified diffs like cvsps -g is a
> fundamental operation for me and I guess for most other kernel developers.

It's MUCH easier with arch.

# Synchronise the trees
$ cd pure_version_tree
$ tla update
# see what has changed EASYLY
$ tla what-changed --diffs
You can do tla what-changed --keep and browse the ,,what-changed dir too.
You can do tla what-changed revision to see what changed from a particular 
revision. It's really powerful.

> > > if the arch exports works well enough, after we've something like cvsps
> > > for arch, I may prefer to shrink it to one line only too. We'll see. the
> > > 2.4 version is the most challenging from my point of view, since my 2.4
> > > tree is quite huge and hard to maintain.
> > 
> > Try these tools instead of working from scratch.
> 
> I will try ;) thanks.
> 
> I hope it's not a too-corner case usage. At least I feel like if it can
> cover my needs, then I doubt it can't cover any other much simpler
> usage, i.e. like the way they use bitkeeper in 2.[56] ;)

The only deficit by now woud be the tla get --hardlinks, but that is 
easyly overcomed, as explained. The rest is only practice, focusing and 
understanding how arch works. Once you get it, maybe you'll refine the 
explained techniques as possibilitiesare almost infinite.

Setup your enviroment, learn and try that Marcelo and Andrew use it. Need 
help' Post to the list. I don't see how bk can do better.

> So far I always thought it was nearly impossible for me use a revision
> control system for my 2.4 tree. I even planned to write a sort of
> "patch" management system by evolving my scripts, so that people could
> also pick patches and that auto upgrades on new releases and tells you
> when something rejects so you fixup and it keeps the log. And my idea
> close to what arch is already (but I was thinking at each branch to be a
> single patchset, I'm biased). Reading the docs arch is presented more as
> a regular but more powerful revision control system, but if you see it
> as a patch managemnet system, then you can exploit best the features
> like the fact a 'tag' branch is zerocost and its patchsets won't be
> different from a normal patch, in the sense their size won't depend on
> how many files are in the tree etc...

You'll learn t love it. Many more things can be done like patch queue 
managment, bug tracking, etc, but those are issues for 
yet-another-never-ending-thread.

Pau