[Pan-users] Re: Pan and rsync

[Duncan]

Joe Zeff posted on Thu, 05 Aug 2010 22:09:00 -0400 as excerpted:

> On 08/05/2010 04:49 PM, Duncan wrote:
>> That could be it.
> 
> And then again...
> 
> As I started to read each group, Pan marked all of the old posts read,
> leaving me with nothing but the newest ones.  Interesting.  Last time,
> when I didn't check the newsrc file, it didn't do that, just expected me
> to read them.  Now, it's marking them read as I enter the group.  OK, I
> can live with that.

(Referring to the files as I type, I didn't have all this memorized, tho 
Charles might! =:^)

Yeah.  The (per-server) newsrc file(s) simply lists per-group xref 
sequence numbers (and whether the group is subscribed, etc).  These are 
the ranges of posts it has seen and marked read, but they /only/ track 
subscribed and marked-read.

*Most* of the the rest of the group data is in the groups subdir, in the 
files named after the individual groups.  These contain pan's header 
cache, formatted in three parts as mentioned in the remarks at the top of 
the file.  The first two parts are xref and authors shorthands.  These 
substantially reduce string duplication and thus pan's memory usage, as 
well as file-size on-disk.  The third part is a per-post per-header record 
of the overview data, message-id, subject, author (shorthanded if it's one 
of the most frequent posters, looks like the shorthand lists are 72 
entries long), references line (if present), time, xref (shorthanded), two 
lines of attachment info.

The other file of interest here is newsgroups.xov (xover, the overviews 
index file).  This file consists of lines tracking per-group total article 
count, unread count, and per-group-per-server high-water-mark, which is 
the xref-sequence number of the highest visible (generally, newest) post 
on the server.  The difference between the stored high-water-mark and the 
one pan gets when it checks with the server is the number of messages on 
that server that pan believes it doesn't know anything about yet.

Keep in mind that it's only the newsrc files that are standardized.  The 
rest are pan-specific format.

You manually synced (when rsync failed) the newsrc files, which tracked 
the messages you'd read, but apparently, the newsgroups.xov file wasn't 
synced either, so when pan checked and saw a much higher high-water-mark 
on the server, it requested all the messages between its (old) stored high-
water-mark and the current one it just got from the server.  That's the 
header download.  Once it compared those headers with the records in the 
newsrc files, however, which it does on entry to the group, it realized it 
already knew about them and that you'd already read them, so it then 
marked them read.

But it's likely a good thing you did download them.  If the groups subdir 
didn't sync either, as now seems likely, those headers/overviews wouldn't 
be listed there either, and had the newsgroups.xov file been synced, pan 
would have had not actual record of those headers as it would have 
believed it had already seen them (and that they'd simply expired or you 
deleted them), based on the high-water-mark from the newsgroups.xov file.  
If you expire the headers anyway, no big deal, unless you wanted to 
actually see them for some reason, but if you keep everything around 
unexpired, as I do, that would have been a gap in the archive, as I said, 
messages that from pan's viewpoint, you'd seen and then deleted.



Meanwhile, I may have figured out why you're not getting a good sync.  
Near the bottom of the rsync manpage, there's a list of environmental vars 
it uses.  In addition to the rsync-specific ones which don't look 
interesting to us, there's three more general ones, one (which it checks 
two vars for) of which DOES look interesting.

According to the manpage, rsync (on the local/client side) looks for the 
USER or LOGNAME var, using this as the default username sent to rsyncd.  
If neither is set, it defaults to user "nobody".

I suspect that however you're running rsync, both of these (USER would be 
most common) are either unset or not exported, so rsync isn't seeing 
them.  User "nobody" is unlikely to have permissions to write to the pan 
parts of your homedir, so...

I don't know what your user umask, the default perms mask, or your group 
setup.  On high security systems, user's home group will be the same name 
as the username, a 1:1 correspondence, and default umasks will default to 
0077, meaning group and world perms default to nothing.  On moderate 
security systems, there'll be one or more "users" groups, with more than 
one login user per group, and umasks will default to 0027 or similar, 
world still gets no permissions but the group gets read/execute, just not 
write. by default.  Low security systems often default to 0022, both group 
and world read/execute, not write.

If user "nobody" is in the same primary group as your user, the medium 
security 0027 umask would give it read but not write access unless you'd 
changed the default permissions, or no access if it's not in the same 
group.  With low security, it'd get read but not write access either way.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman