[Pan-users] Re: Problems with some replies

[Duncan]

HarryB <address@hidden> posted
address@hidden, excerpted below, on  Thu, 11
Dec 2008 22:38:22 -0600:

> Some of what you have written is over my head, but I want to fix this
> problem, so I will do my best to understand what you are writing about.

Yes, some of it was a bit technical.  From the below, however, it does 
seem we're on the right track.

> I found that when composing a message, under the tab, "More Headers",
> one can specify whether to "Add "Message-Id header" or not. This
> apparently changes the message ID. With it checked the message ID is as
> you described above with a date/time stamp and the domain name that is
> entered by the user in the "Posting Profile" box. However, if it is
> unchecked, the message ID is entirely different.

Thanks for pointing that out!  I had either forgotten or never noticed 
those checkboxes, and tho I thought there should be a way to control 
message-ID in pan and might have looked there if I needed to do so, I had 
never bothered looking.

So you taught me something useful in this thread too! =:^)

> Here is an example from one of my test posts:
> <address@hidden>

Hmm, I read this list as a newsgroup on gmane (gmane.org, which provides 
a mailing-list to web and a mailing-list to news gateway and archives the 
lists as newsgroups, which I prefer to mailing lists -- I use pan to read 
the pan mailing list as a newsgroup! =:^), which munged that to prevent 
spam since it looked like an email address.  I'm going to do some testing 
of my own, but meanwhile, if you could post that with a space before any 
@ symbols, it might help as gmane shouldn't munge it in that case.  From 
the below it does seem it may be important.

Oh, and you didn't say which way you had been posting before changing it, 
or whether changing it made a difference!  As we're zeroing in on the 
references header and it'll contain those message-IDs, it could.  If you 
aren't doing so already, try a new test thread with it toggled opposite 
to however it was to begin with.  If you're lucky, that's all it'll take.

> There is another option there, but I don't know what it does: "Add
> "user-Agent" header".

This should add a header such as the following:

User-Agent: Pan/0.133 (House of Butterflies)

That would be unique between clients as well (some may use X-Newsreader 
or a similar header instead, or in addition), and while it could have 
been the problem, the below seems to indicate that the problem is in the 
differing references header, instead.

BTW, neither you nor I have mentioned this yet, but it may be helpful in 
your investigations so I will now, in case you didn't know:  You can use 
the View > Body pane > Show all headers in body pane toggle (I think the 
default hotkey is "h", I've changed some of the defaults here so can't 
say for sure) to see/not-see full headers, including references, user-
agent, Message-ID, etc.

> I did some tinkering based on your suggestion to open the reply in a
> text editor and modify the references header. Never having bothered to
> really look at this stuff, I learned that some of that information
> actually means something!

=:^)

> Here is what I did: I found a reply from Free Agent that was maybe 4 or
> 5 replies into the thread and was one that I was not able to reply to in
> Pan. I then eliminated all of the references except the first and the
> last and tried to reply to it in Pan. The message was posted to the n/g
> and it might even have ended up in the correct place in the thread.

OK, this is definitely progress (and what I was referencing with all 
those "below" references above)!  It does confirm we're on the right 
trail!  However, as you note below, manually changing each one is a HUGE 
pain!  Once we figure out what's being filtered in a bit more detail, 
hopefully we can either avoid it (that's what I'm hoping happens with the 
use-message-id toggle test I suggested above), or have enough info to 
present to your ISP as a reasonable request to get them to modify their 
filters.

> So, what do I do now? Apparently my ISP doesn't like it when Pan has
> more than two references in the header while who knows how many in Free
> Agent are acceptable. 

Well, it's not going to be quite that.  Rather, they're probably using 
some sort of spam filter that is keying in on something in those message-
IDs, and when it sees too many of whatever it is, it blocks the message.  
It's (probably) not blocking more than two references, but more than two 
(or three or whatever) of some element in those references.  In fact, it 
may be keying in on two separate parts of the message ID, in which case 
three references following the pattern would be six hits.

The challenge is to figure out what element of the message-ID it's keying 
on, and if possible, remove it.  We need more data, especially since I 
don't know for sure what the behavior of pan is without that supply 
message-id checkbox is, and couldn't see your sample due to the munging, 
AND, now that I know about the checkbox, don't know whether you had it on 
or off at first.

> So, I can fool my ISP's filter by changing the
> reference information in the header. I don't know if that will always
> allow a replied message to be placed in the proper location in a thread,
> and it is, of course a rather unwieldy way to post to the Usenet.

It's definitely unwieldy.  The suggestion is no more than a 
troubleshooting measure, however.  Hopefully, once we've traced it a bit, 
we'll be able to avoid the problem.

As for the correct location in the thread... the way it works is this:

The references header lists parent and grandparent and great-grandparent 
(but not siblings at any level)... up to the original post.  As the level 
of nesting gets deeper and therefore the references header longer, it's 
allowed to cut some of the middle ones out (as you did).  The original 
post message-ID should never be removed, however, and the immediate 
parent should also never be removed.

When threading, a good client will place the message in the appropriate 
place based on the references headers as best it can.  However, given the 
way USENET works, the number of independent servers there are, and the 
variables in propagation, it's not uncommon at all for a particular 
server to be missing the immediate parent of a post when it comes in.  In 
this case, the client should re-parent as best it can, threading under 
the grandparent if it's in the references and the grandparent is 
available, under the great grandparent if the grandparent isn't available 
either, etc, and ultimately, under the original post, if it hasn't 
expired already and no other intermediate generations are available.

Obviously, then, the more intermediate generation message-ids remain in 
the references, the better a client can re-parent in the event some of 
the intermediates are missing, for whatever reason.  It follows then that 
the references header should only be trimmed if necessary.  As long as 
you keep the original post and immediate parent message-ids (first and 
last in the list), it should be able to put it in the right place if the 
parent is available to do so, and under the original post (as long as 
it's not expired) if not.  However, if the parent is missing and there's 
intermediate generations, re-parenting under the original post is better 
than new-thread, but not ideal.

But, as long as the parent remains, it should find the right place in the 
thread, and as I said, hopefully we can nail this down and either solve 
or work around the problem, so this will be just a temporary issue.

> Hopefully I will have some time tomorrow to do some more tinkering. And
> maybe I have provided you with some useful information. I'm sorry if
> I've done a poor job of describing what I did, but much of this is new
> to me.

Your job was rather excellent, actually!  The only problem was the munged 
message-id, which wasn't your fault at all, since you had no reason to 
suspect the munging that gmane does.  Really, that's more my problem, 
since it's my decision to use gmane and pan rather than a normal mail 
client.  Unfortunately it affects you in this case since I'm working on 
your problem. =:^(  But as I said, repost that info putting a space 
before any @ symbols and gmane shouldn't munge it.

Meanwhile, thanks again for pointing out that checkbox! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman