Re: [Pan-users] where does pan cache it's incompleted downloads>

[Duncan]

Pedro posted on Thu, 04 Jan 2018 16:39:35 +1100 as excerpted:

> hi, sorry about that but I have no way of knowing about any reference
> headers in an email.
> 
> a. I deleted the title and replaced it with a new one thinking this was
> making a new thread.
> 
> b. I replied but changed the recipient to what I assume is this 'groups'
> email address. - address@hidden
> 
> c. I deleted the contents of the replied to thread and I cannot see any
> other information that I am sending.

?etouq erofeb ylper ,noitseuq erofeb rewsna ,thguoht drawkcab referp 
yllaer uoy oD

(Do you really prefer backward thought, answer before question, reply 
before quote?)

.senil desrever ,llits retteb eb lliw siht ebyaM

(Maybe this will be better still, reversed lines. [1])

Seriously, there's a reason pan highly discourages posting the reply 
before the quote with a warning if you try to send a post that has the 
reply first, instead of posting contextual replies, quoting what you are 
replying to and then replying to it.  Quotes followed by replies in 
context are simply more natural to follow, make clear what part of the 
post the reply was intended to reply to, and make it far easier for 
others to reply in turn in the same way.

It's unfortunate that Thunderbird and other popular mail and/or news 
clients aren't as similarly strict, warning about such things, but 
letting posters continue if they insist.

(FWIW, it's also part of GNKSA, Good Net-Keeping Seal of Approval, with 
pan one of few news clients that gets a 100% GNKSA pass.  There have been 
on-list discussions about whether pan should continue that in one point 
or another in the past, but the general feeling has been that while one 
or another point might be inconvenient at times, most pan users strongly 
support that GNKSA in general, strongly enough to support continued 100% 
compliance even if there's one or two individual points they don't 
necessarily entirely agree with, because once pan lost that 100% 
compliance mark due to loss of compliance on a single point, it would be 
all too easy to slip further and further away in other areas as well, 
until GNKSA was ignored entirely, and pan had lost much of the reason 
many people prefer it even for text groups.

But it's not /all/ bad! At least you spared us posts in html! =:^)

To answer your points, however...

[FWIW, this is a long post and a lot of information, but I imagine others 
besides you might find some of it interesting as well, so I think/hope 
it's worth the several hours I put into composing it. That's the 
advantage of newsgroups/mailinglists, where more than the person replied 
to can see the post. =:^) ]

Most clients don't display /all/ headers, unless you specifically ask 
them to.  In this case, the normally hidden references header, and the 
similar in-reply-to header, list the message-id of the post they're 
replying to, with in-reply-to listing only the immediate parent, while 
references, more common on news, lists all or most of the upthread.

Of course the message-id header itself is another such header, designed 
to uniquely identify individual messages, so they can be referred to in 
the above headers, listed in *.nzb files, etc.  Since message-ids are 
intended to be unique, pan uses them (with a few substitutions to make 
them work better as filenames) as the name of the individual message 
files in its cache, as well.

As mentioned, most clients will show you all headers on received messages 
if you tell them to.  (Sent messages are another matter, since some 
headers are added in transit or not assigned until you actually send the 
message.  See the lines and size headers, as well as the received headers 
added by mail servers as they get and pass on the message, for instance, 
and the similar entries in the path header for news.)

In pan the default hotkey to toggle view full headers on a received 
message is (IIRC, I've customized many of my hotkeys) the "H" key.  In 
claws, my mail client, it's the "V" key (view raw message, including 
headers).  Presumably, thunderbird has a similar option, which may or may 
not be assigned a hotkey.

> d. what strictly usenet group is this anyway? I can use pan.

While I don't believe I specifically said it was a newsgroup, it /does/ 
happen to be one, but a somewhat longer explanation may be necessary...

First, news and mail use the same general message format and many of the 
same headers, specified originally for mail messages in the RFCs. (RFCs 
are Internet Requests for Comment, the documents that eventually become 
the Internet Standards defining the protocols and formats required for 
interoperation of the great variety of different machines and software on 
the internet.)

News makes use of this originally mail message format, putting it to use 
for a one-to-many purpose, where a post is sent to many readers who 
connect to news servers to download it, as opposed to internet mail's 
normally one-to-one format, sent from one person to another specific 
person, tho it may be CCed or specific others as well.

Because the news and mail message formats are extremely similar, 
technically the same but for a few headers, which can be ignored by the 
other one, once someone has software handling one of the two, it's quite 
easy to add a little bit more code to handle the other one as well, 
almost "for free", since it's very little more code once one has the 
other one working properly.

This is why it's so common for clients that handle mail, to also handle 
news, altho not necessarily /well/, since news, particularly binaries, 
tends to be far higher message volume and bandwidth, and handling that 
many more messages as fast as necessary for efficient binary processing 
isn't easy.

But it also makes it reasonably easy to "gateway" news to mail, and mail 
to news, where that might be useful.

Which is where gmane comes in.  It's a service that "gateways" messages 
between mailing lists and newsgroups that match them, on a news server it 
hosts.

Now gmane has some interesting history, including a web-accessed version 
of the news service that was transferred to a new owner in the middle of 
2016, that never really came up again after the transfer, so the web side 
doesn't work, but the news side still does, and there's some 
(unfortunately limited) still /somewhat/ useful info about the news side 
on the website as well -- as long as you understand that some of it is 
stale and doesn't refer to current news-side status and conditions.

But you can get some idea, anyway...

https://gmane.org

Meanwhile, the news server is:

news.gmane.org

You can connect to it using the normal unencrypted news port, 119, or the 
encrypted snews port, 563. (Assuming you have pan built with gnutls 
support to enable secure connections, just set the security/TLS setting 
in the server properties to use secure connections, and it should 
automatically set the port to 563.)

It's still run by Lars, the guy who originally started the whole thing, 
with a few volunteer admins helping a bit.

The gmane.* hierarchy is the gated mailing-lists, with the pan-users 
mailing list as...

gmane.comp.gnome.apps.pan.user

Particularly since the info that was formerly on the website about 
posting, how to request new lists be added, etc, is now outdated or 
missing, you'll almost certainly want to subscribe as well to

gmane.discuss


That should let you /read/ the groups/lists you want, assuming they're 
carried by gmane.  Posting is rather more complicated.  I used to refer 
people to the gmane website's posting instructions, but that page is 
broken now, so that doesn't work...

Basically, the first time you post to a "group" via gmane, gmane will 
email the address you used to post (which thus must be valid) asking that 
you confirm your intent to post.  Once you confirm, gmane forwards the 
message to the mailing list, and it's upto the mailing list what to do 
with it.

Once gmane conforms an email address for a specific list, gmane will 
forward any new messages it gets from that address for that list.  But 
send to a different list and you'll need to confirm again, for the new 
list.

What the mailing list does with the message once gmane has forwarded it 
is out of gmane's hands and depends on the list.  Some lists are "open" 
and will, possibly after a few automated anti-spam checks, post the 
message to the list.  Others are private, and only accept posts from 
users who have subscribed to that list. (Note that this is "subscribed" 
from the list-serv's perspective, and is entirely independent of whether 
you've "subscribed" to the gmane group in pan or whatever other news 
client you may use.)  Then there are the "moderated-posting" groups, 
which will normally automatically post your message if you are 
subscribed, but will put non-subscriber posts in a moderation queue for a 
human to approve before they're actually posted to the list.  Finally, 
there are some "read-only/announcement" type lists that don't accept 
posts from normal subscribers at all, only a few admins.

Meanwhile, once a message has actually been posted to the list, it 
doesn't matter whether via gmane or directly to the list, gmane gets it 
via gmane's own subscription to the list, and will gateway that post to 
the newsgroup that matches that list.

Again, gmane only actually posts messages it gets from its subscription 
to a list, regardless of how they were posted to the list.  Confirming 
your email address to gmane won't necessarily get it posted to the gmane 
group, it'll just get the message forwarded to the list.  If the list 
rejects it, it won't get posted to gmane either, because gmane only posts 
to the group what it gets from its subscription to the corresponding list.

So posting via gmane is actually a two-step process, all behind the 
scenes once you're setup to do it, but for that first post to a group/
list, requiring first a confirmation for the gmane challenge that you 
actually want it posted, then, depending on the list's own policy, a 
possible wait for moderation or subscription there, as well.

Note that if you do want or have to subscribe to the list, once you're 
using gmane you may want to put your actual list subscription into either 
digest mode, so you only get periodic messages including everything to 
the list for the period instead of each individual message, or vacation 
mode, which lets you post, but won't actually send you the messages via 
mail, useful when you're reading them via gmane anyway.

Meanwhile, there have been a few complaints to gmane.discuss that the 
usual gmane first-post authorization process is broken in some cases, and 
people never get that first-post challenge from gmane to respond to, to 
authorize gmane to forward the message to the list-serv.

For this sort of case it's worth keeping in mind that replying via gmane 
isn't your only choice.  Since gmane is actually gating the posts from a 
mailing list, you can actually reply via mail to the mailing list itself 
as well, bypassing gmane entirely for the reply.  Since you're replying 
via mail to the list itself, the list-serv will apply its normal policy 
and foward the mail to the list if you're a member, or to a moderator if 
it's a moderated list and you're not a member, and gmane won't see the 
message at all unless/until the message is actually posted to the list 
and gmane sees it via its list subscription.

Finally, it's worth noting that for some lists, including the pan list, 
gmane obfuscates email addresses, or indeed, anything that looks like an 
email address (of the form user @ example.com , without the spaces, of 
course).  The obfuscated form will have a domain name of public.gmane.org, 
while the username portion looks like nonsense.  However, to gmane these 
obfuscations are reversable, and while I haven't actually tried it since 
the web side split off, at least before that, if you actually sent 
someone a mail via the obfuscated public.gmane.org address, gmane's mail 
server would, after ensuring it didn't look too spammy and gmane wasn't 
getting a whole bunch of them for other addresses at the same time, 
forward them to the real email address.

So it was, and should be still tho I've not tried it recently, actually 
possible to use your obfuscated @public.gmane.org address as a forwarding 
address to your real address.  Quite cool, actually! Maybe I should try 
it again one of these days to see if it still works.  =:^)

Meanwhile...

I'm not actually sure if gmane still hosts the gwene.* groups too, I've 
not looked since all the changes, but if so, they're similarly gated from 
web feeds (atom or rss) to newsgroups.  However, since those are XML 
based, you'll likely find pan doesn't work well with them.  But 
thunderbird likely will.

> that said, thanks for the warning and the information.. I'll check on
> it, those cache directories are there and I can see the cache settings
> in pan.
> 
> I doubt I will get what I want though as the msg files are scrambled. I
> wanted to start watching movies whilst downloading them.

That's not "scrambled" per se, it's "encoded".  Here's the deal.

The original email format was designed for 7-bit ASCII text only and it 
wasn't, and isn't, possible to send binary files directly.  There's 
actually quite some history behind the three major plus additional minor 
encoding schemes, but the general idea of all of them is to encode 8-bit 
binary into some form that can be transmitted over mail and news paths 
without corrupting the files.

The earliest major format was UUE, Unix-to-Unix-Encoding.  This chose 64 
ASCII characters that could transmit cleanly using 7-bit ASCII, using 
them to encode 6 bits (64=2^6) of the binary for each encoded ASCII 
character.  Since each ASCII character is transmitted using 8 bits, but 
it only encoded 6 bits of the binary plus overhead (line terminations, 
etc), this encoding form is slightly more than 33% larger than the 
original binary file.

That is UUE takes a message a bit over 4 KiB to encode 3 KiB of binary 
file.

But UUE was never entirely standardized, and a new, more standardized 
scheme was eventually chosen, MIME's Base64.  MIME itself stands for 
Multipurpose Internet Mail Extensions, and was a larger set of 
specifications that standardized a bunch of related stuff, including 
headers that specified the mime-type of a message part, the encoding used 
for it (including not only Base64, better for binary, but quoted-
printable, better for mostly text which came thru mostly unchanged, but 
could handle occasional binary characters), ways to specify how non-ASCII 
characters were encoded in otherwise ASCII-only headers so they wouldn't 
break backward compatibility, etc.

But while much more formally standardized, MIME Base64, as the name 
suggests, also used 64 ASCII characters (tho slightly different ones than 
UUE) to encode 6 bits of binary, so it had the same 33%+ encoding 
overhead (with MIME quoted-printable better for mostly text, but much 
much worse for mostly non-text binary).

So Base64 is also 4 KiB encoded for 3 KiB of binary file.


Then, over a decade later, along came yenc, aka yEnc, aka y-encoding.  
This was primarily pushed by binary news users, actually mostly posters 
since they define how they post and downloaders have to use something 
that understands it if they want to do anything with the download, who 
were fed-up with the 33% overhead of UUE and Base64, since by then news 
was /almost/, tho not entirely, 8-bit clean, and yenc could take 
advantage of that, for news only, to be much more efficient.

As a primarily news-poster-enforced choice, yEnc wasn't nearly as 
standardized as MIME/base64, and remains rather less mainstream than the 
old UUE as well, particularly since many primarily mail clients that also 
support news don't support it, and not all mail servers or paths support 
it.

But the one thing yenc *DOES* have going for it is the much better 
encoding efficiency, only ~5% overhead, compared to the 33% overhead of 
the other two major types of encoding!  It does this by using all the 8-
bit ASCII characters it can get away with -- the CRLF sequence is 
reserved as the standard line termination, and there's a few others 
reserved as well, these must be escape-encoded -- and using much longer 
lines, so there's less overhead.  Because it can take advantage to eek 
out slightly better efficiency, yenc is also "shifted" by a few values 
from standard 8-bit ASCII.  But it's only "guaranteed" to work over 
direct news transfers, not when gated to other formats, and not for mail.

But in an age where some posters and many downloaders were still 
connecting using dialup at 56kbps or less down, 33kbps or less up, 5% 
overhead compared to 33% overhead shaved enough time off the uploads and 
downloads that it was WORTH IT, for many.

There were a few other minor advantages as well, such as the fact that 
yenc checksums the encoding, so if it's corrupted, that's known 
immediately.

And within a couple years, no news client could seriously claim good 
binary handling if it didn't handle yenc, because that's what most of the 
binary posters were posting in!


So your "scrambled" is actually encoding, probably yenc, if some of the 
"characters" don't appear as standard ASCII text, or mime/base64 or 
possibly uue, if it's all ASCII.

These days, clients including pan normally transparently encode when 
posting and decode when saving binaries, so a user not familiar with the 
old days may not even be aware of that encoding, but back in the early 
days when it was UUE, posters had to run their binaries thru a separate 
encoder to encode them as 7-bit ASCII, then copy the encoded version into 
their emails as text.  Similarly, downloaders had to feed the raw text 
messages, or even edit them first to remove the wrappers so it was just 
the bare encoded binary, to the separate decoder, which would then output 
the decoded binary which could then be saved as a real binary once again.

And when yEnc first appeared, people who had been used to clients who 
could by then handle mime/base64 and UUE transparently, again found 
themselves having to manually feed the raw messages to their yenc-decoder 
before they could save them.  But it was still worth it to shave an hour 
of every four off their download time, and even if it wasn't, that's what 
the posters were posting in (yenc-power-post was one of the first ones 
that handled large-scale posting in yenc, and it quickly became popular, 
because at 28-33k, uploads were even slower than 48-56k downloads, on 
dialup!), so they had little choice.

Today any self-respecting news binary downloader, including pan, can 
handle yenc, but that doesn't mean you can't still use the separate 
encoder/decoders if need be.

If interested, try uudeview or one of the similar tools.  That should let 
you manually decode the binaries found in the raw cached messages.

Since you're specifically interested in decoding incomplete posts, you'll 
be interested in uudeview's -d/desperate option, which allows just that.  
However, with most file types you'll need at least the first part 
correct, or your viewer/player app probably won't be able to tell what 
codecs it needs to view/play it properly.

For that matter, you should be able to force pan to save an incomplete 
file as well, tho I think it'll try to download the missing parts first 
if they're on the server, and will only save the incomplete file if it 
can't see or get the parts to download.  And as I said, you'll probably 
need the first part, regardless, or it'll save but won't play.

(I haven't done binaries in quite some time now, but when I did, for 
movies I found I needed at least the first part.  For most movie types, 
after that I could skip some parts and play the others I had, with some 
temporary corruption where parts were missing until the next full anchor 
frame, but it would play.)

> On 03/01/18 16:28, Duncan wrote:
>> Pedro posted on Wed, 03 Jan 2018 13:26:35 +1100 as excerpted:
>>

---
[1] Reversed lines:  Courtesy of the "rev" command, part of the util-linux 
package.  See the rev (1) manpage.  Here's the line I used for the second 
one:

echo "Maybe this will be better still, reversed lines." | rev

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman