Re: [Pan-users] article cache size

[Duncan]

David Chmelik posted on Fri, 8 Mar 2024 18:24:47 -0800 as excerpted:

> On 3/8/24 5:49 PM, Duncan wrote:
>> David Chmelik posted on Fri, 8 Mar 2024 05:19:17 -0000 (UTC) as
>> excerpted:
>>
>>> What [cache] size do you recommend if I currently use 1,500+
>>> newsgroups, and some are binary but dead, so let's say all
>>> plain-text, but some are high- traffic like the Linux kernel
>>> listserv on gmane?

>> At a guess, I'd say start with a gig.
>>
>> If you want to be extra safe or see messages you know you downloaded
>> disappearing, double that to 2 GiB

> Thanks!  I did read most your reply but wasn't aware how cache worked.
> If I could archive newsgroups I've read back to when I started in 1996
> or even to Usenet's beginning, in case I want to read old threads, I'd
> do it, but most likely just want headers...

For gmane it is an archive (a mailing-list archive, gated to and served by 
a news server), so they should actually have their messages back to when 
they setup, shortly before 2002 (which was my first message to the pan 
list), for lists they subscribed to that early, of course.

But if you do consider grabbing entire groups worth of old messages from 
gmane, do either ask first in gmane.discuss (they may may be able to 
arrange an archive file download instead), or do it only a bit at a time.  
Because that's a bunch of stress to put what is after all a free server 
under, trying to do the entirety of a bunch of groups at once, and I've 
read that in really bad cases they've banned IP addresses and/or users as 
a result.

Meanwhile, FWIW gmane's actually the only active "groups" I read.  But I 
have several of my ISP's own discussion newsgroups archived and still 
available in pan from that era as well (until they first quit hosting 
their own news... and eventually quit outsourcing from a commercial 
provider as well).

That archive's nice to have and now irreplaceable, AFAIK, so I keep it 
backed up! =:^)

Tho I should mention that with unexpiring cache if your storage is still 
spinning rust you *WILL* eventually notice pan's initial after-boot 
startup time getting longer... and longer...  Defragging (or backing up, 
testing the backup, deleting the working copy, and restoring from backup, 
effectively a manual defrag) will help, and setting pan to start at system 
startup, or setting up a startup job to cat everything in the article 
cache to /dev/null, thus loading it into the system's file cache so when 
you do start pan it's already cached, is something I also did for awhile.  
(Pan's initial startup time was 10 minutes plus and increasing before I 
set that up.)

With ssds, even back at the SATA-III level (before M2 direct-access SSDs 
were a thing, my system's actually SATA-III era old tho I did upgrade from 
spinning rust to SSD...[1]), the problem's MUCH diminished, tho an 
occasional defrag does still help.  With modern M2-attached SSDs I'd guess 
it's not noticeable regardless.

Meanwhile, for non-gmane public groups, many commercial providers actually 
go back years on many groups (even binary groups but perhaps for that big 
Netherlands ISP group, I forgot the name but back a few years ago anyway 
it was the the most heavily trafficked binary group by FAR, with traffic 
exceeding that of the rest of the binary groups put together, 
apparently... because of the number of TV series and the like posted to it 
because for ordinary people not profiting off it there the penalty is a 
relatively limited fine not potentially multiple years' worth of 
income!).  But of course for binary groups especially that's well beyond 
the Eternal-September level, and will cost some real money, download time, 
and archive-space.

(FWIW I have an unexpiring 1 TB block account through some provider I'd 
actually have to look up in my binary pan instance to remember at this 
point as I don't regularly use it.  They have years of retention.  But 
while that would presumably do for text group archive downloading, for 
binary archive downloading I imagine an unlimited-download monthly account 
would be more cost-effective.)

> I don't think I really need to archive messages once I read them, though
> it'd be nice if I could access my posting history back to the '0s or
> 1990s, even if it was different email addresses...

Not sure about pre-2k for binary groups especially, because disk space 
only really started getting cheap enough for the providers to effectively 
archive them sometime after 2k, but for text groups I believe some of the 
commercial providers do go back possibly that far.  So you might be in 
luck if it's worth the subscription cost for you.  And for text groups 
even for archiving a block account is actually feasible, and might be the 
way to go.

And of course for gmane back to 2001 or so I think, or whenever they 
started archiving the list in question (for some of them they actually get 
the pre-gmane-subscription archives from the list host, if the list host 
is amenable of course, so some might actually go back to the '90s, not 
actually sure).

> in fact I selected
> 'clear cache on exit' in pan, because I thought cache had to do with
> just what was used in a session... now that I read your explanation,

Oops!  Well, you know now and can fix it! =:^)  (Plus, at the default size 
it really is a session cache, at least for binaries altho it seems you 
don't do them much.)

---
[1] OT but on the topic of upgrades: I actually have the money set aside 
to upgrade but the one I want is an AMD threadripper too expensive to ship 
when I'm not home all day to accept it.  So I gotta either setup a 
shipping address and link it to the CC plus notify the bank so it doesn't 
trigger all sorts of alarms when I do the purchase, or maybe, take a short 
vacation this summer to pick it up and hopefully get a tour of the 
system76 HQ in Colorado, since I'm reasonably close in Phoenix, and a mid-
summer CO trip would be nice break from the Phoenix heat, particularly if 
I stopped in the mountains on my way there/back/both.  Yes a 
threadripper's expensive but when it's effectively my only computer and 
it's upgraded once in a decade...

Tho right now I'm kind of enjoying just having that sort of money clear in 
my bank account... and I'm not absolutely sure I want to actually do that 
tradeoff just yet.  But I gotta do /something/, even if not that big, 
/eventually/, as this decade-old thing is sslloowwww on Gentoo trying to 
build modern say firefox/chromium (all nite just for the one package).
(Plus, my last machine, one of the original 3-digit AMD Opteron dual-
socket setups, only lasted eight years and this one's like 11 now, so how 
much more do I dare push it?  Tho that one died due to one of the infamous 
bad capacitors of the era so 8 years wasn't actually half bad.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman