Re: [Pan-users] Re: Inefficiencies in Pan threading?

[Calin A. Culianu]

On Sun, 14 Sep 2003, Duncan wrote:

> Calin A. Culianu posted
> <address@hidden>,
> excerpted below,  on Sun, 14 Sep 2003 01:21:25 -0400:
>
> > I am still a little puzzled by the sporadic delays and stalls I get while
> > downloading from my news server.  I previously thought this was dead time
> > Pan was spending decoding downloaded chunks, but perhaps it is caused by
> > something else outside of Pan (such as the news server itself).  From your
> > description it sounds like Pan may not be the source of the delay -- it is
> > after all trying to minimize "dead time" by interleaving downloading with
> > decoding.. is this a correct assessment?
> >
> > Would you have any insights into how I can prevent the stalling I am
> > seeing?  Or should I look outside of Pan altogether to solve the problem
> > (like maybe my news server is being slow sporadically..)?
>
> As you mention, that addresses the threading issues, but doesn't do
> anything for the idle time issues.  When I've seen that, which isn't
> often, it has been due to stale "dead" connections that the server still
> thinks I'm using, because they didn't get closed properly, for whatever
> reason.  Three questions..
>
> Are you sure your news provider allows you four connections to the server?
> Obviously, if not, and you have PAN configured for four, it's going to
> cause problems when PAN keeps trying to get more connections than the
> server will allow.

They claim four connections is the limit.  I have manually tested this
from a telnet to port 119 on the news server.. and up to 4 connections are
ok, the 5th gets dropped right away, so I am sure 4 is the correct limit..
:)

>
> What are your caps like, both on your overall pipe, and on each
> connection?  If you have dialup or a low bandwidth DSL, and the
> connection caps are high enough, only one or two full connections may fill
> your available bandwidth, meaning you are not using your available
> bandwidth to best efficiency in the best case, and possibly causing
> connections to starve and be dropped as idle.  If that's happening, and
> the pipe is already full, the connection termination may not be able to
> get thru properly and the server may think the old connections are still
> active.  If these stale connections fill up your quota, then PAN won't
> have any choice but to keep waiting and trying, until one of them times
> out, and it can connect again.
>

I have cable modem.  The caps aren't easily discernible from the marketing
literature that my ISP gives me, but it seems like I can expect about 200
kB/s down and about 40kB/s upstream.. which is rather fast.  The
connection quality to most places in the United States is _really_ good.
I rarely get dropped packets and lag is very low usually.  So between
myself and most major internet backbones I can say for the most part the
connection is quite good... but I haven't investigated this fully.

> In either of the above cases, dropping your number of connections to
> something more in line with the resources available may actually improve
> your throughput, eliminating most of the idle time.  Since PAN tends to be
> fairly efficient using the connections available, it's possible you need
> fewer connections to fill your pipe to capacity than might have been
> needed with another news client.
>
> What is your connection quality?  Do you have any problems with dropped
> packets?  Does your ping time to the general internet or that server in
> particular have large swings?  (Try installing the program "MTR", "Matt's
> TraceRoute", if you don't have it installed already, and see what it says

I am unaware of any volatility and/or lack of reliability to my news
server.  It seems like I get less than 60 millisecond ping times to that
server consistently.. but maybe I should play around with MTR as you
mention... since it's possible that during the stalled times I may be
getting temporary hiccups in the network and/or dropped packets.

> the connection is like.  It's an invaluable tool for checking out route
> stability.)  If the connection quality isn't the best, it could be
> connections dieing on the vine, so to speak, due to ACKs being dropped
> somewhere along the line.  This will exacerbate the problems above if you

Hmm.  As far as I know TCP waits a _long_ time resending those ACKs.  It's
TCP/IP stack-dependent (and it's configurable on most good OSes like
Linux).  It's unlikely an unstable connection would break outright, but it
is possible to get very long stalls on an unstable connection...

> have them as well, as some of the connection maintenance messages just
> won't make it thru, causing even MORE problems.  This can be quite common
> on dialup, as its reliance on analog isn't an ideal solution anyway, and

Yes, I know what you mean!  On dialup, the link layer itself goes nuts if
it gets too much line noise.  It has to go through a quite long delay as
it renegotiates and resynchronizes with the remote modem.  If you get too
much line noise you can experience delays for tens of seconds at a time
and quite often as well!

> even the slow connections dialup has are made worse by the unreliability
> in far to many cases.  On broadband, assuming you aren't having
> the always possible physical connection issues, the problem is often
> overloaded routers or pipe segments somewhere between you and the server
> in question.  This can be at the gateway hop, others under control of your
> ISP, or further out on the internet backbone, if your news provider isn't
> located on all-private network from you.  Unfortunately, lowering the
> connection count or pretty much anything else you may do doesn't have a
> lot of effect here, except for switching providers, either ISP or NSP,
> depending on where the problem is, to something more reliable.

Yes this is a valid point.  I should really investigate the reliability to
my NSP in a more systematic way.  I will get MTR right now and play around
with it...

>
> BTW, this belongs on the user list, not the devel list, as it's not a
> devel issue.  It was possible it might BECOME a development issue, from
> your standpoint, if PAN indeed DID have bad threading or whatever, but
> unless you KNOW that's the case and have a proposed solution, the user
> list is the appropriate place for the post.  Not that it matters a /whole/
> lot as the developers are normally active in both, but it /does/ help to
> keep things more organized if posts are kept on topic for the list in
> question, meaning user for anything that isn't known to be specifically
> development related.  Just a friendly reminder for next time, OK?

Yeah you're right.  I wasn't sure how active either lists are so I was
hoping to get as many people as possible with my message.. :)  But you are
of course right this isn't really a development topic but an end-user
topic... I will keep that in mind for next time.


So basically you are saying that Pan is pretty good about utilizing as
much of the network resources as it can, and that it is more likely that I
have a connection problem to my news server.  I am going to investigate
this further.  On some levels I hope you're wrong so that if I do find
some bug in Pan, maybe I can get involved and fix it myself.  However Pan
seems rock solid and very well written so maybe it isn't to blame.. :)


Thank you very much for your discussion and insights, Duncan!

-Calin