[Pan-devel] Re: Ya know Pan2 is really single threaded, right?

[Duncan]

"Calin A. Culianu" <address@hidden> posted
address@hidden, excerpted below, on 
Sat, 17 Mar 2007 20:44:27 -0400:

> I have been reading the code and trying to add a worker thread pool
> model to the code.
> 
> I can see now that it introduces a lot of complexity and potential bugs
> to the code -- already the code assumes that everything runs in 1
> thread.
> 
> But I will do my best to aviod race conditions by locking shared data as
> best I can, and to only introduce threading where it will actually
> benefit the user (better network utilization, not stalling on uudecode,
> etc).

FWIW, pan has used at least three threading models since I discovered it 
back with 0.11.x (the old GNOME 1.x code).  I'm not sure when it went 
multi-threaded in the first place, but at one point anyway, pan was I 
think trying to handle all the downloading in multiple threads on its 
own.  Then Charles switched to using gnet for the network handling, and 
it's apparently reentrant on its own (the non-coder is showing here, I'm 
vague on the details), thus sparing him worrying so much about the 
threading details on the networking side -- he only had to worry about 
the local stuff.  That worked quite well (in terms of keeping up the 
network utilization).

I'm not sure what lead him to dump gnet in the rewrite, unless it was the 
C to C++ jump, or simply trying to keep dependencies down, but from my 
read between the lines, he judged the maintenance costs of trying to keep 
threaded code bug-free without the library more hassle than it was worth.

>From what I've observed and from comments yours and others, it's the 
decoding and saving that's the problem at this point, not so much the 
network stuff, except due to the decoding and saving bottleneck stalling 
things long enough to fill the buffer and therefore stall the download as 
well.  Therefore, if it were me, I'd try multi-threading that only, 
first, creating a pool of decode-and-save threads and having the 
otherwise single-threaded pan hand off to them.  Then possibly put the UI 
in a (single) separate thread to keep it more responsive.  If I'm reading 
Charles correctly, I think his thought is that the networking isn't the 
issue, and as such he doesn't need to multithread it, and can therefore 
avoid gnet, his previous partial solution, as a dependency.

I'm running pan on a dual Opteron here, 8 gigs of memory, and my news 
volumes on 4-way SATA RAID-6 (so basically two-way striped, the bus being 
sufficient, the disk I/O shouldn't be a bottleneck), and more and more 
folks are getting dual-core and will soon be going quad core.  What's 
frustrating here is seeing the network stalling due to either CPU or I/O 
wait times, on a single thread, when I've a second CPU idling and all 
those local memory and RAID resources.  At significantly less than 10 
Mbps, the network should be the bottleneck on a setup like that.  I 
assume you are seeing some of the same issues, but being a coder with 
multithreading experience, are even more frustrated than I am with it.

> I have experience with multithreaded coded so hopefully I can do this
> somewhat elegantly and in a bug-free manner.
> 
> Likely he didn't make the code multithreaded because his initial pass
> was to just get Pan2 working, and perhaps he intended to go in later and
> added multithreading?

I don't think that was the issue, as this was a rewrite in part to get 
things right from the beginning.  Thus, if he thought multithreading was 
necessary to get it right, I believe he'd have written it with that in 
mind, but you say the assumptions are single thread, so it's difficult to 
believe he found multithreading a necessary or realistic likelyhood, or 
he would have made that sort of assumption from the beginning, even if 
the initial implementation strictly serialized it.  To do otherwise would 
have nullified one of the big reasons for the rewrite, to get it right 
from the beginning, this time, knowing what he knows now about what works 
and what doesn't.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman