gnats/665: Re: gnats/665: queue-pr -r doesn't process the incoming messages in the right order

[bug-gnats]

The following reply was made to PR gnats/665; it has been noted by GNATS.

From: Chad Walstrom <address@hidden>
To: address@hidden
Cc: Stephane Chazelas <address@hidden>
Subject: Re: gnats/665: queue-pr -r doesn't process the incoming messages in 
the right order 
Date: Wed, 09 Aug 2006 12:14:50 -0500

 address@hidden  wrote:
 >  I guess you refer to the arbitrary "2000" limit on the size of the
 >  incoming queue. Is there any other "magic number" you're thinking
 >  of?
 
 That's one, and the other is the character array you allocated for the
 file name.  32 characters doesn't necessarily translate to a 32 bit
 number.  Potentially, that would be your overflow point with the
 number of files allowed in the queue if file names are going to be
 sequential.  A more scalable solution would be to use a Maildir style
 naming convention, where date-time when the file was received in
 combination with the host name and and a delivery identifier is used.
 
 See: http://cr.yp.to/proto/maildir.html
 
 This is actually a very good method for queues received asynchronously
 on different machines.  For example, if someone is using gnats on an
 NFS share, this naming style would allow it to happen without locking
 and without danger of name collision.
 
 Incorporating this into a queue processing mechanism would allow you
 to use the "unique:info" pair to rename the incoming queue files to an
 info flag such as "P" for processing, "E" for error.  queue-pr could
 be told to simply ignore both and only process those files w/o the
 info tag appended to it.  That way,  you wouldn't have to worry about
 two queue-pr's running at the same time...  Unless there is
 database-locking semantics to look out for.  I'll dig into that more.
 
 >  That's one I was not very confortable with. That limit can be
 >  removed. The idea was to prevent problems like: an admin not being
 >  aware that the "queue-pr -r" is no longer scheduled (or that there
 >  are some files (like core file) littering the "processing"
 >  directory (though he will be sent emails for every queue-pr -r)),
 
 Right now, the admin is sent messages that something bad happened.
 You could just as easily set up a cron job to scan the directory with
 "find" or "ls" on a periodic basis.
 
 >  but that's true that the scheduling may be temporarily disabled
 >  (because for instance the machine processing the queue is being
 >  serviced) and in that case you would want to be able to queue every
 >  message until the processing machine recovers).
 >  
 >  or like some vicious circles where the processing of one msg
 >  generates 2 new files which in turn generate 4, 8...
 
 Agreed.
 
 >  The idea was to make sure those kind of problems were not left
 >  unnoticed, but you're right that they will get noticed when the
 >  file system or the directory is full or when it takes so much time
 >  to add an entry to a directory that several instances of queue-pr
 >  -q run at the same time.
 
 Right.  I'm going to try to dig into it a bit more today if I can.
 -- 
 Chad Walstrom <address@hidden>           http://www.wookimus.net/
            assert(expired(knowledge)); /* core dump */