RE: [bug-mailutils] Large file support in Mailutils

[Alain Magloire]

> -----Original Message-----
> From: address@hidden [mailto:bug-mailutils-
> address@hidden On Behalf Of Sergey Poznyakoff
> Sent: Thursday, December 22, 2005 7:11 AM
> To: address@hidden
> Subject: Re: [bug-mailutils] Large file support in Mailutils
> 
> Hello,
> 
> > First of all I'd like to announce that our custom mail.local based on
> > GNU Mailutils is operational on our production mail servers. Our mail
> > servers have about 200K users and we deliver ~1-1.5 million messages
> > to mailboxes daily.
> 
> Great.
> 

Yes.

> > I compiled both my mda and GNU Mailutils (snapshot
> > 20051102) with -D_FILE_OFFSET_BITS=64. However this by itself was not
> > enough. I investigated the issue and the problem was the use of fseek(3)
> > instead of feeko(3), so I substituted all occurences of fseek() with
> > feeko() in Mailutils sources (found in
> > ./mailbox/file_stream.c,./lib/getpass.c,
> > ./mimeview/mimetypes-gram.c,./mimeview/mimetypes.y). This was enough
> > to fix the issues observed and have large file support.
> [..]
> > So I ask for these changes to be included upstream. Of course I am not
> > sure if this is enough to cover all possible issues involved.
> 
> Of course we are aiming to supporting large files in mailutils. However,
> simply changing fseek to fseeko and ftell to ftello is not enough in
> general. There are more problems involved. Most probably these do not
> affect your installation, however they should be solved before releasing
> the package:
>

Glad to see, it worked for you,  this give us a good hint that we did not
rely too much on platform dependent behavior, thanks!
 
> 1. Using fseeko creates portability problems. It can, however, be solved
>    using appropriate autoconf magic.
>

Agreed, we need to pass the right flags to the compiler, is the OS support
this.  Question is not fseeko() and friends part of Posix now? Or whatever
XOpen standard?

 
> 2. The major problem is the use of off_t in the libmailutils. This data
>    type has different sizes depending on whether the library was
>    compiled with large file support or not. Thus, if one compiles
>    libmailutils with large file support and then links it to the program
>    that is compiled without that support, any mailutils functions taking
>    off_t as an argument will get incorrect data, which will lead, *in
>    the best case*, to a coredump. In the worst case it will trash whatever
>    user files involved, as well.
> 
>    Compiling two variants of libmailutils (with and without large file
>    support) is an awkward solution. Therefore I have introduced
>    (2005-11-15) a new data type `mu_off_t' which is equivalent to 64-bit
>    off_t and have changed all related functions and datatypes
>    accordingly. This makes libmailutils safe to use no matter in what
>    combination it is linked, although this creates a small overhead in
>    the usual 32-bit mode.
>

It sounds good to me.  So basically if I read you right, you proposing to
always use 64 bits i.e. mu_off_t ==> 64 bits.

 
> 3. Most of the libmailutils code formats off_t for printing
>    using "%lu" format directive, which does not work for 64-bit data
>    types. Using "%llu" is impossible due to portability problems. The
>    recent snapshot (mailutils-20051216.tar.gz) solves this problem as
>    well. However, these changes are not committed yet, because I do not
>    quite like the way I implemented them. I am searching for a more
>    elegant solution.
> 

Then the question is: where do we actually need to print/format mu_off_t?
On the top of my head, I do not remember any occurrences, so it may be a
non-issue.  And in the event that we need to print/format is it safe to
downcast it to an off_t? The other possibility is simply to always use our
auxiliary functions like asnprintf() that supports the format and to pass
the buffer down to printf() and all for printing.