Re: New undocumented mat-file format!!!!

[David Bateman]

Christopher Hulbert wrote:
> On 4/12/07, David Bateman <address@hidden> wrote:
>> John W. Eaton wrote:
>> > On 12-Apr-2007, David Bateman wrote:
>> >
>> > | While struggling to reverse engineer matlab's undocumented function
>> > | handle saving/loading code, I noticed the statement below in
>> matlab's
>> > | release notes
>> > |
>> > | <quote>
>> > | New MAT-File Format Based on HDF5
>> > |
>> > | In Version 7.3 (R2006b), you can save MAT-files in a format based on
>> > | HDF5. Unlike earlier MAT-file formats, the HDF5-based format is
>> capable
>> > | of saving variables that occupy more than 2 GB of storage, including
>> > | large arrays created on 64-bit systems.
>> > |
>> > | To save a MAT-file in the HDF5-based format, use the -v7.3 option
>> to the
>> > | MATLAB save function or the "w7.3" mode argument to the C or Fortran
>> > | matOpen function. The default MAT-file format is the same as that in
>> > | Version 7.2 (R2006a).
>> > | </quote>
>> > |
>> > | This certainly going to turn around and bite us in the future,
>> though as
>> > | its not the default format at the moment we should be ok for a
>> little
>> > | long. We'll certainly be incompatible with their HDF5 format,
>> though at
>> > | this point I can't find any documentation on exactly what this
>> format is.
>> >
>> > Isn't there a program that dumps the structure of an HDF5 file so at
>> > least it might not be too hard to figure out what is going on
>> > (compared to an arbitrary proprietary binary format).
>> >
>> Yes h5dump. However, if matlab did the same thing for the hdf5 format as
>> they did with their compressed file format, then I fear it will be
>> encapsulated in a standard mat-file, this might cause issues using a
>> dumping utility.
>
> I've started working (slowly) on the HDF 5 format for Matlab's 7.3
> save option for my own MAT file library. The HDF 5 MAT files make use
> of a user data section at the beginning of the file that writes the
> standard MAT file header into this area. After that it's pretty much
> HDF 5 as long as you follow what they do. In my opinion, it's a mess.
> Sometimes they write structure fields as references, sometimes they do
> not. It makes it a pain to make sure you get the basics correct.
> However, I think their move to HDF 5 is a good one.
>
> Chris

% h5unjam -o test.mat
MATLAB 7.0 MAT-file, Platform: GLNX86, Created on: Thu Apr 12 18:30:16
2007 HDF5 schema 0.05 .                      IM

So the header is slightly different in that in that after the date the
file includes "HDF5 schema 0.05", and then standard HDF5 files after. I
haven't looked far enough to see the issue you mention with the use of
HDF5 groups yet.

D.



-- 
David Bateman                                address@hidden
Motorola Labs - Paris                        +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 6 72 01 06 33 (Mob) 
91193 Gif-Sur-Yvette FRANCE                  +33 1 69 35 77 01 (Fax) 

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary