New undocumented mat-file format!!!!

[John W. Eaton]

On 12-Apr-2007, David Bateman wrote:

| While struggling to reverse engineer matlab's undocumented function
| handle saving/loading code, I noticed the statement below in matlab's
| release notes
| 
| <quote>
| New MAT-File Format Based on HDF5
| 
| In Version 7.3 (R2006b), you can save MAT-files in a format based on
| HDF5. Unlike earlier MAT-file formats, the HDF5-based format is capable
| of saving variables that occupy more than 2 GB of storage, including
| large arrays created on 64-bit systems.
| 
| To save a MAT-file in the HDF5-based format, use the -v7.3 option to the
| MATLAB save function or the "w7.3" mode argument to the C or Fortran
| matOpen function. The default MAT-file format is the same as that in
| Version 7.2 (R2006a).
| </quote>
| 
| This certainly going to turn around and bite us in the future, though as
| its not the default format at the moment we should be ok for a little
| long. We'll certainly be incompatible with their HDF5 format, though at
| this point I can't find any documentation on exactly what this format is.

Isn't there a program that dumps the structure of an HDF5 file so at
least it might not be too hard to figure out what is going on
(compared to an arbitrary proprietary binary format).

At some point after 3.0 I think we should consider

  * implementing a thin wrapper around the HDF5 library

  * writing load/save for the new Matlab HDF5 format using the
    wrapper, so it would be in .m files and easy to modify

  * dropping (at least some of) the old formats we have (we could
    provide a standalone conversion program that could take any of the
    old formats and rewrite them to use the new HDF5 format

| In any case this is just a heads up of an up coming issue.

OK, thanks.

jwe