[lmi] Compressed '.mst' files [Was: May 2018 wx upgrade]

[Greg Chicares]

On 2018-04-30 18:32, Vadim Zeitlin wrote:
> On Mon, 30 Apr 2018 17:52:03 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> As for liblzma support, wasn't some small bit of remaining work needed,
> GC> in lmi itself,
> 
>  Yes, lmi hasn't been modified at all yet. But this shouldn't require any
> changes in wxWidgets (hopefully).
> 
> GC> to bring all the pieces together so that xz-compressed MST files would
> GC> be decompressed automatically? (That doesn't need to be done just yet;
> GC> I'm just asking so that we don't forget.)
> 
>  It's in my TODO list but maybe we should use Savannah issue tracker
> instead...

Looks like it doesn't work yet:

  /opt/lmi/data[0]$xz cover.mst          
  /opt/lmi/data[0]$ls cover.mst.xz 
  cover.mst.xz

results in messagebox:
  Template file "cover.mst" not found.

and renaming:
  /opt/lmi/data[0]$mv cover.mst.xz cover.mst
produces a PDF file with no cover page. (I guess the unparseable
'xz' contents are gracefully ignored.)

> GC> Oh, and will the wx-xmlwrapp-libxml2 chain of tools recognize xz
> GC> compression without any identifying extension?
> 
>  I'm not sure, looking at libxml2 code it looks like it doesn't rely on the
> file extension at all, but I don't quite understand what exactly does it
> rely on to detect the input format. If I do read its code correctly, it
> seems to apply LZMA decompression by default for all input, so we shouldn't
> have any problems with this, but I should probably test it.
> 
>  In the worst case, we could always do the same thing as we plan to do for
> the .mst.xz files, i.e. load and decompress them ourselves (which is simple
> now using the LZMA decompression support in wx) and then use xmlwrapp and
> libxml2 with plain XML data. But, again, we probably won't have to do this
> and, in any case, this is not really related to wx.

Just to restate the objective: we don't actually need to make MST
files smaller; instead, we just want to veil them in the obscurity
that is a side effect of compression. The problem is that the new
implementation is too easy to understand, and end users might be
tempted to modify the templates. We don't need strong encryption;
it's enough to interpose an obstacle so that modification requires
real work, so no one can claim "I was just examining the contents in
'notepad' and some changes must have gotten saved by accident".
For this purpose, it's best to have no '.xz' suffix, because, well,
"that would be telling".

Our current plan is to release
 - today's HEAD (more or less) at the end of October--so that any
   potential end-user conflicts due to recent library updates will
   be found--without any MST files; and then
 - at the end of November, to remove the old PDF implementation and
   release its replacement into production
so the ability to read xz-compressed MST files won't be needed
this month--though we'd be glad to have it earlier for testing,
e.g., in case another wx upgrade is needed to support it.