Re: [lmi] Compression of MST files

[Greg Chicares]

On 2018-02-23 17:10, Vadim Zeitlin wrote:
> On Fri, 23 Feb 2018 16:50:12 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> Vadim--The MST files (unlike their XSLT cousins) are readable
> GC> enough that an end user might be tempted to modify them in an
> GC> inappropriate way, if we distribute them in source form. If we
> GC> were to compress each MST file with xz and distribute only the
> GC> compressed copies, how easy would it be to use liblzma (which
> GC> we already distribute) inside lmi to decompress them on the
> GC> fly (into RAM, not onto disk)?
> 
>  It shouldn't be very difficult, liblzma is supposed to have a simple to
> use API (which I've never used, but it's described as being similar to
> zlib), but it will require writing some code and depending on another third
> party library and xz tool in the makefiles.

That's okay. In 'install_libxml2_libxslt.make' we already build
xz-5.2.3, producing liblzma and many xz* utilities, and building
and installing libraries is usually the hardest chunk of makefile
work.

>  We could use a compressed tarball instead (which is presumably better than
> ZIP as slightly more difficult to modify for MSW users) and then use

Yes, 'zip' is automatically decoded by msw; yet what we really
seek here is not compression, but obfuscation.

> wxArchiveInputStream to extract it into memory to do it in a simpler way at
> the price of worse compression that we probably don't really worry about.

We'll still be doing compression manually for distribution, so
it would be just a little easier for us to use only one archiver
(we already build libxml2 with '--with-lzma' so that we can
compress (obfuscate) xml product files with xz).

Here's an idea: why not add a '--with-lzma=/path/to/lzma' option
to wx and use it for wx streams? If that's not too hard, then it
would certainly be ready by the time we retire XSLT and promote
MST to production.

>  But I'd also like to note that IMO XSLT files were not that difficult to
> modify, especially if it's just to change some text, and if nobody has ever
> done it during all the years they were used, it might be a bit premature to
> worry about people modifying MST files. Maybe we could just checksum them
> and give a warning/error if the actual files don't match?

Yes, we were just discussing that internally this morning. We
really don't know whether anyone has ever modified any XSLT
files when they shouldn't. It's hard to modify them to add a
new behavior, but it's easy to alter one of the many hardcoded
strings they contain, so ideally we would have obfuscated them
long ago. With MST, we're entering a new world where we can do
what we really want from day one.