Thank you for your response. It was helpful to know that this isn't going
to be an easy fix. We're already using regexp-opt, which is supposed to
optimize and shrink the regex.
In that case I don't think there's much you can do (you may be able to
tweak regexp-opt to reduce the compiled regexp size, but you'll just get
a few percent further, which probably isn't of any use to you).
I can anticipate the regexes getting much much bigger, and I'm keen to
avoid having to dig into the guts of the regexp C code, so I think we're
going to figure out a way to do our regexing outside of Emacs.
Looks like your best/only option. Of course you may also be able to do it
all in Emacs by not using regexps. E.g. if your code looks anything like
(re-search-forward (concat "foo" (regexp-opt mighty-big-list) "bar"))
you may be able to use
(while (and (re-search-forward (concat "foo\(.*\)bar"))
(not (member (match-string 1) mighty-big-list))))
And of course, use a hash-table rather than a list.
Stefan