`make-overlay' very slow (was: hiding lines)

[Werner LEMBERG]

>>> I've now come up with the code as shown in the attachment.  It
>>> works fine, however, I would like to not have empty lines; in
>>> other words, invisible text should not take any space in the
>>> buffer.  Does Emacs supports such a `compressed invisibility'?
>>
>> 
>> Make the newline invisible.
> 
> Thanks!  This perhaps deserves a notice in the documentation.

I've now tried the code with my real-world table (containing
approx. 420000 lines):

  git://repo.or.cz/srv/git/wortliste.git

and I found out that it is, alas, unusable.  It starts with about 1000
replacements per second on my GNU/Linux box (using simply `-' as the
regexp), but soon the speed decreases: After approx. 10000
replacements it only can handle 100 replacements per second -- the
attached code now contains a small counter to show the progress.  If
you comment out the call to `make-overlay', the code really flies...

Am I doing something wrong?


    Werner

(defun make-lines-invisible (regexp &optional arg)
  "Make all lines matching a regexp invisible and intangible.
With a prefix arg, make it visible again.  It is not necessary
that REGEXP matches the whole line; if a hit is found, the
affected line gets automatically selected.

This command affects the whole buffer."
  (interactive "MRegexp: \nP")
  (let (ov
        ovs
        count)
    (cond
     ((equal arg '(4))
      (setq ovs (overlays-in (point-min) (point-max)))
      (mapc (lambda (o)
              (if (overlay-get o 'make-lines-invisible)
                  (delete-overlay o)))
            ovs))
     (t
      (save-excursion
        (goto-char (point-min))
        (setq count 0)
        (while (re-search-forward regexp nil t)
          (setq count (1+ count))
          (if (= (% count 100) 0)
              (message "%d" count))
          (setq ov (make-overlay (line-beginning-position)
                                 (1+ (line-end-position))))
;         (overlay-put ov 'make-lines-invisible t)
;         (overlay-put ov 'invisible t)
;         (overlay-put ov 'intangible t)
          (goto-char (line-end-position))))))))

(global-set-key "\C-cz" 'make-lines-invisible)