[Orgmode] Re: Having (too) many files in org-agenda-files

[Sébastien Vauban]

Hi Carsten,

Carsten Dominik wrote:
> On Sep 29, 2010, at 10:32 AM, Sébastien Vauban wrote:
>> Carsten Dominik wrote:
>>> On Sep 28, 2010, at 8:45 PM, Sébastien Vauban wrote:
>>>> Of course, I have many, many files in Org mode. All files I write (or
>>>> touch) in fact.
>>>>
>>>> Of course, I would like to search through my files at some point in time.
>>>> The problem is the load-time of my Emacs, now 221 seconds, coming from 20
>>>> seconds before the heavy use of Org...
>>>
>>> 4 minutes of startup time is entirely unacceptable. And I think you need
>>> to identify what is causing this.
>>>
>>> fontification at display time is standard, I believe.
>>
>> It seems not, from what I see in the Messages buffer. I really don't have
>> the impression of having fiddled with that, really.

>>> You must be doing something strange [...] maybe forcing global
>>> fontification for each file or so.
>>
>> To repeat myself, no to that question. But ispell and flyspell are called
>> for Org files. Maybe there are interactions?
>>
>> [Using org-agenda-text-search-extra-files] reduces my load time from 221
>> seconds down to 92 seconds. Already a huge diff!
>
> Still bad though. I am wondering what is causing the fontification message.
> I do not get this, so it must be something in your setup. You should try to
> find out when this is happening and why.

You've seen, from my reply to Matt, why my *Messages* buffer is more verbose
than yours. It's on purpose, just to be able to figure out more easily what's
going on. Though, I don't have yet...


> Also, you might consider to remove (org-agenda-list) from .emacs. I think it
> is pretty much always a bad idea to put a command like this into your
> startup. Just make it a habit to call it early after starting Emacs.

Honestly, that would not change that much. Having to wait 92 seconds at
startup, or a long time just a few minutes later has the same impact for me.
And: don't try to make me stop using Org ;-))


> It also seems to me you some of the extra packages might be activated
> several times in each file. One possible reason could be that you have put
> the code to turn them on into several hooks like text-mode-hook,
> outline-mode-hook, and org-mode-hook. Turning on org-mode will first run
> text-mode-hook, then outline-mode-hook, then org-mode-hook ..... Maybe you
> are also calling font-lock-fontify-buffer explicitly in one of your hooks?

The only fontification customs that I do are here:

--8<---------------cut here---------------start------------->8---
;; special words
(setq keywords-critical-pattern
      
"\\(BUGS\\|FIXME\\|TODO\\|todo\\|XXX\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
(make-face 'keywords-critical)
(GNUEmacs (set-face-attribute 'keywords-critical nil :foreground "red" 
:background "yellow" :weight 'bold))

(setq keywords-org-critical-pattern
      "\\(BUGS\\|FIXME\\|XXX\\|[^*] 
TODO\\|[Ee][Rr][Rr][Oo][Rr]\\|[Mm][Ii][Ss][Ss][Ii][Nn][Gg]\\|[Ii][Nn][Vv][Aa][Ll][Ii][Dd]\\|[Ff][Aa][Ii][Ll][Ee][Dd]\\|[Cc][Oo][Rr][Rr][Uu][Pp][Tt][Ee][Dd]\\)")
                                  ; smaller subset of keywords for ensuring no 
conflict with Org mode TODO keywords

;; FIXME Highlighting all special keywords but "TODO" in Org mode is already a
;; good step. Though, a nicer integration would be that "TODO" strings in the
;; headings are not touched by this code, and that only "TODO" strings in the
;; text body would be. Don't know (yet) how to do that...
(make-face 'keywords-org-critical)
(GNUEmacs (set-face-attribute 'keywords-org-critical nil :foreground "red" 
:background "yellow" :weight 'bold))

(setq keywords-normal-pattern "\\([Ww][Aa][Rr][Nn][Ii][Nn][Gg]\\)")
(make-face 'keywords-normal)
(GNUEmacs (set-face-attribute 'keywords-normal nil :foreground "magenta2" 
:background "yellow"))

;; set up highlighting of special words for proper selected major modes only
(dolist (mode '(fundamental-mode
                svn-log-view-mode
                text-mode))  ; no interference with Org mode (which derives 
from text-mode)
  (font-lock-add-keywords mode
                          `((,keywords-critical-pattern 1 'keywords-critical 
prepend)
                            (,keywords-normal-pattern 1 'keywords-normal 
prepend))))

;; set up highlighting of special words for Org mode only
(dolist (mode '(org-mode))
  (font-lock-add-keywords mode
                          `((,keywords-org-critical-pattern 1 
'keywords-org-critical prepend)
                            (,keywords-normal-pattern 1 'keywords-normal 
prepend))))

;; add fontification patterns (even in comments) to a selected major mode
;; *and* all major modes derived from it
(defun fontify-keywords ()
  (interactive)
  (font-lock-add-keywords nil
                          `((,keywords-critical-pattern 1 'keywords-critical 
prepend)
                            (,keywords-normal-pattern 1 'keywords-normal 
prepend))))

;; set up highlighting of special words for selected major modes *and* all
;; major modes derived from them
(dolist (hook '(c++-mode-hook c-mode-hook change-log-mode-hook cperl-mode-hook 
css-mode-hook
                emacs-lisp-mode-hook
                html-mode-hook
                java-mode-hook
                latex-mode-hook lisp-mode-hook
                makefile-mode-hook message-mode-hook
                php-mode-hook python-mode-hook
                sh-mode-hook shell-mode-hook ssh-config-mode-hook))
  (add-hook hook 'fontify-keywords))
--8<---------------cut here---------------end--------------->8---

The goal is to highlight words such as FIXME, WARNING, XXX, TODO in as many
files as possible.

For Org, a small custom (of the custom) is done in order not to catch all TODO
words: the TODO words in headlines (i.e., preceded by `* ') are *not*
highlighted.

They are well highlighted when found in other portions of (normal) text.

Other custom:

--8<---------------cut here---------------start------------->8---
;; highlight columns 78 to 80 in some modes
(when (try-require 'column-marker)
  (dolist (hook '(emacs-lisp-mode-hook
                  cperl-mode-hook
                  shell-mode-hook
                  text-mode-hook
                  change-log-mode-hook
                  makefile-mode-hook
                  message-mode-hook
                  texinfo-mode-hook))
    (add-hook hook (lambda ()
                     (interactive)
                     (column-marker-1 78)
                     (column-marker-2 79)
                     (column-marker-3 80))))
--8<---------------cut here---------------end--------------->8---

for getting markers in columns 78 to 80, showing me I'm good for a `M-q'...

I've done multiple tests, now:

- with none of the 2 above sets of customs
- with the 1st custom disabled, and the column-marker one enabled
- with both enabled

Times observed (in seconds):

|      | no highlight | column marker only | all highlights |
|------+--------------+--------------------+----------------|
|      |           63 |                 82 |             91 |
|      |           65 |                 84 |             96 |
|      |           76 |                100 |            130 |
|------+--------------+--------------------+----------------|
| mean |           68 |                 89 |            106 |

(note that times are really not that stable, while I always launched Emacs in
the same conditions, without doing anything else during that time!)

So, the above 2 sets of customs have a real impact on the load time.

But I'm still unsure how I have to change all of the above snippets of code,
in order to keep the same functionality, but without the time delays when
Emacs starts up...


> Hope some of this helps. If not, let me see you full configuration - maybe I
> can spot something else.

My .emacs file is there:

http://www.mygooglest.com/sva/emacs-init-sva.el

Though it's well documented, it's not a tiny one... But very well structured,
so quite easy to find stuff related to each other.

I'm always interested by any comment...

Best regards,
  Seb

-- 
Sébastien Vauban