bug#15225: 24.3.50; todo-mode: Some bugs and suggestions

[Stephen Berman]

On Fri, 30 Aug 2013 23:48:28 +0530 Kanthimathi R <rkanthimathi@hotmail.co.in> 
wrote:

> Quick feedback: 
> --------------
>
> The interface for todo-mode looks good.  Info documentation is good.
> The starting few nodes are a bit too dense while the later nodes seem
> like a lucid read.

Thanks for testing Todo mode and providing extensive feedback.  Sorry
for not replying sooner; I was offline for a couple of weeks.  I've
tried to respond to all your points below.  In some cases I need more
details to be sure what you're suggesting, and it may be a while before
I can implement all suggestions I agree with.  The clear bugs I'll try
to fix sooner.

> Recommendation: 
> ---------------
>
> I would recommend it for simple todo-lists.  Once the entries become
> more, then categorization of entries looks like a rocket science.  (Or I 
> should say, new users should be sufficiently trained.)

It's not clear to me what you're complaining about here.  Can you be
more specific?

> Verdict: 
> --------
>
> Too many moving parts and the module needs to stabilize a bit.  I
> INVARIABLY run in to problems.
>
> Giving up:
> ---------
>
> 1. For example, as I was exercising todo-mode for preparing this bug
>    report, the file ran in to an inconsistent state - A category with no
>    todo or done items was reporting 6 entries in `F c' table.  

This would certainly be a bug, but I can do nothing about it without a
reproducible recipe.

>    I wondered whether deleting the categories sexp and C-x C-q ing would
>    repair the todo file.  

Before you did this you had typed `q' to exit Todo Categories mode and
return to Todo mode, where you typed `F e' to enter Todo Edit mode,
right?  Note that the user manual urges caution when using `F e'.  By
deleting the categories sexp you corrupted the internal file format,
which makes little or nothing work afterwards.  I don't think
todo-mode.el should be required to recover from corruption due to user
manipulation of the internal format, but perhaps the user manual should
explicitly warn against doing that.

What I assume happened is that you deleted the categories sexp, but left
the first line empty.  If you had deleted the entire first line, it
should have been possible to repair the sexp by typing `M-x
todo-repair-categories-sexp RET', but with an empty first line, you get
an "End of file during parsing" error when read is called on the empty
string.  I could add code to prevent this particular error, but since it
arises as a result of user manipulation of the internal format, I think
the best advice is "Don't do that."

>                           Unfortunately, that doesn't help.  So I am
>    attaching a "broken" todo file.

To fix this file, first delete the empty first line.  Then setq
todo-current-todo-file to the absolute file name of the file.  (This
wouldn't be necessary if the file were already the current todo file,
but since we're now attempting to visit the file in Todo mode, it is not
yet current, and the corruption prevents it from becoming so.)  Now
invoking todo-repair-categories-sexp will make the file recognizable by
Todo mode again.

> To add to the entries in the attached file,
>
> 1. Undo support while in todo-mode.

Please don't.  Just fix the file as described and use the Todo mode
commands to add new items and otherwise manipulate the file.

> 2. The following markers used in the todo file
>
>       --==-- 
>       ==--==
>       
>   are they consistent with the outline-mode.  If not, some consideration
>   should be given so that they are compatible with outline-regexps.

These are part of the internal format and unless you use `F e', they are
not exposed, so as far as the user of Todo mode is concerned, they are
not available to manipulate, whether by Outline mode or any other way.
Why do you want to make them outline-aware?

> Additional comment:
> ------------------
>
> As some one who is familiar with (but doesn't use) Orgmode's data model,
> I can share some opinions, if permitted.

I'd be interested, since I know very little about Org (though your
"samurai sword" post to emacs-devel gave some hints).

> With no further ado, here comes the attachment.  Repairing the attached
> file is reader's responsibility.
>
> 
> 
> --==-- todo-mode
> [Aug 29, 2013] `C m' MUST create new files on demand
>       
>       1. Visit the odt category.
>       2. Type `C m' on the first entry
>       3. When prompted for a file-name, give a non-existing file name.
>       4. Note that completion insists on a MUST-MATCH.
>       5. I was expecting that the new file will be created and the entry 
> moved to that category.

I guess this is a reasonable expectation.  My use-case for `C m' is that
I have a main todo file containing unrelated categories I add as the
need arises, and I have more specific todo files, and occasionally I
find that a category in the main file fits in better with one of the
more specific files, so I move it to that.  So my assumption was that
users would only want to move categories between existing files.  But I
guess it makes sense to be able to create the file on moving the
category and I'll make that change.

> [Aug 28, 2013]  `i i' should append or prepend items by default
>       
>       GTD says separate out collection and processing.
>       
>       Insertion of an item is a "collection" activity.  Setting a
>       priority is a "processing" activity.  So I recommend that `i i' add
>       an item to the top or the bottom of the file.  Let the user move
>       the raise or lower the items at a later point in time during the
>       "processing" phase.

I'm reluctant to eliminate prioritization from `i i' (and hence from
most of the other item insertion commands).  This has always been a
feature of todo-mode.el, implicit in the command's name, as the manual
points out: insertion does not entail appending or prepending.  But I
guess it's reasonable to have an option for prioritized item insertion
to make typing RET on being prompted for a priority default to first or
last item.  Yet note there's also `i h', which inserts a new item with
the priority of the item at point, and since navigating to a category
always put point on the first item, subsequently calling `i h' amounts
to making the new item the first one (as does calling `i h' outside of
Todo mode or when point is in the done items section, as a special case
that in effect takes up your concern).
        
> [Aug 30, 2013] When I `C m', most often I want to stay in the same category
>       
>       The current behaviour is to move to the new category.  I
>       essentially need a pop or a prefix binding.

Making this possible with a prefix argument seems like a good idea.
        
> [Aug 30, 2013] A key binding for editing just the header line (say e h)

That functionality is bound to `e d t'.  (I had at one point bound it to
`e h' but since the other commands with the prefix `e d' are for editing
parts of the date string, using it for the whole date/time header is
more consistent.)

> ==--== DONE 
> 
> --==-- todo-mode/bugs
> [Aug 29, 2013] With desktop mode on and todo-mode leaves a stacktrace
>       
>       Strip your .emacs as below.
>       
>        (custom-set-variables
>         ;; custom-set-variables was added by Custom.
>         ;; If you edit it by hand, you could mess it up, so be careful.
>         ;; Your init file should contain only one such instance.
>         ;; If there is more than one, they won't work right.
>         '(calendar-view-diary-initially-flag t)
>         '(desktop-base-file-name ".emacs-desktop.junk")
>         '(desktop-path (quote ("~")))
>         '(desktop-save-mode t)
>         '(diary-file "~/.emacs.d/todo/emacs.todo")
>         '(diary-number-of-entries 30)
>         '(savehist-mode t)
>         '(todo-wrap-lines t)) 
>       
>       emacs
>       M-x todo-show
>       Make sure that todo buffer shows up fine
>       C-x C-c and save the desktop file
>       
>       Re-load emacs
>       M-x toggle-debug-on-error       
>       M-x todo-show
>       
>       See following stacktrace.       
>       
>       Debugger entered--Lisp error: (error "Category nil is missing 
> todo-category-done string")
>   signal(error ("Category nil is missing todo-category-done string"))
>   error("Category %s is missing todo-category-done string" nil)
>   todo-category-select()
>   todo-show(nil 1)
>   call-interactively(todo-show record nil)
>   command-execute(todo-show record)
>   execute-extended-command(nil "todo-show")
>   call-interactively(execute-extended-command nil nil)
>   command-execute(execute-extended-command)

This error actually has nothing to do with desktop: you also get it with
this recipe:
1. emacs -Q 
2. C-x C-f ~/.emacs.d/todo/emacs.todo RET
3. M-x todo-show
After step 2, the file emacs.todo is in fundamental mode (because, as
you note below, todo-mode is not yet loaded), but when todo-show finds
that a buffer is already visiting the file, it doesn't check the mode,
assuming it is already in todo-mode, and this leads to the error.  This
can be avoided by making todo-show check the mode and call todo-mode if
the buffer isn't already in it; however, automatically putting files
whose names end in ".todo" into todo-mode, as you suggest below, would
also avoid the error and be a more general solution.

There is, however, another problem that shows up with a saved desktop
containing a buffer visiting a todo file: namely, although the restored
buffer is in todo-mode (that is, after fixing the above error), it is
not properly displayed with narrowing to the current category.  I think
I can fix this by writing a function to add to
desktop-buffer-mode-handlers

> [Aug 28, 2013] Highlighting reports an error
>       
>       F H on an item results in
>       
>         call-interactively: Symbol's value as variable is void: hl-line-mode

Shortly before installing the package into trunk I had wrapped require
inside eval-when-compile to silence the byte compiler and must have
neglected to test it.  I should have used eval-and-compile and will
change it accordingly.
        
> [Aug 29, 2013] todo-mode not recognized with emacs -Q
>       
>       1. emacs -Q
>       2. C-x C-f ~/.emacs.d/todo/emacs.todo
>       3. The file is not visited in todo mode and hence not fontified.
>       
>       I think `auto-mode-alist' should have an entry for todo-mode /even if/ 
> todo-mode is not loaded apriori.

It's a good idea to make todo files recognizable without previously
loading the package, but it's not necessary to add to the default value
of auto-mode-alist; I'll follow the practice of many other packages and
put an autoload cookie before each top-level add-to-list sexp in
todo-mode.el, and also before the corresponding mode functions (I should
have done this earlier but wasn't aware of the practice).

> ==--== DONE 
> --==-- todo-mode/design
> [Aug 29, 2013] Clarify [Aug 29, 2013], Aug 29,2013 and &Aug 29, 2013 entries
>       
>       The former is more like when the entry was created.  

This depends on the setting of the option `todo-include-in-diary'.

>                                                            The second one
>       is what the diary expects as an active entry.  The last one is
>       disabled for diary processing.
>
>       From functional standpoint, 1 and 3 seem not very different.

No, 3 prevents marking the date of the entry in the calendar (see
`diary-nonmarking-symbol'); the entry itself still appears in the diary.
In contrast, 1 prevents the todo item from appearing in the diary.

>       I think [ ] is there mostly for filling up places and to improve
>       overall aesthetics.  We should really consider NOT having a date
>       associated with an item.

That would be a rather large UI change; I'm not sure how easy it would
be to implement.  I'm also not clear about what advantage it would
bring; do you have a use-case in mind?  Note that you can toggle the
file-wide display of the date/time header with `F h'.

> ==--== DONE 
> --==-- todo-mode/info
> [Aug 28, 2013] Mention that entry's date is controlled by 
> `calendar-date-display-form'

I regard this as an implementation detail that isn't appropriate for the
Todo mode user manual.  I think it's enough to know, as mentioned in
(info "(todo-mode) Todo Items as Diary Entries"), that todo items must
be recognized by the diary, if they are to be included in the diary.
The date format of todo item headers is not a Todo mode option, but is
automatically whatever the value of `calendar-date-display-form' is.
Users who know about this variable see that this is so, but those who
don't needn't be concerned with it.
        
> [Aug 28, 2013] Improve documentation
>       
>       Documentation is good and useful.  Please consider giving real-life
>       examples of how YOU use it.  What does a todo file stand for and
>       what could be possible categories, when a need for splitting or
>       merging categories arise etc.

My intention and hope was that the manual's descriptions should indicate
or at least suggest potential uses, leaving specifics up to to users'
owns preferences or needs.  Of course, my preferences and needs are
already reflected to an extent in the choice of commands and options and
how they work, and there are a few hints about possible workflows.  I
may consider trying to expand on these somewhat, but I don't want to add
lengthy descriptions of how I use Todo mode, which might give the
impression that other uses are unsupported (and I hope the package is
flexible enough to support uses I haven't even thought of).  However, if
the manual's descriptions are unclear anywhere, let me know and I'll try
to improve them.
        
>       After experimenting with todo mode, I consider todo-mode as
>       essentially a fontified conventional diary file with some bells and
>       whistles.  An introduction that says something in similar vein with
>       a info link to diary syntax will set the mood right for further
>       exploration.

This is certainly an important feature of Todo mode; isn't the node
mentioned above (which is part of the overview chapter of the manual)
sufficient?  But I also think this aspect shouldn't outweigh other uses
for todo lists that Todo mode can facilitate.
        
>       The section that talks about y k c d sequence seems to occur a bit
>       too early in the manual.  It should be placed a bit late in the
>       manual.  Mentioning the mnemonics upfront, y => diarY k => marK etc
>       is *good* though.

Since the item insertion key sequences are a central aspect of editing
in Todo mode, I think they belong in that chapter of the manual, and
that chapter shouldn't come too late, since editing (which includes
adding new items) is one of the most common functions of Todo mode.
(BTW, Stefan Monnier has proposed a different implementation of todo
item insertion key sequences that has a nice UI with direct feedback of
available completions; however, this relies on lexical binding, which
Todo mode currently can't use, since it makes essential use of some
dynamically scoped variables from the calendar package.  But he's
working on transitioning the calendar code to lexically binding, so this
feature may soon be possible in Todo mode.)
 
> [Aug 28, 2013] Does C stand for "upcased"-C or Ctrl key
>       
>       For example, in
>         (info "(todo-mode) Category Editing")
>       
>       and possibly other nodes, it is easy to be carried away by
>       mistaking C as control.  The documentation mentions upfront that C
>       is actually alphabetical-C but it is buried deep within.  More
>       importantly a regular Emacs user can get carried away easily.

Well, it's mentioned in the short chapter Key Binding Conventions, so
that seems pretty easy to find; but I can perhaps emphasize that it's
not the control key.

> ==--== DONE 
> --==-- todo-mode/nuisance
> [Aug 29, 2013] Replace C-x C-q with C-x C-s
>       
>       Sometimes I end up doing C-x C-q after an `F e' in the todo-mode.
>       It ends up enabling/disabling read-only mode on the todo file.  So
>       the choice of C-x C-q for repair needs some review.

I assume you mean you sometimes type `C-x C-q' in Todo mode, after
having typed it to exit Todo Edit mode (which you enter with `F e').  I
don't think this should cause any problems, since the self-insertion
keys are disabled in Todo mode.  My rationale for using `C-x C-q' to
exit Todo Edit mode is twofold: the mnemonic value of `q', and also
since exiting Todo Edit mode switches you from a writeable buffer to a
read-only one, it recalls the global value of that key sequence (which
is otherwise useless in Todo Edit mode).  Binding `C-x C-s' to
todo-edit-quit, on the other hand, naturally leads to the expectation
that this also saves the buffer, which it doesn't.  I could make it do
that, but I'm not convinced that the current behavior and key binding
are problematic.

> [Aug 30, 2013] C-x C-q should ensure newline at the end of entry's description

Why?  New items are always added on a new line.  Are you suggesting
items should be separated by an empty line?  I'd prefer to let users
simply append a newline if they are so inclined.

> [Aug 30, 2013] Review C-a in edit mode
>       
>       When I press RET the enty wraps at the window's edge, the cursor
>       rests at column 3.  But if I do C-a, it goes to the start of the
>       line.  These two behaviour taken together is confusing.

I don't see this behavior; with the default setting of todo-wrap-lines,
C-a puts point on the beginning of the visually indented line.  Can you
provide a recipe for what you describe?
 
>       I see that C-x C-q "repairs" the entry description by adding the
>       needed prefix on writing to *.todo file, I feel this 3 (or
>       whatever)-space rule could be relaxed.  As I am typing this bug
>       report, I realize that I need to do e m, C-x C-q one more time to
>       do the proper repair.

I'm not clear what you are referring to.  The "repair" done by `C-x C-q'
on leaving Todo Edit mode is to ensure all lines of the todo item but
the first, which contains the date/time header, is indented.  This is so
that if the item is included in the diary, it will be seen in its
entirety in the Fancy Diary display (see (info "(emacs) Format of Diary
File")).  Are you saying you want to suppress this indentation?  Is that
what you are calling "the proper repair"?

>       After experimenting with 10 odd entries I feel that `e m' and `C-x
>       C-q' switch is a bit hard on my wrists.  Too many context switches.
>       I wonder whether this context switch can be avoided.
>       
>       I would much prefer NOT switching between todo and edit modes and
>       rather have a single-shot repair of all edited items.  May be
>       repair on save could be a good option.

Since you recommend below to disable `F e', which allows editing the
entire todo file, are you suggesting here to make it possible to
simultaneously edit all items in a category?  While this would not
expose the entire internal file format, it would still pose a risk of
creating an inconsistant state and possible format corruption at the
item level (and also at the file level if you want to include the done
items section).  I don't know how easy it would be to avoid such a risk,
which makes this alternative not much better than just using `F e' with
the necessary caution.  But perhaps the packages you mention below have
relevant ideas I haven't thought of; I'll take a look, and thanks for
the pointers.

> [Aug 29, 2013] Adding entries doesn't scale.  Too much context switch with `e 
> m'
>       
>       Consider ideas from all.el or occur-edit mode for editing "in place".
>       
>       For repair, see how src block are edited with C-C ' in org-mode.
>       
>       Btw, RET seems a good replacement for `e m'.

I suppose RET could be an alternative binding, though it would be just
as plausible for `e e'.  In fact, the latter is better, since
todo-edit-item (bound to `e e') calls todo-edit-multiline-item (bound to
`e m') if the item is longer than one visual line.

> ==--== DONE 
> --==-- todo-mode/wish
> [Aug 28, 2013] Menu entries for todo-mode
>       
>               This is particularly important for new users like me to 
> memorize the key bindings.

I wrote a menu for an earlier version, which extended the menu of the
original todo-mode.el, but as the number of commands increased, I felt
putting them all in would make the menu unwieldy, in particular all the
item insertion commands.  But if the code for these can be reimplemented
as mentioned above, maybe a menu of the other commands would be
manageable.  I'll try to look into it, but I'd welcome suggestions for
how to structure the menu.
        
> [Aug 28, 2013] Disable `F e'
>       
>       Don't recommend it but ENFORCE it.  Do this by putting a 'disabled 
> property to the command.  Something variation of ....
>       
>         (put 'todo-edit-file 'disabled t)

This might be a good idea.

> [Aug 28, 2013] Enable selective display
>       
>       (add-hook 'todo-mode-hook (lambda nil (set-selective-display 1)))
>       
>       May be the above lambda can go as part of `custom-add-frequent-value'.
>       
>            (custom-add-frequent-value 'todo-mode-hook 'blah)

I'm not clear what you want selective display to be used for; is it to
"fold" longer todo items?  Is that really so useful?  Or do you have
something else in mind?

> ==--== DONE 
> --==-- todo-mode/minor
> [Aug 29, 2013] Improve aesthetics
>       
>       Alignment of numbers and headlines.  After about 15 entries (the
>       file you are reading is a good example) the entries look staggered.

I'm not clear what you mean here (I don't see any staggering of items
when I visit this file in Todo mode); can you be more specific?

> ==--== DONE 

Thanks again for your feedback.  I'll reply to this thread when I check
in changes prompted by it.

Steve Berman