Re: Automatic (e)tags generation and incremental updates

[Dmitry Gutov]

Hi Tom,

On 09.01.2021 23:49, Tom Tromey wrote:

> Dmitry> - Call some xref command like 'M-.'. See message "Generating new tags
> Dmitry>   table..." (it's saved in /tmp/...), then see navigation happen.
>
> On "xdg"-ish systems, it would make sense to use the XDG cache directory
> here.

I should look into this later. If we end up using putting TAGS files 
into prject directories (see below), that will be moot.

> Dmitry> - Pressing C-M-i instead should also trigger tags table generation.
>
> Dmitry> - When you switch between projects, the previously generated tags
> Dmitry>   tables are discarded. It's not too hard to improve, but that would
> Dmitry>  involve some choices/tradeoffs.
>
> What are the tradeoffs?

There are several questions.

- Do we want to store the generated files openly in the root directories 
of each project? I.e. save them as TAGS. That might look more 
familiar/comfortable by the old-timers, and some users might even 
pre-generate such files, if the process takes a long time. And the 
contents will be able to reliably survive for a long time. Storing them 
with "garbled" names somewhere in /tmp of XDG cache risks having to 
fully renenerate the indexes at least every time the machine reboots. 
The downside to creating TAGS files is it's unfamiliar to newcomers who 
usually expect index cache to be hidden, and they'll have to either 
update .gitignore or risk checking them in.

I don't see a lot of the old timers in this discussion now, and most 
existing users of etags are likely satisfied with the current workflow, 
so perhaps that kind of familiarity is not important. So it probably 
comes down to being able to generate such files only once per project, 
and only update them later. I'm not such what size of project that will 
become a significant advantage at, but it's likely that at that point 
etags.el's other performance limitation will come into play. We'll need 
some real feedback on that. Until then, the generated files will stay in 
/tmp. Might even keep them off disk entirely, actually, though that 
would require some changes to etags.el (help welcome).

- Do we keep such file in memory every time after the user has switched 
to a different project (and, say, maintain a {project -> file} alist in 
memory), or close and reopen upon switching. If the files are stored on 
disk, implementing the latter is plainly easier with the current 
etags.el code. Is eliminating the delay worth the code complexity and 
increased memory usage?

- Being able to pick up an old TAGS depends on our ability to compare 
the current project contents against a list of files and one timestamp 
(TAGS modification datetime) quickly, much quicker than simply 
regenerating such file would take. And if we can't, there's no point in 
keeping them around.

> I tend to think that conceptually each buffer should point to its
> corresponding tags table.  Then some separate logic could be used to
> decide when to kill some tag file buffer.

Ideally, perhaps, etags.el would provide an interface for polling a 
specific tags table (for completions or locations) by simply binding one 
or two local variables. At the moment, though, the route there seems to 
be through file-local variables and through calling visit-tags-table 
with non-nil second argument.

> Dmitry> - When files are deleted, or otherwise changed outside of Emacs
> Dmitry>   (perhaps with 'git checkout'), nothing is updated. I have a few new
> Dmitry>  ideas, haven't started on them yet. Workaround: toggle
> Dmitry> etags-regen-mode off and on, which will result in full rescan when you
> Dmitry> use 'M-.'.
>
> It seems to me that the default ought to be to update the tags table on M-.
> One nice way to do this would be to run etags in the background, so that
> the work of updating would be done in parallel with the user typing,
> since presumably you'd want to ensure that etags has finished before
> jumping to the result (or fulfilling a completion request).

It sounds clever, but UUIC that would only benefit users who call M-. 
with C-u or who have customized xref-prompt-for-identifier to t. Even 
among those, it would only be able to help (without sacrificing 
correctness) only those who don't use something like icomplete-mode for 
tag input. Because the completion table already depends on the tags 
index (which is out of date). On the flip side, even when the tags 
generation is synchronous, you can start typing right away.

If general, doing updates when Emacs is idle and/or asynchronously are 
quality-of-life changes that can come later after we improve correctness 
(i.e. make sure the index is up to date even after external changes).

Doing that will require some more processing implemented in Elisp which 
can still create annoying stutters in Emacs, whether the process calls 
are asynchronous or not. Debugging those and working on algorithmic 
complexity there is easier when work happens at predictable points in 
time. Asynchronous calls also make error handling more difficult (and 
the current Emacs threads -- even more so).