bug#26710: Fwd: 25.2; project-find-regexp makes emacs use 100% cpu

[Dmitry Gutov]

On 02.05.2017 10:15, Eli Zaretskii wrote:

> Can you explain the significance of xref--regexp-syntax-dependent-p's
> tests?  I don't know enough about xref to grasp that just by looking
> at the changes.

When it returns nil (the regexp is not affected by syntax-table):

If the file containing the hit is not open, we now skip inserting the 
first few lines of that file into the temporary buffer, and calling 
set-auto-mode.

And, whether it's open or not, we skip the syntax-propertize call.

> > With this, project-find-regexp for 'emacs' finally completes in ~10
> > seconds on my machine.
>
> It takes about 15 here (and 45 in an unoptimized build).  I guess this
> slowdown is expected, since this is a 32-bit build --with-wide-int, so
> should be 30% slower than with native ints.

Thanks for testing. To be more accurate, it's about 10 seconds in my 
normal session, and about 6 seconds starting with 'emacs -Q'. My laptop 
is most likely faster.

> If the processing is in filter and sentinel functions, I'm not sure we
> will need any further speedups, because the UI will remain responsive.

The filter and sentinel functions are not allowed to have direct access 
to the final output buffer, hence the need for abstraction.

I guess you favor the "one callback per hit" approach, then.

Still, if the filter function and sentinel functions take a lot of time 
(and/or get called a lot), like it will be in this example, the UI can't 
as responsive as usual, can it?

> > > But that doesn't need
> > > to involve threads, and is being done in many packages/features out
> > > there, so I'm not sure what did you ask me to do with this.
> >
> > I imagined that the xref API that allows this kind of asynchronous
> > results might look better and more readable if it's implemented with
> > threads underneath.
>
> If you need advice for how to implement something like that, I can try
> helping with threads.

I'd like a more general advice first. E.g. do we want to go this road? 
The dir-status-files like scheme should work without threads, too.

It seems a bit brittle, though: if the process filter is supposed to be 
calling the callback for each item, the callback has to be in place 
right away. And the process will be started before that happens.

We'll probably be saved by filters having to wait until the current 
command finishes executing, though.

> > The main thing to understand is the xref API, not the internals of the
> > package.
>
> Well, I lack that understanding as well.

I'm hoping it's not too hard to obtain even just by reading the 
Commentary section in xref.el. But hey, you don't have to.

The callbacks approach seems viable, too.