Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I've just described in the other thread that I would like to have
>> finding references and finding files to be able to operate on either
>> sub-projects or super-projects on demand.  This is the problem I'm
>> facing, and it's not new.  For example, there is only: find file in the
>> very large project, and find file in the current directory.  There is no
>> "find file in this section of the repo, which is a sub-project in
>> itself".
>
> It would be much more helpful in a dedicated bug report where we could
> discuss the details, collect the votes and see what kind of design
> will ultimately satisfy the requirements. Instead of drowning it all
> in this thread, which is only moderately related.

I think we're doing fine here but I've changed the subject line to
"unbury" it from the thread.

>> IMO, it's not "improper" to describe problems and use cases: in fact I
>> prefer that people describe over jumping to vapourware solutions.  But
>> if you're really looking for a suggestion as to _how_ to design it, I
>> suppose my problem would be well dealt with a negative prefix on C-x p g
>> and C-x p f giving me a choice of which project to operate on.  But that
>> is only one possibility: new commands are also acceptable.
>
> Note that you can more-or-less do this now: press 'C-x p p', select
> the parent project from the list, then choose 'f' or 'g' to run the
> command there. So to justify the added complexity one should say that
> they do want to use this feature frequently enough to justify the
> added complexity (which will reduce the number of keystrokes).

Both cases -- focus on super-project/focus on sub-project -- are
definitely common.  I have no problem in using an existing interface,
even if it idiosyncratic.

So if the inner-most sub-project is the default but somehow I can
explicitly call up a "project picker" that shows me the super-project as
one of the first options, that probably works fine.

> And even if we do, we might not need the additional notion of
> sub-projects. E.g. 'search in the parent project (if any)' might work
> as "take the root, go up a directory, search for a project there; if
> it exists, use it". Though Stephen L. might want a generic for that,
> since his projects do not correlate with directory tree.

And in my case, sub-projects don't necessarily exist exactly one-level
down super-project's root.  Sometimes they do, but now always.

Of course, just as data point, I've been solving all my problems outside
project.el for a good long while now, even before I was faced with this
gigantic monorepo and even before project.el was a thing.  I use Leo
Liu's ack-el.el, which greps from the super-project's root by default,
and from wherever I want given enough C-u.  And I have a similar
separate command for finding files from arbitrary points in the
hierarchy.  And I have a similar command for compiling.  I just happen
to think having this in project.el would make a lot of sense.  I've
recently experimented with it and found the new project.el features
pleasant to work in smaller or well-behaved projects.  It's not powerful
enough for the big stuff, but it ought to be.

> OTOH, if one of the operations will require a step "get all
> subprojects of the current project", then a separate notion might be
> required, with a separate hook.
>
>> As to how one defines sub-projects, I think having
>> project-find-functions be used to compose a list of projects (as opposed
>> to to stopping after finding one) would be a nice way.  It would be nice
>> if the elements of project-find-functions could be informed of the
>> projects found so far.  But keeping the "stop after first" behaviour and
>> having members return a more complex object is maybe also good.  I
>> really don't care much what you eventually choose, as long as I can
>> solve my problem.
>
> If we wanted to define projects and subprojects through the same hook,
> the step "get all subprojects of the current project" might not be
> feasible to implement. Because of performance-related considerations.

Those remain to be seen, I can't guess what things you're imagining.

I'm not sure we need that step at all.  After all there is no "get me
all the projects in my hard drive" either, and at least I haven't missed
it.

Anyway, from my POV this type of sub-project definition would not be
very nice.

  (add-hook 'some-new-subproject-find-hook
            (lambda (dir)
              (cons 'super-project
                    (some-code-which-scans-and-returns-a-list-of-subprojects))))

I would prefer this, which I think is more readable and in-line with
current project.el:

  (add-hook 'project-find-functions
            (lambda (dir)
              (let ((super (project-current)))
                (when (and super
                           (string= (file-name-directory
                                     (directory-file-name dir))
                                    (project-root super))
                           (my-special-condition dir))
                  (cons 'transient dir)))))


Maybe 'my-special-condition' could read a directory local variable to
decide if sub-projects are enabled or not.

For my idea to work, project-current would have to be changed to

1. Call all members of project-find-functions: return the innermost
   project found thus far.
2. Not call project-find-functions recursively when called from within a 
function
   in project-find-functions, just return the innermost project found
   thus far.

I don't see any big performance problems in this alternative.

But, again, the above are suggestions: _any_ alternative that lets me
define inner sub-projects within a larger super-project and let me
choose where I want to grep, find-file, compile, would be an
improvement.

João