Re: Unified project interface

[Stephen Leake]

Dmitry Gutov <address@hidden> writes:

> On 07/30/2015 11:17 AM, Stephen Leake wrote:
>
>> I don't understand; the user sets load-path. If the user wants to use
>> project-* and xref-* functions with the current project, why is it not
>> in load-path?
>
> It's not always the current practice, AFAIK.

I still don't understand why. But I've proposed
`project-add-search-path' (below and in another email), so that can be
used to handle this case.

> And adding it to load-path interactively is not something we should
> force the users to do.

Yes, if the user doesn't want a file in load-path, they don't put it
there. But then they will have to use project-add-search-path instead.

We have to expect some help from the user to indicate what's in the
project!

> Consider that when xref-find-references doesn't find anything, it will
> say "nothing found", without further explanations, leaving the user to
> puzzle out why and what. We should err on the side of showing too
> much, rather than too little.

Sorry, I can't guess what you have in mind that the user is doing here.
What is the user searching for, what did they do to specify
project-search-path, and what do they expect xref-find-references to
show?

>> 1) Because many Emacs functions don't recurse. For example, I'd like to add:
>>
>> (defun project-find-file ()
>>    "Prompt for a file name, find it on project-search-path."
>>    (interactive)
>>    (let ((filename
>>       (completing-read
>>        "file: "
>>        (apply-partially
>>         'locate-file-completion-table
>>         (project-search-path (project-current))
>>         nil))))
>>
>>      (find-file (locate-file filename (project-search-path 
>> (project-current)) nil))))
>>
>> This works fine with "load-path" instead of "project-search-path". But
>> it won't work unless project-search-path is flat;
>> locate-file-completion-table and locate-file do not recurse.
>
> I guess you'd need to implement a "flattening" function for that,
> indeed.

If project-search-path is recursive, yes. Which is certainly possible to
implement here; I propose a "recursive" project flag below. If that's
set, this function can use directory-files and directories in
project-ignores to do the recursion.

The predicate for locate-file-completion-table (set to nil above) needs
to handle the files in project-ignores as well.

> I don't really see myself using this command, though.

That doesn't mean it's not a valid use case; you need to consider all
users.

I often want to look at project.el (or xref.el, ada-mode.el, etc); I
bind this function (currently using load-path) to M-F12, so M-F12
proj <ret> finds it for me easily.

The Ada user group requested this function for Ada projects (it has a
similar implementation).

>> 2) As I've mentioned before, not recursing allows computing the set of
>>     included/excluded directories once, instead of computing it again on
>>     each recursion.
>
> And I countered that, for many commands, computing that set is
> unlikely to be the bottleneck. And cache invalidation is not
> user-friendly.

Which is faster/friendlier depends on the details, so we'll have to wait
for actual implementations. All I'm asking is that we do not make design
decisions at the generic project API level that rule out this approach.

>> 3) Because some tools provide and expect a flat search path (AdaCore
>>     tools, gcc include path, emacs load-path, compilation-search-path,
>>     ff-search-directories).
>
> That's not really true.
>
> There are many directories inside /usr/include, but C_INCLUDE_PATH
> won't mention most of them. Instead, #include <postgresql/pg_config>
> will use /usr/include/postgresql/pg_config.h, as long as /usr/include
> is in C_INCLUDE_PATH.

Yes. So if the user wants to search all relevant includes for some
declaration, they have to treat C_INCLUDE_PATH as recursive. But the
compiler (which _defines_ C_INCLUDE_PATH) does _not_ treat it as
recursive. So it's confused.

> There's only one directory in Emacs load-path with "cedet" in its
> name, but (require 'semantic/db) will load lisp/cedet/semantic/db.el.

Similar situation: `load' does not treat `load-path' as recursive, but
text searches need to. I had not considered that.

I think cedet is the only exception to the 'flat load-path' rule (at
least in core emacs); I hope it goes away, and is not copied by other
projects.

I would provide a special syntax to handle this in
project-add-search-path:

(defun project-add-search-path (project path &optional make-recursive)
  "Add PATH (a list of directories) to the search path for PROJECT.

If a directory in PATH ends in \"/**\", also add all directories under
that directory.

If MAKE-RECURSIVE is non-nil, the full project search path is pruned so
that only highest-level directories are present, and all uses of
`project-search-path' recurse into subdirectories. In this case, it is
likely you will also need to specify ignored directories with
`project-add-ignores'.

If MAKE-RECURSIVE is nil, uses of `project-search-path' no not recurse
into subdirectories."
... )

The Ada project files support the ** sytnax.


User projects can have directory structures for which recursive
search from the vc root is wrong:

myproj/
    .git/
    .gitignore

    build/
        Makefile
        ada_project.gpr
        *.exe
        obj/
            # contains object files for ada files

    lisp/
        # elisp files for emacs-25+

    lisp-emacs-24/
        # elisp files defining emacs-25 functions

    lisp-emacs-23/

    ada/
        # ada files

The elisp project (depending on current Emacs version) will only want to
search one of:

(lisp)
(lisp lisp-emacs-24)
(lisp lisp-emacs-23)

The Ada project will only want to search the "ada" and "build"
directories (for the Makefile and Ada project file).

There will be cases where the user will want to search all files; then a
separate project that uses project-add-search-path to add all dirs would
be appropriate.

No search should include the "obj" directory.

This could be handled if the user can explicitly specify project ignores
(not just rely on .gitignore etc; git does _not_ ignore the lisp-emacs-*
and ada directories). More complicated structures are harder to handle
with ignores; you'll end up with a long flat list of ignores, so it
might be simpler to have a flat list of includes instead.

> You're right about compilation-search-path and ff-search-directories,
> though.
>
> It seems to be true for the Ada source search path as well, but since
> /usr/lib/gcc/x86_64-linux-gnu/4.9/adainclude/ has no subdirectories,
> the point is moot. Traversing it recursively wouldn't make anything
> worse.

User Ada projects can easily have directory structures for which
recursive search from the vc root would be wrong; see above.

It would be surprising to an Ada user (used to flat search paths), if the
Emacs project silently started using recursive search paths.

>>      It is not at all clear how to convert a flat path into a recursive path
>>      with ignores; the solution you have for elisp load-path is a kludge,
>>      and could easily be wrong for some users.
>
> I'd like to see one of those users first.

Me, and many other teams at NASA - structures similar to the above,
where the non-included directories can also be release version,
debug/release options, target hardware/os option, compiler
vendor/version, etc.

The jdee project has lisp and java directories.

The kludge is in the root implementation of project-search-path, so it
applies to all project backends that only customize
project-search-path-function.


I have no objection to allowing some project backends to use recursive
path/ignore dirs; I just want to also allow some backends to use
non-recursive path/ignore files. Or the user can choose, via
project-add-search-path above.


>>> xref-location lives there, and it's the best purpose-agnostic location
>>> class that we have now.
>>
>> So move it to "location" instead of "xref". The longest journey starts
>> with one step ...
>
> Until it's used in at least two different places, creating a separate
> package for it is, I think, premature.

Right. We now have those two places; xref-find-definitions,
project-find-regexp.

Unless project-find-regexp uses the compile location data structure,
which would probably be better.

Although it would be even better if project-find-regexp uses the compile
location data structure wrapped in a location-compile class, derived
from the root location class, renamed from xref-location.

--
-- Stephe