bug#16555: 24.3.50; Company and CAPF: dealing with completion values containing extra text

[Dmitry Gutov]

When dealing with code completion for programming languages, it's usual
that candidates have associated text - most often, as function names,
they include the argument list.

Related issues:

* When inserting the "common part", we must ignore the extra text.

* The argument list is useful, for example when the candidate is
  inserted in the buffer. After it was explicitly selected, we also
  insert the arguments list, but replace each argument with self-erasing
  field when you type over it. We do that in company-clang and
  company-eclim (and could also do in company-senamtic, were someone to
  request that).

* C-like languages usually provide method overloading, and then the
  argument list is a part of a method's identity. Any metadata is
  associated with the tuple "method name" + arguments, so we want to
  pass the method name with arguments to functions that retrieve
  documentation, location, etc.

* If the completion table values only contained method names,
  `delete-duplicates' would remove all but one of the methods with the
  same name. But we do want to remove duplicates.

At the moment the following approach emerged (for examples, again, see
company-clang and company-eclim): the completion values include the
argument lists (but nothing extra except that), and the respective
backends define an undocumented command: `crop'.

That command is only used in two situations:

1. When `company-complete-common' is called, we insert the common part
among all candidates, but before that we call (backend-function 'crop
"common-part"), to make sure not to insert the paren or anything after
it.

2. When `company-auto-complete' feature is used, typing any character
from `company-auto-complete-chars' insert the currently selected
completion into the buffer. In that case we also only want to insert the
method name, and so backend's `crop' function is called.

When the candidate is inserted normally (by selecting it with M-p or M-n
and then pressing RET), `crop' is not called. The full candidate value
is inserted into the buffer, and then we call `post-completion' backend
command, to allow it to remove the arguments list, or do something more
advanced (see the * #2 above).

This behavior is inconsistent, and `crop' doesn't sounds too good as a
command name.

At the risk of breaking some existing code, and thanks to `crop' stil'
beging undocumented, I'd like to define a `value' command instead that
would return the "cleaned" candidate text, presumably without the
arguments list. It would get called anytime before the candidate text
gets inserted, and to get the above mentioned templatification behavior,
the `post-completion' code will need to explicitly insert the rest of
the candidate text (the full candidate will be passed as the first and
the only argument). Which will be the main difference from the backend's
writer perspective.

I see two problems:

* Including extra text in completion table seems like it won't mesh well
  with the completion-at-point-functions interface, and specifically
  with the completion-at-point as the frontend. How would one write a
  CAPF function for clang or eclim with argument lists in mind?

* `company-update-candidates' currently calls `company--safe-candidate'
  on the "common part", on the assumption that the `crop' command will
  return something meaningful for a string that's not itself a
  completion candidate (and currently, they all do, because they look
  for `(' instead of, say, using text properties or a hash table
  lookup). If the command is called `value', it would be more tempting
  for the implementor to only expect it to be called on actual
  candidates. And then it'll return nil or do something unexpected when
  called on something like "foo(".

Thoughts?