bug#44509: 28.0.50; Error querying with new gnus-search and notmuch

[Eric Abrahamsen]

jao <jao@gnu.org> writes:

> On Wed, Nov 11 2020, Eric Abrahamsen wrote:
>
> [...]
>
>> This code that comes straight from the old nnir.el, and has always
>> looked a bit fragile to me. The problem is that it really just expects
>> each message to be in a regular file, with groups as folders.
>>
>> So you're using notmuch as a search engine for a local leafnode nntp
>> server, and indexing its message store directly?
>
> Yes.
>
>> Is there any leafnode setting that could influence how it stores its
>> messages? Can it be convinced to store them in hierarchical folders?
>
> It is exactly what it is doing.  Leafnode messages in the nntp group
> gmane.emacs.bugs are stored in /prefix/gmane/emac/bugs/<article no.>
> exactly as nnmail would do.  And the result of the search by notmuch is
> displaying those paths correctly.  The problem is that, in the code i
> mentioned, when that output is processed, the code is expecting to see
> in the results list gmane.emacs.bugs instead of pathnames.  So i don't
> understand how that code has ever worked, even on the old nnir.

I see, I had your problem backwards. My guess is that no one has tried
to do this before, so the issue has never come up.

BTW, while there's no longer a "gmane" search engine, it would be nice
to provide an engine for searching other nntp servers. In your case
you're running it locally, but do you know if a remote leafnode server
handles any sort of search functionality? (I might as well google this
myself.)

>> I suppose I could change the group-regexp to munge periods, but that
>> could cause breakage in other cases, and I would be hesitant to do that.
>
> In may case, the problem is the other way around: the regexp should
> accept /, but is only accepting periods (because it's constructed
> directly from the group name, which, for nntp at least, contains
> periods, not slashes).
>
>> Otherwise, all the indexed search engines have a
>> `gnus-search-indexed-extract' method that's used to actually return the
>> file name from the results buffer.
>
> See above.  It's really very easy for me to fix this with an around
> method like this one:
>
>      (cl-defmethod gnus-search-indexed-parse-output :around ((e 
> gnus-search-notmuch) s q groups)
>         (let ((gs (mapcar (lambda (g) (replace-regexp-in-string "\\." "/" g))
>                           groups)))
>           (cl-call-next-method e s q gs)))
>
> because the problem in my case is the group name, not the search
> results. Note that, for cases (if any) where the group names already
> look like "gmane/emacs/bug" (which i suspect is what nnml and nnmaildir
> are doing and that's why it works), that's a no-op.
>
> So yes, for me it's a solved problem, even without defining a new
> engine.

Okay! Glad that's sorted. You might still want a new engine, if you have
other notmuch engines that shouldn't inherit this behavior.

>>> On other news, i was trying to find a way in Gnus to go from Message-ID
>>> to article no. for IMAP groups or nnmaildirs (which would make using
>>> notmuch with dovecot really trivial), but without luck: anyone knows of
>>> an easy way?
>>
>> Come to think of it, nnimap can already accept article numbers
>> as message-ids. So if notmuch returns its results as message-ids, it
>> should work transparently.
>
> Yes, i thought that at first.  i got stuck when i discovered that
> nnselect apparently only accepts article numbers to build its groups.  i
> guess one could extend nnselect to also accept message ids, and that
> that would be useful also in other contexts.

It looks like nnselect can accept message-ids when requesting individual
articles, but not when categorizing groups/articles, nor when requesting
headers. My guess is that might be quite a bit of work to implement,
particularly since the id->number routine will be different for each
backend. You can see that happening in `nnselect-request-article'.

>> The problem is that while the mechanism is there, it works by searching
>> each message-id and getting the proper article number that way. My guess
>> is that you'd be negating most of the speed advantage of using notmuch
>> like this, and you'd still be better off using dovecot's own full-text
>> indexing. You don't have to use xapian!
>>
>> https://doc.dovecot.org/configuration_manual/fts/
>
> Ah, thanks for that, i'll definitely take a look.  The nice thing about
> using notmuch would be that is one less moving piece to maintain, and
> that then one could move very, very transparently between notmuch's
> emacs interface and gnus, using the same message files and indexes.
>
>> BUT, if you really wanted to do this, it would be relatively easy to
>> check for "--output=messages" in 'switches, and use that instead of
>> "--output=files".
>
> Yes, it's trivial to get a vector of results of the form 
> [group message-id score] ... a full engine like that is just a few
> lines:
>
>     (require 'notmuch-query)
>
>     (defclass jao-gnus-notmuch (gnus-search-notmuch) ())
>
>     (cl-defmethod gnus-search-run-search ((engine jao-gnus-notmuch) srv query 
> groups)
>       (let ((qstring (gnus-search-make-query-string engine query))
>             (res))
>         (dolist (group groups)
>           (let* ((folder (format "folder:%s"
>                                  (replace-regexp-in-string "\\." "/" group)))
>                  (ids (notmuch-query-get-message-ids qstring folder)))
>             (dolist (id ids)
>               (push (vector (gnus-group-full-name group srv) id 100) res))))
>         res))

Hmm, I guess it wouldn't be useful for me to implement this in the
general case, since the ids aren't going to be accepted by nnselect.

> That's all (well, obviating thread seaches, but those are not hard
> either).  Except that it doesn't work because nnselect wants 
> [group article-no score].  If you tell me how to obtain article-no from
> message-id, i'm done :)

For nnimap, it's `nnimap-find-article-by-message-id'.

Eric