Re: WAYTO: indexed man pages

[Ingo Schwarze]

Hi James,

James K. Lowden wrote on Sat, May 29, 2021 at 06:01:52PM -0400:

> how indexes could be implemented in

That problem was solved in 2016, and five years of experience have
shown the solution works well.

  https://man.openbsd.org/mandoc.db

> then man(1) might be adapted to use it,

It already is.  Try

  man -O tag=set ksh

on a system running a reasonably recent (less than two years old)
version of the mandoc implementation of man(1).

By the way, the same tags that man(1) reaches with -O tag
can be navigated to on the web as well:

  https://man.openbsd.org/ksh#set~2

Imagine the noise trying to search by typing "/set" in less(1)...

> and then man page authors woud have cause to designate indexed terms.

Five years of experience show there is surprisingly little need for
that.  The mandoc implementation of deep linking to anchors fully
automated the task without requiring any additional markup.  In
practice, it just works without needing to bother the author, just
regular semantic markup is sufficient without adding manual anchors,
in the vast majority of cases.

All the same, the mdoc(7) language did recently start supporting
manual tagging for the very small fraction of situations where it
helps, and for the relatively few authors who are willing to go to
such lengths:

  https://man.openbsd.org/mdoc#Tg~2

> The best "find term by index" feature in documentation viewing software
> isn't very good: in the info viewer, you can type " i " and a string,
> and enter.   It also supports tab-completion on indexed terms, which is
> as close as info ever comes to "very cool".  

That's not the state of the art.
Mandoc has been doing semantic searching since about 2004:

  https://man.openbsd.org/apropos#Macro_Keys
  https://man.openbsd.org/apropos#EXAMPLES

I often use queries akin to

  man -ks 2 Vt=timespec

when i wonder which system calls deal with the "struct timespec"
data type.  And the like - you get the picture.

> The nearest thing in less(1) to support for an index would be its tag
> support. If groff emitted a tags file, less could use it to navigate
> within the man page.  

Yes, the mandoc implementation of man(1) has been doing exactly that
for years.

> I would bet it's technically feasible to use "one giant tags file",
> indexing the whole man-page corpus.

Well, actually, you get one mandoc.db(5) file per manual page tree,
so for example if you have your base system in /usr/, your ports and
packages in /usr/local/ and your X11 in /usr/X11R6/, then you get

  /usr/share/man/mandoc.db
  /usr/local/man/mandoc.db
  /usr/X11R6/man/mandoc.db

and man(1) transparently knows how to use those.
If you have specialised collections of manual pages installed,
you may also have files like

  /usr/local/lib/tcl/tcl8.5/man/mandoc.db
  /usr/local/plan9/man/mandoc.db
  /usr/local/share/doc/posix/man/mandoc.db
  /usr/local/lib/eopenssl11/man/mandoc.db

such that these don't get confused with native system documentation
but can be accessed when desired using the -M option or the MANPATH
environment variable.

> There is the, ahem,  *slight* problem of how to denote index anchors,
> i.e., locations in the document that the index "points at" for a term
> included in the index. Currently in mdoc we have only "points to"
> macros:
> 
>       .Sx point to a heading
>       .Xr point to a page
>       .Lk point to a URL (mandoc)
> 
> IOW mdoc has nothing akin to the HTML functionality 
> 
>       <a href=#foo>
> and
>       <span id="foo">.  

It has: .Tg
Only, it has not yet been implemented in groff.

Then again, it is very rarely needed.
The OpenBSD base system currently contains 82 instances of .Tg
in its 3325 mdoc manual pages, so on average one every fourty pages.

> Possible ways to denote a target in mdoc: 
> 
> 1.  Extend .Lk to denote a target
> 2.  Introduce a new macro, perhaps .Ix

Please, do not reinvent what is already in production and working well
for years.

> 1.  If grotty produced etags output, less(1) could support navigation
> by "man page tags". 

The mandoc implemention of man(1) has already been doing exactly
that for years.  And the tags you can access with less(1) :t are
more or less the same that you can navigate to on the web using
fragment identifiers.

> 3.  If the above two were accomplished, man page authors would finally
> have the opportunity to enhance their documents with index anchors. 

Little need to, it mostly just works without providing manual anchors.

> IMO hyperlink nonsupport is the Achilles heel of the man page system.

Maybe it was a decade ago.

All that said, specific suggestions to improve the system are always
welcome, and so are patches.  During the last decade, i have
received useful suggestions from probably more than a hundred
people, and patches from probably more than twenty, too.

To name just one person who helped move tagging and indexing support
forward even further recently: Klemens Nanni provided valuable input
on several occasions, but others helped, too.  And of course,
Kristaps Dzonsons laid the foundations about a decade ago, using
SQLite back in the day.

Yours,
  Ingo


P.S.
One feature that is not yet implemented, that i have been considering
for more than half a decade and mentioned multiple times at
international conferences as a desideratum, is how to do something
similar to clicking on a web link on a terminal or in a virtual
terminal.  In a text editor, you have the concept of the cursor,
but in a pager, you typically do not; in particular, less(1) always
shows you a part of the text, but doesn't have a concept of where
in the text shown you are.  Of course, text browsers like lynx(1)
have solved that, but i haven't made up my mind how much i like
their solution and whether that should be integrated into less(1).
Or whether a different concept like pressing a specific key to mean
"i would like to follow an external link now", then typing in the
first few characters of the anchor name and pressing ENTER, would
be better.  And whether and how that should be integrated with the
long-established :t functionality.  Or something even different,
like numbers shown in parentheses after the links, such that pressing
the "2" key lets you follow the second external link.

Then again, i feel that's a problem of relatively little urgency.
On the web, this problem doesn't exist in the first place.  On a
glass console, you can easily get another shell with tools like
tmux(1) or screen(1) and run a second instance of man(1) there,
then switch back and forth using normal job control.  If you are
using a window manager with virtual terminals, it's even preferable
to open the new page side by side, each in their own terminal window,
such that you can readily read both in parallel and compare the
texts.  So while some situations might profit from something like
"clicking on a link", most situations likely won't have any use for
such a feature in the first place, at the console.