Re: Is this a new idea? A conceptual essay on user interface design

[Garreau\, Alexandre]

This mail I answer too is now quite old but I just read again this
mailing-list and I found it interesting thus I’m answering it (make me
notice if this is bad behavior).

On 2015-10-05 at 16:12, address@hidden wrote:
> The idea proposed here is the result of a long and mostly conceptual
> research […] as I was increasingly dissatisfied by modern interfaces
> […] I'm not a professional developer, just a hobbyist, and so far just
> did […] on my own.

I’m not yet a “prod” developer I would say (rather than professionnal, I
guess there are non-pro devs who actually work on packaged/well-known/
useful-enough free software), and I have no skills in GUI programming (I
only tested to make Qt interfaces once a few time after I learned to
program and found it horribly complex and repelling, as C++ and OOP
already feeled to me) except CSS but it’s crap and not true GUI to me.

> Second, if it is indeed a new idea, and I believe it is because in a
> year of research on the topic of user interface design I never
> encountered such a system,

Do you mean concerning final user interface or development interface?

> I want to make the idea public as soon as possible to prevent some
> non-free software producing entity to have the same idea and patent it
> while I still struggle with the prototype.

I think I would have thought like you in a such situation but when you
lack recognized skills or fame (like us two, at least in this
discipline) you can’t manage to implement it freely without talking
publicly to a lot of people for a long time, and anyway I hope these
—big— entities that makes proprietary software and patents don't give
any credit to us and won't think taking our ideas would be worth doing:
worth the work, worth the risk, etc. I guess commercial people that
patents ideas and/or launch commercial software (the worst proprietary
software to compete with I guess) has not the same skills as needed to
recognize a good idea.

> The goal is to write a system that is capable of displaying every kind
> of data and manipulating it in every way the user wishes--and
> hopefully in a coherent way. That is of course impossible as different
> kinds of data need different interfaces and we all know that you have
> to write a new interface for every single purpose. It is a well known
> fact in interface design that there is no "perfect universal
> interface".

That recalls me what web tries (tried?) to do with ontologies (DTD, XML
Schemas), XSLT and CSS, and semantic web in general. That’s almost
it. Except nobody knows rdf/sparql/dtd/xml schemas, it is enough complex
to lack any easy tutorial, only a few people program with XSLT (that
anyway tastes like a failed scheme written with that horrible XML
syntax), and CSS fails at staying interoperable and making stuff
accessible and *really* well rendered (as do TeX), yet I guess I learnt
that recently it became turing-complete at least (but I guess in a
horribly hacky manner) so maybe will it stop to be so much not enough
expressive so that people stop making CSS generators.

I already had the idea to do something with these concepts, but I
terribly lack skills, knowledge and focusing/attention abilities. Anyway
a P2P system of ontologized semantic microformats (and breaking
compatibility with ascii/unicode) sounds good to me to make stuff more
accessible, modular, simple, processable (by AI for instance) and
factored.

> But what if we could break down the data to atoms that can be easily
> rendered by a series of abstractions like we do all the time in
> computer languages?

Doesn’t already any GUI/typesetter/layout-engine work like that? Did you
look at TeX? XML/CSS?

> After all, most types of data really are just collections of other types of
> data.

You mean like XML? or lisp?

> Those abstractions could be collectively written and shared in the
> spirit of the free software movement resulting in the best possible concrete
> approximation of the the stated abstract goal.

Unfortunately I think sharing isn’t that simple. A such system would
probably break into several competing incompatible implementations
because of Conway’s Law:
http://wingolog.org/archives/2015/11/09/embracing-conways-law

Except maybe if we make ontologies that can describes interfaces to
anything and they are self-descriptive and can describe each other so
that stuff can be kept compatible… But people said me that sounds a lot
like AI and I lack the linguistics and cs skills for knowing what is
doable there.

> The universal interface is a system that allows arbitrary type-checked
> combination of procedures (functions, binary programs...) and data (a shared
> RDF store, a personal RDF store, traditional files) and is able to produce a
> meaningful visual language that represents the concerning data and procedures
> to the user.

Why only visual? having a set of RDF and semantic stuff —especially when
really simple and modular like you say— would allow to make accessible
stuff that could be rendered on visual, audio, ascii-text, braille, any
not-existing-yet human-machine interface or a combination of several ;)

The only reason I see to favorise vision is because most of human being
have working eyes and then most humankind based most stuff on vision and
favorized visual interfaces as the only superior interface for
everything artificial, so that graphical interfaces get more developed
than audio or sensitive interfaces (and then it’s a vicious circle where
since alternative interfaces don’t exist or are advanced enough, none
get built or developed), and that's especially true for free software
sadly (since Apple’s Siri, Google and M$ stuff since to dominate market
there, and that’s especially bad for our freedom and even worse for
blind people that then can't use as much free software than valid people
for certain stuff/devices).

And yet that’s not so true for everybody: anybody growing old will loose
vision abilities, blind people exist and have to be taken into account,
and a large portion of population is getting myope or long-sighted and
that causes problems with either large distant screens like
TV/cinemas/projectors, or little near screens like smartphone’s, then
having adaptative interfaces would be really great to improve currently
existing interfaces.

> The approach presented here stems on the observation that the RDF data model
> allows to build a data store for all the structured data you want to process
> as all kinds of data can be projected to a semantic graph--with the exception
> of images and binary data of any kind that needs to be processed by special
> purpose software.

Images are binary, and using ascii for absolutely everything a bit
powerful seems the wrong thing to me (especially as graphics/video and
audio are taking more and more space in computing), and that's a unix
legacy burden I guess.

Although you can consider the better you compress (at least lossily)
graphics and audio, the more “semantic” it get, as seeable for audio
compression algorithms specialized in voice or music (like do Opus), or
video compression algorithms that specialize to keep/render first what
human brain note, or certains techniques that “vectorize” graphics to
make them smaller.

> But those will be representable as a URL and/or hash value
> inside the data-graph.

URL implies client/server thus centralization. Hash-value can be used in
a P2P network as well, but are only for immutable data, otherwise you
would better need either a pair of a public-key and an uuid, or the hash
of something describing how a content is allowed to be updated using
public keys to design people identities.

> Of course there are numerous other means of representing all kinds of
> structured data but RDF seams quite fit for the purpose and many
> converters for all kinds of data to RDF already exist so I use it here
> as an example.

Can it work with binary data? is there anything simple explaining how
does it work?

> We call a _layouter_ a procedure that can return a layout for a specific type
> of data in a standardized way. If there is no user-defined layouter the
> `layout` procedure will try to find and call (`eval/apply` in LISP/Scheme or
> another scripting language, or a binary) the best suitable layouter based on
> simple semantic reasoning on the type of data the URL represents. For example,
> an URL could be of a RDFS-class "person", in wich case a procedure to display
> a person will be loaded (this will have to be present on your distribution,
> but please assume for a moment we have it). If no suitable layouter is found a
> generic layouter will be called (just displaying the RDF-Connections in
> sequential order using _Dublin Core : Label_-tags where possible). If the
> given size is too small `layout` will just return a symbol of the URL (which
> can be clicked on to reveal more information).

That sounds a lot like cascading stylesheets (I mean like CSS, but not
mandatorily CSS), in a turing-complete version.

Also what if the minimal size required is not large enough to contain
the URL? Showing URL seems bad to me because first a simple link is
enough (and you can copy/paste the address it links to anyway), and then
not only URL are getting unreadable anyway these day (with base64 hashes
or id in them, because nobody choose anymore urls hierarchies names on
big automated plateforms everybody uses) or we can directly use hashes
in order to make that P2P.

> The layouter returns _atoms of the layout_ and _URLs_ (with optionally
> specified layouters). Each atom and URL has a position and size.

Why should the outer layout be able to *specify* the layouter of inner
content? why not just recommand (with that being overwrittable then)?

> ### Atoms of the layout:
>
>   * Symbols: For example <4 unicode characters and a color--optionally
>   also a image. For each URL we will compute a symbol--if none is
>   specified--based on the RDFS class. Data types will replace file
>   types after all.

But an URL can return a layout that then return something else than a
symbol… so why “for each URL we will compute a symbol”?

Also why <4 characters? Let’s take arbitrarily large words. Or at least
reasonnable words (I think the longest word ever is greek and is two or
three lines long) or sub-word (parts of words that can get hyphenized).

>   * Text

How is that different of symbol?

>   * URL of an image

Then an image? how is it different?

>   * 2D framebuffer (dynamic images, any X window, video)

Why making video and animations a separate type from image? it looks to
me the error html4 and xhtml1 did before html5 invented <video>.

>   * Boxes: They can contain lists of all possible atoms of the
>   interface, including boxes. They can be rendered as sequential
>   lists, a flexible box layout or as a box containing the atoms
>   according to the coordinates produced by a procedure (or by hand),
>   resulting in a free-form drawing area.

Why not separating horizontal and vertical boxes like TeX (and in some
ways CSS with inline/block) does?

And whata about the spaces? is there a “space” atom? strechable spaces?
shrinkable spaces? I mean like TeX does. And what about lines and
borders? is it rendered like CSS (with border, border-radius, padding,
margin, etc.) or like LaTeX with complex combinations of individual
spaces and \rule?

Anyway making vectorial images an axiom would also be a wrong idea
because this layouting format should be able to produce arbitrarily
complex/weird vector graphics, so that vectorial pictures can contain
other stuff too and become modular too.

> For each returned URL `layout(URL, size)` will be called again.

again until what? (if I well understood “URL” outside the quotes reffer
to the same URL as the “URL” inside quotes).

> The recursive procedure described here is the heart of the new approach which
> I have called the "Glasperlenspiel". It will break down the data into
> displayable atoms with a size and position attribute. The recursive procedure
> stops if there is no screen space left or no data to be displayed any
> more.

Why not continuing to render when there’s no more space (like HTML/CSS
does with scrollable pages)?

> The beauty of this approach is that individual layouter procedures do not have
> to care about how to display all data they want to display. When it is not
> desirable to specify a behaviour it can leave that to the layouters of the
> individual parts of its data. For example the _person_ layouter might call an
> _adress_ layouter that displays an adress as text or if there is screen-space
> left also as a map. This _adrdress_-layouter can be called by other layouters
> souch as _shop_, _institution_, _cafe_ etc. So we can build a layout based on
> abstract definitions of layouters that build on other layouters.

That just sounds like any modular GUI/layout-engine to me…

>     define layout (node_URL, size (width, height), pos (x, y))
>     (
>      let layout_list (eval (select_layouter (URL, size)))
>        foreach l in layout_list
>          if atom? l
>            display l
>          else
>            layout l  
>     )

URL is undefined at line 3, node_URL is never used, what is the
“display” function for and why not directly returning “l”? and how does
work select_layouter (it seems to achieve the core of interesting work,
otherwise it’s just a simple recursive function)?

> The typographical projection of our data forms a tree which can be serialized
> and standardized. The actual rendering could be performed by a renderer
> suitable for your system. It could be either a nice Cairo/Pango screen layout
> (with the ability to include arbitrary X windows), ncurses or javascript in a
> webbrowser. Each renderer will have certain advantages and disadvantages that
> can be made transparent to and reflected by the layouter-procedures. If we
> look a bit into the future the implementations could include natural language
> SQL to find data, 3D space, braille displays and a whole lot more, the
> approach presented here is not limited to a 2D screen. The implementation for
> a 2D-screen will be called the typographical one.

That’s why I think expliciting the cascading approache and improving
user configuration power would be most useful. That’s why you don’t only
need a language/system to specify how to display stuff but also how to
edit it with the interface/display you provide. Otherwise it will not
empower users so that to become that revolutionary while a CSS-like you
can program it’s visualization AND editing *with* this CSS-like is a
more empowering (just as extending emacs with emacs makes it a lot more
powerful).

I *like* the idea to render stuff in 3D, a lot of stuff would get more
easy to represent graphically this way (schemas, graphs, etc. even
simply 3D stuff, like maps, objects, etc.).

> This would result in a highly modular system that can handle all kinds of data
> and combination of that data. It also will eliminate the need to switch
> between applications and load, save and then load the data again in a
> different application.

That sounds to me as the power of html/css without its crap and inside
something like Emacs.

> If the data is stored in RDF already we can define concepts like "inbox"
> easily in SPARQL. For mor complicated queries we will write small procedures.
> Like actors these _filters_ can be classified using RDF and made accessible to
> the user by menus, by loading something as a default on a home screen etc.
> They may be tied to a specific layouter or leave that selection to the system
> or the user.

We also really need something more simple and accessible than RDF, XML,
SPARQL, SQL, etc. because that’s way much research stuff and I don’t
think most users or beginning devs will handle/understand it (currently
I know none who does).

> From the **user's** point of view the system is more usable because data can
> be reused and rearranged in virtual all combinations. One doesn't have to
> "switch applications" any more. For example if you click on an image in a
> formated text document the system will display the commands to modify the
> image _just in that place_ where the image is in the text.

I don’t see the difference from web. Except it would maybe has a chance
not to become horrible crap (thank to lisp), not being took over by big
proprietary-software companies (thank to lisp), and not derivating from
Tim Berner Lee original’s idea of read-write decentralized web (without
even client/server), where everything is much like an extensible
decentralized wiki.

> Also object-oriented GUI systems like
> NextStep (GNUStep) provide the possibility to re-use parts of the UI. I think
> of the new approach as of pushing this concept to the extreme.

I’m not sure this has anything to do with OOP. OOP is just *a way* to do
it. I find lisp way enough powerful for not needing OOP (though anyone
can easily implement it).

> If we define the `size` not in pixel but in characters we can have a
> user defined base font size for the whole system, which will be nice
> for visually impaired people.

There’s a unit for that in CSS, it’s “rem” (root em, the em of the
document root font, and an em is the largeness of a capital M in the
current font).

> And finally because the layouters are defined for different sizes it will be
> easy to deploy the system to platforms with different screen sizes ("mobile").

That’s not enough. On smaller surface I can’t see how you can
deterministically find what size to give to stuff, especially with
absolute space… you should use relative unit sometimes… or even more
semantic information instead sometimes…

In CSS the size propagates bottom-top (from inner units to outer
units, except for screen/document), not top-bottom (from screen to each
unit), except you say so.

> From the **developer's** point of view it will be easier and faster to write
> applications to display special data because one can draw from a wide array of
> existing functions for the sub-structures your data consists of. If one
> invents a new distributed communication protocol for instance, one could just
> put the data into the store and let the layouters for chat or messages that
> already exist do thier job.

In some ways it already exists since multiprotocol chatting software
already exist. See libpurple/pidgin, Emacs, etc.

> Also I expect the difference between the developer and the user to blurr
> increasingly.

Like Emacs you mean?

> If the user can easily configure the system by selecting and
> rearranging layouters and actors he is in fact acting as some kind of
> developer. Another hint in this direction is maybe that when I imagine
> the system described here it it looks a bit like the morphic UI for
> the smalltalk system that was designed to teach people programming.

For me it look a lot like Emacs. But I have yet to learn Smalltalk…

> If we allow that a layouter can be any existing traditional X window
> application that takes a URL as an argument (or by writing a small wrapper it
> can be all kinds of applications) the system would be backward compatible
> comparable to a Window User Interface that is backward compatible to the text
> terminal by displaying a terminal emulator.

We can be backward compatible with terminal without passing through Xorg
(which seems quite heavy te me for just that). Also X has several
problems nodaways in regard to that: you can't display a single window
twice on screen (I don’t know if you can effectively put the same window
on several workspace with an X WM), and I think you can't draw an X
window inside another X window (otherwise if I’m wrong it would be so
great!!!).

> So for data that we don't have a layouter (yet) in the Glasperlenspiel
> System we always can fallback to the traditional application. This
> will only be easily possible on a GNU/Linux System as the GUI has to
> mimic a window manager for the layout-boxes that contain X windows.

On UNIXes system to be precise. Proprietary UNIXes as Mac OS X and BSDs
could still do that. But since our main proprietary competitor is
Windows that would be a great argument to more easily fight against it,
especially in the mobile area (yet the problem is that Windows doesn't
have its power because it is well made but because it has financial
partnerships that makes it installed by default everywhere and makes
everything else more hardly installable, so we can only make GNU/Linux
more famous and attractive).

> The complexity of writing a first version of the system can then be further
> reduced by using existing applications for specific purposes. For example in
> each case the user wants to edit text we could display the user's favourite
> text editor eliminating the need to write a sophisticated text input widget
> and still have all the advantages of an editor like emacs in every place you
> enter text.

Yeah… except everything of that sound really like UNIX philosophy of a
simple program specialized for each simple stuff, and then you would
better use vim… because Emacs is not really an editor but a true
development environment, or even a complete shell (in the original
meaning, otherwise emacs already *contain* a shell). A better suited
approach would be to make emacs more graphical (like Guile’s Emacsy I
guess) and implement your system inside emacs (btw, fyi, emacs already
features an X WM)… if Guile, Emacsy and Emacs don’t merge before (I
don’t know how likely is this, because I’m a lot biased by the fact I’d
love guile and emacs to merge and because people started trying to do
it).

> ## Conclusion
>
> The system will of course be impossible to build by writing it from scratch.
> But I think it may be possible to use existing free software like a
> programmable tiling window manager, a RDF library, a script language
> interpreter, a graphical toolkit etc.

Emacs does everything of that I guess. Guile too maybe, or otherwise it
can be done easily.

It’s been a long time you sent this email and nothing happenned since,
did you abandoned the idea or…?