Is this a new idea? A conceptual essay on user interface design

[lama]

I wrote the essay in html and converted it using html2text by Aaron Swartz.

The html-version can be found here:
http://lama.musca.uberspace.de/glasperlenspiel/research_proposal.html



# The Glasperlenspiel Universal Interface

**A proposal for a new approach to human-computer interfaces**

_Lukas Seiß, Weißenbrunn--Berlin, 5th of October 2015_

The idea proposed here is the result of a long and mostly conceptual research
in user interface design over the last years that I started as I was
increasingly dissatisfied by modern interfaces used at home and in hospitals.
I recently graduated from Medical School and I had to spend long hours looking
at the user interfaces in hospitals during the last years and I also have a
background in programming so I have the perspective of both the user and the
developer here.

I very recently discovered a strange new concept that I would like to share
with you for a number of reasons: First, I'd like to know if anything like
this has ever been attempted and maybe failed. I have every intention of
writing such a system and I'd like to learn from the experience of others.
Second, if it is indeed a new idea, and I believe it is because in a year of
research on the topic of user interface design I never encountered such a
system, I want to make the idea public as soon as possible to prevent some
non-free software producing entity to have the same idea and patent it while I
still struggle with the prototype. If you think I should publish it on
someplace else than this mailing list, for that purpose, please let me know.

Also I'm not a professional developer, just a hobbyist, and so far just did
the research on my own. By taking a free software approach I hope there could
be a open discussion of the ideas presented here. I'll publish my thoughts on
how to design the first implementation and the code I produce on a public git
archive shortly.

Any feedback is welcome.

## Abstract

The goal is to write a system that is capable of displaying every kind of data
and manipulating it in every way the user wishes--and hopefully in a coherent
way. That is of course impossible as different kinds of data need different
interfaces and we all know that you have to write a new interface for every
single purpose. It is a well known fact in interface design that there is no
"perfect universal interface".

But what if we could break down the data to atoms that can be easily rendered
by a series of abstractions like we do all the time in computer languages?
After all, most types of data really are just collections of other types of
data. Those abstractions could be collectively written and shared in the
spirit of the free software movement resulting in the best possible concrete
approximation of the the stated abstract goal.

## Introduction

The universal interface is a system that allows arbitrary type-checked
combination of procedures (functions, binary programs...) and data (a shared
RDF store, a personal RDF store, traditional files) and is able to produce a
meaningful visual language that represents the concerning data and procedures
to the user.

That sounds like a ridiculously huge goal but please consider that the command
line of a POSIX system or the windowing user interface of modern operating
systems actually do provide the described service. The novelty of the approach
presented here is the means of how the graphical interface is constructed.

The approach presented here stems on the observation that the RDF data model
allows to build a data store for all the structured data you want to process
as all kinds of data can be projected to a semantic graph--with the exception
of images and binary data of any kind that needs to be processed by special
purpose software. But those will be representable as a URL and/or hash value
inside the data-graph. Of course there are numerous other means of
representing all kinds of structured data but RDF seams quite fit for the
purpose and many converters for all kinds of data to RDF already exist so I
use it here as an example. The problem with the graph is of course that if we
just render it as a graph with nodes and edges we will produce a image that is
enourmously large and impossible to comprehend. So, I propose the following.

## Recursive Layout Procedure

The idea is to display the data as follows: The recursive procedure `layout
(URL, size, [layouter])` is called for a data URL we want to display. We know
the position (x, y) on the screen already and we pass to the layout procedure
the URL we want to display and the size of screen area we have for it. The
`layouter` is optional and a certain type of procedure (see below).

We call a _layouter_ a procedure that can return a layout for a specific type
of data in a standardized way. If there is no user-defined layouter the
`layout` procedure will try to find and call (`eval/apply` in LISP/Scheme or
another scripting language, or a binary) the best suitable layouter based on
simple semantic reasoning on the type of data the URL represents. For example,
an URL could be of a RDFS-class "person", in wich case a procedure to display
a person will be loaded (this will have to be present on your distribution,
but please assume for a moment we have it). If no suitable layouter is found a
generic layouter will be called (just displaying the RDF-Connections in
sequential order using _Dublin Core : Label_-tags where possible). If the
given size is too small `layout` will just return a symbol of the URL (which
can be clicked on to reveal more information).

The layouter returns _atoms of the layout_ and _URLs_ (with optionally
specified layouters). Each atom and URL has a position and size.

### Atoms of the layout:

  * Symbols: For example <4 unicode characters and a color--optionally also a 
image. For each URL we will compute a symbol--if none is specified--based on 
the RDFS class. Data types will replace file types after all.
  * Text
  * URL of an image
  * 2D framebuffer (dynamic images, any X window, video)
  * Boxes: They can contain lists of all possible atoms of the interface, 
including boxes. They can be rendered as sequential lists, a flexible box 
layout or as a box containing the atoms according to the coordinates produced 
by a procedure (or by hand), resulting in a free-form drawing area.

For each returned URL `layout(URL, size)` will be called again.

The recursive procedure described here is the heart of the new approach which
I have called the "Glasperlenspiel". It will break down the data into
displayable atoms with a size and position attribute. The recursive procedure
stops if there is no screen space left or no data to be displayed any more.
The beauty of this approach is that individual layouter procedures do not have
to care about how to display all data they want to display. When it is not
desirable to specify a behaviour it can leave that to the layouters of the
individual parts of its data. For example the _person_ layouter might call an
_adress_ layouter that displays an adress as text or if there is screen-space
left also as a map. This _adrdress_-layouter can be called by other layouters
souch as _shop_, _institution_, _cafe_ etc. So we can build a layout based on
abstract definitions of layouters that build on other layouters.

**Note:**This looks a bit like a LISP-parser, i.e. the approach presended here 
resembles a 2D typographical language of boxes that contain boxes and 
atoms--and therefore a given layout result can be serialized as a tree in a 
standardizable way.

Ffirst abstract definition of the function in pseudocode:  

    
    
    define layout (node_URL, size (width, height), pos (x, y))
    (
     let layout_list (eval (select_layouter (URL, size)))
       foreach l in layout_list
         if atom? l
           display l
         else
           layout l  
    )
          

## Rendering

The typographical projection of our data forms a tree which can be serialized
and standardized. The actual rendering could be performed by a renderer
suitable for your system. It could be either a nice Cairo/Pango screen layout
(with the ability to include arbitrary X windows), ncurses or javascript in a
webbrowser. Each renderer will have certain advantages and disadvantages that
can be made transparent to and reflected by the layouter-procedures. If we
look a bit into the future the implementations could include natural language
SQL to find data, 3D space, braille displays and a whole lot more, the
approach presented here is not limited to a 2D screen. The implementation for
a 2D-screen will be called the typographical one.

## Manipulation of data

Because the data is present in a system-wide database we can manipulate it by
any procedure or program that we grant access to that database. In particular,
we will define small procedures for each action we want to perform. Those
_actors_ will be classified in the RDF graph itself so that we can display
menus and symbols for actions depending on context (for example if the user
selects a certain URL-Symbol the suitable actions for the RDFS-Classes of that
symbol will be displayed).

This would result in a highly modular system that can handle all kinds of data
and combination of that data. It also will eliminate the need to switch
between applications and load, save and then load the data again in a
different application. One could for example have a set of actions to
manipulate pixel data with multiple versions of the brush function that can be
modified and shared between users.

## Finding data

If the data is stored in RDF already we can define concepts like "inbox"
easily in SPARQL. For mor complicated queries we will write small procedures.
Like actors these _filters_ can be classified using RDF and made accessible to
the user by menus, by loading something as a default on a home screen etc.
They may be tied to a specific layouter or leave that selection to the system
or the user.

## Discussion

In my opinion such a hypothetical system has some advantages over traditional
window-based and web-based applications:

>From the **user's** point of view the system is more usable because data can
be reused and rearranged in virtual all combinations. One doesn't have to
"switch applications" any more. For example if you click on an image in a
formated text document the system will display the commands to modify the
image _just in that place_ where the image is in the text. Actually such a
system has been developed by Jef Raskin, it is called the "Canon Cat" and
described at length in his book "The Humane Interface". But the Canon Cat
lacked the possibillity to handle all kinds of Data. It could only perform a
limited set of pre-build actions and it had no paradigm like RDF to handle
differnt kinds of structured data. Also object-oriented GUI systems like
NextStep (GNUStep) provide the possibility to re-use parts of the UI. I think
of the new approach as of pushing this concept to the extreme.

The the system proposed here is also highly configurable as a whole, not just
individual applications like today, and hopefully still consistent as a whole
--this is a emergent property beacuse it is conceptualized on a higher level
of abstraction than the traditional ones. Everyone can set his own prefererred
layouters and actions. After all, each user has her own unique needs. For
example in a hospital setting we will have doctors, nurses and other staff
will have interface tailored to their needs (and the different speciality
branches in the hospital will want to have individual interfaces too) drawing
from a shared set of abstractions of patients and medical data. If we define
the `size` not in pixel but in characters we can have a user defined base font
size for the whole system, which will be nice for visually impaired people.
And finally because the layouters are defined for different sizes it will be
easy to deploy the system to platforms with different screen sizes ("mobile").

>From the **developer's** point of view it will be easier and faster to write
applications to display special data because one can draw from a wide array of
existing functions for the sub-structures your data consists of. If one
invents a new distributed communication protocol for instance, one could just
put the data into the store and let the layouters for chat or messages that
already exist do thier job. You could write whole new applications without
ever having to write a interface if the data used by your application is
already layoutable by the system. After all some developers hate developing
interfaces, so they can leave that job to other developers that like it now.

Also I expect the difference between the developer and the user to blurr
increasingly. If the user can easily configure the system by selecting and
rearranging layouters and actors he is in fact acting as some kind of
developer. Another hint in this direction is maybe that when I imagine the
system described here it it looks a bit like the morphic UI for the smalltalk
system that was designed to teach people programming.

If we allow that a layouter can be any existing traditional X window
application that takes a URL as an argument (or by writing a small wrapper it
can be all kinds of applications) the system would be backward compatible
comparable to a Window User Interface that is backward compatible to the text
terminal by displaying a terminal emulator. So for data that we don't have a
layouter (yet) in the Glasperlenspiel System we always can fallback to the
traditional application. This will only be easily possible on a GNU/Linux
System as the GUI has to mimic a window manager for the layout-boxes that
contain X windows.

The complexity of writing a first version of the system can then be further
reduced by using existing applications for specific purposes. For example in
each case the user wants to edit text we could display the user's favourite
text editor eliminating the need to write a sophisticated text input widget
and still have all the advantages of an editor like emacs in every place you
enter text.

## Conclusion

The system will of course be impossible to build by writing it from scratch.
But I think it may be possible to use existing free software like a
programmable tiling window manager, a RDF library, a script language
interpreter, a graphical toolkit etc. to write a first version it in a
reasonable timeframe. Because it will be a patchwork of existing software that
will result in a system that allows for an even bigger patchwork of small
programs to be build I could think of no better place to announce the start of
development than the mailing list of the GNU system itself.

I'm unable to list the numerous fine persons that inspired this idea here so
instead I want to dedicate this essay to Olia Lialina and Walter Benjamin, who
do not know me but I see in their work a expression of somemething similar
only in a different language. I hope they wouldn't mind.

  
Best,  
Lukas