RE: [Qemu-devel] [PATCH] Support VNC PointerTypeChange psuedo-encoding

[Ramesh Dharan]

Hi Anthony, sorry for the delayed response to your delayed response.

I actually haven't been working much on the remote display stuff in a while,
though it's something I'm hoping to get involved with again time permitting.
I'm cc'ing Dustin Byford and Johnson Liu, VMware engineers who handle a lot
of our remote display work these days. 

Also, I'm wondering if it would make sense to set up a separate list so we
don't spam qemu-devel with this discussion, and also so that we can all
subscribe to it rather than having to be cc'ed and so forth. If you think
that's a good idea let me know. We can host the list @vmware.com, or put it
somewhere else (nongnu.org? or elsewhere?) if that would be preferred. 

> > (1) no support for keyboard scancodes (VNC uses higher 
> level symbolic keys
> > which aren't necessarily 1:1 mappable back to their 
> scancode equivalents in
> > all the scenarios we care about)
> 
> Yes, right now, we do not address this.

So how do you handle the keyboard today? Do you just hardcode the reverse
mapping back to a pc105 keyboard? 

It would be nice to get something upstream so that all clients have the
ability to send scancode values. We have an extension in place which we can
just document. We use 16-bit "virtual scan code values" (they're 16-bit more
for convenience of implementation, i.e. directly mapped to a lookup table). I
believe the values we actually are equivalent/identical to PC scancodes. 

Meanwhile, in our newest version of Workstation (6.0) we added server-side
support for emulating different keyboard layouts when standard VNC clients
connect. Basically you can set a server config option to tell the server
"pretend that a de105 keyboard is attached" and it will do the reverse
translation of KeySym to Scancode accordingly. This is good enough for some
people, but the ideal situation is obviously to do the translation on the
client-side where you can actually detect the local keyboard and do the right
thing (which is what the VMware remote client does). 

> > (2) no relative mouse support
> >   
> 
> We have an extension that addresses this.

As do we ;-). 

> > (3) no alpha cursor support
> >   
> 
> There is a pseudo-encoding that provides alpha cursors with a 1-bit 
> alpha channel.  An 8 bit alpha channel would be nice of course.

More than nice, it's pretty much required if you want to draw some modern
Linux guest cursors correctly. The cheap hack of treating < 255 alpha as
fully transparent doesn't really work that well, especially for things an
I-Bar cursor with shadow. 

We do support the standard VNC pseudo-cursor encoding as of Workstation 5.5
and ESX 3.0, and if the guest sets an alpha cursor and the client doesn't
support our alpha cursor extension, we strip out the alpha channel and send
it using the standard VNC encoding. But it doesn't look that great. 

> > (4) no dynamic pixel depth/bpp change support
> >   
> 
> This is something I do want to fix.  This isn't just a server-side 
> issue, there is also a race in the protocol when a client uses 
> SetPixelFormat.  A proper notification of when the server 
> switches pixel 
> format after a SetPixelFormat would solve this problem.

So yeah in the original protocol, depending on how you read it, it's just
illegal to ever send a SetPixelFormat after the original request. But at any
rate, client->server format requests aren't as interesting as server->client
requests. You only do the former if the client decides for some reason that
it wants a different pixel format, because e.g. it decides it doesn't have
enough bandwidth or something. 

Server->client changes OTOH, can be guest driven, e.g. there are several mode
changes when booting a Windows guest. When a standard VNC client is
connected, we hide these from the client by doing pixel format conversion on
the server, which is expensive, and potentially wastes bandwidth on the wire
as well (e.g. when the guest is in 640x480x256 but the client wants 32bpp).
When a client which supports our "VMModeChange" extension is connected, we
just send down the mode change at the start of the next UpdateRequest. 

We do implement the standard VNC desktop size encoding (since otherwise we'd
have to just drop the connection every time the guest changed display size). 

But getting back to your original point, since our client never sends mode
change requests upstream, so we haven't addressed the race that you describe.
I agree that if you ever want to drive these on the client you need to do
something about that. 

> > (5) No notion of multiplexing displays on a single port. 
> >   
> 
> Right.  ATM, I don't really plan on addressing this.

So, the old Connectix Virtual Server product (which was in beta when
Microsoft snapped them up) had a super slick way of dealing with this, that
we actually joked about doing but never seriously thought anyone would do.
They rendered an entire "picker" UI using VNC primitives. The UI would show
live thumbnails of each VM, and then when you click on one of the tiles,
you'd get redirected to that console. You couldn't go back out to the picker,
but still, it was pretty neat. 

I've since looked into doing the same thing, but the closest thing to any VNC
UI/widget rendering library that I've found is TclRFB (which, as the name
would imply, is written in Tcl). 

Anyway, what you really want is a way to say "you've connected to a
multi-home server, here's a list of VMs available (with arbitrary or perhaps
structured metadata attached to each VM tag), which one do you want?". And
then have the ability for the client ot pick. But you can't really do this
sanely with just extensions, I think this is a good candidate for actually
extending the base protocol.

> We actually can do this.  I've reserved the 255 client 
> message ID which 
> I'm using to multiplex various other messages.  

Yeah, we did this too. We use '127' as the client message ID, but it's
otherwise exactly as you describe. It's a nice performance boost, though it
makes implementing CopyRect encoding much trickier <g>.

> I currently use this to implement a shared memory pseudo-encoding for VNC.

> The idea is that on localhost, the server can render directly to a shared 
> memory segment that the client also maps.  If the one is smart about how 
> this shared memory segment is chosen, it can be used as an XShmImage.

Yeah we did the exact same thing, with a slight twist. We get the server to
allocate the SHM segment, and the client just attaches to it. If you do it
this way, you don't need a new client->server message, the server just sends
the client the shm id if the client supports the "SharedFB" extension (that's
what we named it). 

There's some complexity involved with how the server can actually figure out
whether the client is on the same machine (e.g. if the client is tunneling
over SSH, this is not easy to do). So in fact what we do is just if the
client supports SharedFBInfo and SharedFBRect, the server just sends along
the messages. The client then tries to map the segment described in
SharedFBInfo. If it fails, it just sends a new encoding set with those
encodings removed, and the server starts sending real pixel data instead. 

Yes, this means that we always try and fail sharedfb so there's a couple
extra roundtrips before remote clients actually get real pixel data. This was
deemed acceptable (by me ;-) for our use cases since most networks are fast
enough. But if this is too hackish for your taste we could bake in something
better. 

> I believe http://wiki.multimedia.cx/index.php?title=VMNC 
> covers some of 
> your extensions right?  Have you gotten the appropriate encodings 
> reserved in the RFB spec?  I don't want to support any encodings that 
> aren't properly registered so this would be an important first step.

So once upon a time, we tried to register encodings, but we didn't get very
far. It sounds like perhaps the process has improved since then. I will try
to get the ball rolling on this again. 

> I'm certainly happy to work toward a common set of VNC extensions to 
> cover virtualization.  It's in everyones benefit to ensure that the 
> largest number of clients Just Work out of the box.

Agreed. I look forward to hearing more of your thoughts on this subject.

--
Ramesh