Re: [Qemu-devel] [PATCH] Support VNC PointerTypeChange psuedo-encoding

[Anthony Liguori]

Ramesh Dharan wrote:
> Hi Anthony, sorry for the delayed response to your delayed response.
>   

No problem.

> I actually haven't been working much on the remote display stuff in a while,
> though it's something I'm hoping to get involved with again time permitting.
> I'm cc'ing Dustin Byford and Johnson Liu, VMware engineers who handle a lot
> of our remote display work these days. 
>
> Also, I'm wondering if it would make sense to set up a separate list so we
> don't spam qemu-devel with this discussion, and also so that we can all
> subscribe to it rather than having to be cc'ed and so forth. If you think
> that's a good idea let me know. We can host the list @vmware.com, or put it
> somewhere else (nongnu.org? or elsewhere?) if that would be preferred. 
>   

I'll send you a note and we can figure out the best place to host it but 
I do think it's a good idea.

> > > all the scenarios we care about)
> > >       
> > Yes, right now, we do not address this.
> >     
>
> So how do you handle the keyboard today? Do you just hardcode the reverse
> mapping back to a pc105 keyboard? 
>   

We can take a keymap argument which can be used to map to various types 
of keyboards (defaulting to a standard pc105 keyboard).

> It would be nice to get something upstream so that all clients have the
> ability to send scancode values. We have an extension in place which we can
> just document. We use 16-bit "virtual scan code values" (they're 16-bit more
> for convenience of implementation, i.e. directly mapped to a lookup table). I
> believe the values we actually are equivalent/identical to PC scancodes. 
>   

Yeah, that would work out pretty well.

> Meanwhile, in our newest version of Workstation (6.0) we added server-side
> support for emulating different keyboard layouts when standard VNC clients
> connect. Basically you can set a server config option to tell the server
> "pretend that a de105 keyboard is attached" and it will do the reverse
> translation of KeySym to Scancode accordingly. This is good enough for some
> people, but the ideal situation is obviously to do the translation on the
> client-side where you can actually detect the local keyboard and do the right
> thing (which is what the VMware remote client does). 
>   

Yes, this would be much nicer.

 

> > There is a pseudo-encoding that provides alpha cursors with a 1-bit 
> > alpha channel.  An 8 bit alpha channel would be nice of course.
> >     
>
> More than nice, it's pretty much required if you want to draw some modern
> Linux guest cursors correctly. The cheap hack of treating < 255 alpha as
> fully transparent doesn't really work that well, especially for things an
> I-Bar cursor with shadow. 
>   

Right, I've seen this in action and it's not very pretty.

> We do support the standard VNC pseudo-cursor encoding as of Workstation 5.5
> and ESX 3.0, and if the guest sets an alpha cursor and the client doesn't
> support our alpha cursor extension, we strip out the alpha channel and send
> it using the standard VNC encoding. But it doesn't look that great. 
>
>   
> > > (4) no dynamic pixel depth/bpp change support
> > >   
> > >       
> > This is something I do want to fix.  This isn't just a server-side 
> > issue, there is also a race in the protocol when a client uses 
> > SetPixelFormat.  A proper notification of when the server 
> > switches pixel 
> > format after a SetPixelFormat would solve this problem.
> >     
>
> So yeah in the original protocol, depending on how you read it, it's just
> illegal to ever send a SetPixelFormat after the original request. But at any
> rate, client->server format requests aren't as interesting as server->client
> requests. You only do the former if the client decides for some reason that
> it wants a different pixel format, because e.g. it decides it doesn't have
> enough bandwidth or something. 
>
> Server->client changes OTOH, can be guest driven, e.g. there are several mode
> changes when booting a Windows guest. When a standard VNC client is
> connected, we hide these from the client by doing pixel format conversion on
> the server, which is expensive, and potentially wastes bandwidth on the wire
> as well (e.g. when the guest is in 640x480x256 but the client wants 32bpp).
> When a client which supports our "VMModeChange" extension is connected, we
> just send down the mode change at the start of the next UpdateRequest. 
>   

Right now, we always do pixel conversion which is pretty annoying.  It 
would be very nice to be able to tell the client to deal with the 
conversion.

> We do implement the standard VNC desktop size encoding (since otherwise we'd
> have to just drop the connection every time the guest changed display size). 
>
> But getting back to your original point, since our client never sends mode
> change requests upstream, so we haven't addressed the race that you describe.
> I agree that if you ever want to drive these on the client you need to do
> something about that. 
>   

Right, we're particularly prone to this race since our VNC server is 
asynchronous.  A SetServerPixelFormat message would solve this problem 
since it would let a client know when it should start decoding pixels 
differently.

In practice, most VNC servers do not support arbitrary pixel formats too 
so it makes a lot of sense for server's to have this message available 
to them.

> > > (5) No notion of multiplexing displays on a single port. 
> > >   
> > >       
> > Right.  ATM, I don't really plan on addressing this.
> >     
>
> So, the old Connectix Virtual Server product (which was in beta when
> Microsoft snapped them up) had a super slick way of dealing with this, that
> we actually joked about doing but never seriously thought anyone would do.
> They rendered an entire "picker" UI using VNC primitives. The UI would show
> live thumbnails of each VM, and then when you click on one of the tiles,
> you'd get redirected to that console. You couldn't go back out to the picker,
> but still, it was pretty neat. 
>
> I've since looked into doing the same thing, but the closest thing to any VNC
> UI/widget rendering library that I've found is TclRFB (which, as the name
> would imply, is written in Tcl). 
>
> Anyway, what you really want is a way to say "you've connected to a
> multi-home server, here's a list of VMs available (with arbitrary or perhaps
> structured metadata attached to each VM tag), which one do you want?". And
> then have the ability for the client ot pick. But you can't really do this
> sanely with just extensions, I think this is a good candidate for actually
> extending the base protocol.
>   

I have seen KVM switches that use VNC implement an extension that is 
basically, SetName.  They then use a client encoding to switch the KVM.  
This is definitely an interesting area to pursue.

> > The idea is that on localhost, the server can render directly to a shared 
> > memory segment that the client also maps.  If the one is smart about how 
> > this shared memory segment is chosen, it can be used as an XShmImage.
> >     
>
> Yeah we did the exact same thing, with a slight twist. We get the server to
> allocate the SHM segment, and the client just attaches to it. If you do it
> this way, you don't need a new client->server message, the server just sends
> the client the shm id if the client supports the "SharedFB" extension (that's
> what we named it).
>   

I chose to have the client allocate the memory since it knew the X 
server's line size which is required to allocate an XShmImage.  My first 
implementation had the server allocate the buffer but required the 
client to tell the server what the line size should be.

> There's some complexity involved with how the server can actually figure out
> whether the client is on the same machine (e.g. if the client is tunneling
> over SSH, this is not easy to do). So in fact what we do is just if the
> client supports SharedFBInfo and SharedFBRect, the server just sends along
> the messages. The client then tries to map the segment described in
> SharedFBInfo. If it fails, it just sends a new encoding set with those
> encodings removed, and the server starts sending real pixel data instead. 
>   

I've thought about this problem too.  My fear is that one could 
accidentally map a shared memory area that's actually valid.

> Yes, this means that we always try and fail sharedfb so there's a couple
> extra roundtrips before remote clients actually get real pixel data. This was
> deemed acceptable (by me ;-) for our use cases since most networks are fast
> enough. But if this is too hackish for your taste we could bake in something
> better. 
>   

I really haven't thought of a better solution myself.  That doesn't mean 
one exists though.

> > I believe http://wiki.multimedia.cx/index.php?title=VMNC 
> > covers some of 
> > your extensions right?  Have you gotten the appropriate encodings 
> > reserved in the RFB spec?  I don't want to support any encodings that 
> > aren't properly registered so this would be an important first step.
> >     
>
> So once upon a time, we tried to register encodings, but we didn't get very
> far. It sounds like perhaps the process has improved since then. I will try
> to get the ball rolling on this again. 
>   

I can help you with that if you want.  It wasn't a painful process at 
all for me.

> > I'm certainly happy to work toward a common set of VNC extensions to 
> > cover virtualization.  It's in everyones benefit to ensure that the 
> > largest number of clients Just Work out of the box.
> >     
>
> Agreed. I look forward to hearing more of your thoughts on this subject.
>   

Same here!  Thanks for the response.

Regards,

Anthony Liguori

> --
> Ramesh
>
>