Re: [PATCH 1/4] sm501: Fix bounds checks

[BALATON Zoltan]

On Mon, 15 Jun 2020, Peter Maydell wrote:
> On Sat, 6 Jun 2020 at 20:22, BALATON Zoltan <balaton@eik.bme.hu> wrote:
> > We don't need to add width to pitch when calculating last point, that
> > would reject valid ops within the card's local_mem.
> >
> > Fixes: b15a22bbcbe6a78dc3d88fe3134985e4cdd87de4
> > Signed-off-by: BALATON Zoltan <balaton@eik.bme.hu>
> > ---
> >  hw/display/sm501.c | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
>
> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
>
> The calculations for sb/se/db/de all have a term which
> multiplies by (width + pitch), which makes me suspect
> they also need a similar fix ?

Maybe. I'll have to check again. Actually is there a simpler way to check 
if two rectangles overlap when they are given with base, x, y, w, h, pitch 
where base is the first byte of the screen, pitch is length of one line 
and x,y is coordinates of top left corner and w,h is dimensions of the 
rect. Now that I think about it we also need to take into accounf the 
bytes per pixel value (1 << format) because base is given in bytes while 
others are in pixels so these formulae likely need some fixes. Pixman has 
some functions for these but those assume common base so to use those we 
would need to bring the two rectangles to common base which I could not 
find out how to do. Probably this is really simple for someone who already 
did a lot of these before.

Regards,
BALATON Zoltan