MultiMonitor: Guest mode hints do not accept negative numbers for xOrigin in setModeHint.

[GilB]

The source of the bug is in VBoxHelpers.cpp (hlpResizeRect) it attempts to maintain monitor order on display resize by comparing the monitor rectangles after the resize and finding intersections between adjacent (according to the previous monitor order) rectangles.

The problem happens if the monitor order changes - for instance, suppose we have 2 monitors - 0 and 1.

---------- ---------
|   0    | |   1   |
---------- ---------

Suppose monitor 0 is 1024x768 starting at 0,0 and monitor 1 is 800x600 starting at 1024,0. Suppose we call setvideomodehint with monitor 1 being moved to -1024,0 in order to move it to the left of monitor 0.

What will happen now is that when we look for monitors X-positive adjacent to monitor 0, we'll find monitor 1. The current code to check whether the monitors now intersect (post potential resize) is: (assume iRect=0 and iNextRect=1)

int delta = paNewRects[iRect].right - paNewRects[iNextRect].left;

this will result in 1024 (right of monitor 0) - -1024 (new left of monitor 1) = 2048. next,

        if (delta != 0)
        {
            DDCLOG(("XP intersection right %d left %d, diff %d\n",
                     paNewRects[iRect].right, paNewRects[iNextRect].left,
                     delta));

            paNewRects[iNextRect].left += delta;
            paNewRects[iNextRect].right += delta;
        }

since delta != 0, we will now move monitor 1 on the x axis by 2048, which will lead to monitor 1 being in the exact same location before we tried to move it. (-1024 + 2048 = 1024) and we'll end up with the original monitor layout.

Note the processing of the mode hint event from line 1689 in the log file.

[GilB]

log file