Wayland fractional scaling incorrectly reports display resolution to VirtualBox

[matt303]

Description:

When using fractional scaling (125%, 150%, 175%) on Wayland (GNOME Mutter), VirtualBox incorrectly detects the resolution of the host system. Instead of using the native resolution, it reads the scaled framebuffer resolution, which is larger than the actual screen size.

This results in incorrect guest resolution detection, causing display scaling issues when running VirtualBox in full-screen mode.

Example Issue:

Host Native Resolution: 2560x1600

Wayland 150% Scaling Renders At: 3408x2130

VirtualBox Reads: 3408x2130 (instead of 2560x1600)

This incorrect resolution prevents proper full-screen behavior and causes display scaling problems in the guest.

Steps to Reproduce:

Use Wayland or Xorg as the display server.

Enable Fractional Scaling with:

gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'xwayland-native-scaling', 'x11-randr-fractional-scaling']"

Set Scaling to 150% (GNOME Settings > Displays > Scale 150%).

Open a terminal and run:

xrandr | grep connected

Expected Output: eDP-1 connected primary 2560x1600+0+0

Actual Output: eDP-1 connected primary 3408x2130+0+0 (wrong, too high)

Start VirtualBox and enter full-screen mode.

VirtualBox incorrectly detects the resolution as 3408x2130 instead of 2560x1600.

Wayland does not support true fractional scaling natively. Instead, it renders the entire screen at a higher resolution and then scales it down for smooth UI scaling.

The scaled framebuffer resolution is larger than the actual screen resolution because it is upscaled first before being downsampled.

This issue does not happen with integer scaling values (100%, 200%) since no intermediate framebuffer is created.

Expected Behavior:

VirtualBox should detect and use the native resolution (2560x1600) instead of the framebuffer resolution (3408x2130).

Guests should scale properly to full-screen without extra borders or incorrect resolutions.

Actual Behavior:

VirtualBox incorrectly detects a much higher resolution (e.g., 3408x2130 instead of 2560x1600).

This results in incorrect guest display scaling and prevents proper full-screen mode.

Additional Information:

This issue does not happen on a Windows 11 host running on the same hardware with 150% scaling.

100% and 200% scaling work fine, but fractional scaling (125%, 150%, 175%) causes the issue.

Tested with both Wayland and Xorg and the same problem persists.

Forcing a custom resolution in VirtualBox (VBoxManage) does not fix this.

Auto-Resize Guest Display also does not fix this and will apply the upscaled resolution.

Possible Solutions:

Provide a way to detect the actual screen resolution and ignore Wayland’s scaled framebuffer resolution.

Allow VirtualBox to bypass Wayland’s scaling misreporting when in full-screen mode.

Improve XWayland support for fractional scaling in VirtualBox.

Debugging Commands:

To confirm the issue, run the following:

xrandr | grep connected

If fractional scaling is enabled, the output will show an incorrect resolution (higher than native), which VirtualBox will also use incorrectly.