Changeset 84988 in vbox for trunk/src/VBox/Debugger

Timestamp:

Jun 29, 2020 10:51:37 AM (4 years ago)

Author:

vboxsync

Message:

Debugger/DBGPlugInLinux: Fix detecting 32bit Linux guests when KASLR is used, DGFR3MemScan() returns VERR_DBGF_MEM_NOT_FOUND immediately if the search range exceeds the 4GB address space and the paging mode is not amd64, bugref:1098

File:

• trunk/src/VBox/Debugger/DBGPlugInLinux.cpp (modified) (3 diffs)

trunk/src/VBox/Debugger/DBGPlugInLinux.cpp

@@ -2825 +2825 @@
  * @param   pThis               The Linux digger data.
  * @param   pUVM                The user mode VM handle.
- * @param   uAddrKernelStart    The first address the kernel is expected at.
- */
-static bool dbgDiggerLinuxProbeKaslr(PDBGDIGGERLINUX pThis, PUVM pUVM, RTGCUINTPTR uAddrKernelStart)
+ */
+static bool dbgDiggerLinuxProbeKaslr(PDBGDIGGERLINUX pThis, PUVM pUVM)
 {
     /**
@@ -2854 +2853 @@
      * So the highest offset the kernel can start is 0x40000000 which is 1GB (plus the maximum kernel size we defined).
      */
-    if (dbgDiggerLinuxProbeWithAddr(pThis, pUVM, uAddrKernelStart, _1G + LNX_MAX_KERNEL_SIZE))
+    if (dbgDiggerLinuxProbeWithAddr(pThis, pUVM, LNX64_KERNEL_ADDRESS_START, _1G + LNX_MAX_KERNEL_SIZE))
+        return true;
+
+    /*
+     * 32bit variant, makes sure we don't exceed the 4GB address space or DBGFR3MemScan() returns VERR_DBGF_MEM_NOT_FOUND immediately
+     * without searching the remainder of the address space.
+     *
+     * The default split is 3GB userspace and 1GB kernel, so we just search the entire upper 1GB kernel space.
+     */
+    if (dbgDiggerLinuxProbeWithAddr(pThis, pUVM, LNX32_KERNEL_ADDRESS_START, _4G - LNX32_KERNEL_ADDRESS_START))
         return true;
 
@@ -2918 +2926 @@
 
     /* Maybe the kernel uses KASLR. */
-    if (dbgDiggerLinuxProbeKaslr(pThis, pUVM, LNX32_KERNEL_ADDRESS_START))
-        return true;
-
-    if (dbgDiggerLinuxProbeKaslr(pThis, pUVM, LNX64_KERNEL_ADDRESS_START))
+    if (dbgDiggerLinuxProbeKaslr(pThis, pUVM))
         return true;