FSTENV assembler instruction - wrong implementation

[Franjo Stipanovic]

Hi,

I believe VirtualBox has wrong implementation of FSTENV assembler instruction. It is not dumping last float-pointing instruction address correctly. I've created a simple application in assembler which can be used as proof.

Try and debug this application with debugger (OllyDbg for example) and take a look at buffer after the FSTENV instruction. It should contain address of FNOP assembler instruction.

format PE GUI
entry main

include 'win32a.inc'

section '.data' data readable writeable
	buffer db "                        ", 0

section '.text' code readable executable
main:
	nop
	mov eax, buffer
	fnop
	fstenv [eax]

	invoke ExitProcess, 0

section '.idata' import data readable
	library kernel32, 'KERNEL32.DLL'
	import kernel32, ExitProcess, 'ExitProcess'

You can compile the example code with FASM.

[Frank Mehnert]

I've tested this with several Linux guests (32-bit guest with/without VT-x, 64-bit guest) and it always worked correctly. Note that the instruction pointer starts at offset 12 of the buffer. Which content has the buffer in your case?

[Franjo Stipanovic]

I'm comparing this case with my windows host on which works fine. This is my buffer: 7F 02 FF FF 00 00 FF FF FF FF FF FF 29 FA C4 77 1B 00 7D 07 0C FF 55 01 23 00 FF FF

Instruction pointer is (starting at offset 12) 77C4FA29. It should be 00402006, that's where FNOP is located.

My host is Windows and guest is Windows also.

[michaln]

I tried the example on Windows host and guest, in both cases I got the expected result. FYI, you don't need any of the Windows API junk for your tiny example, a simple "ret" does the job.

To eliminate the possibility of a debugger interfering, I wrote the following tiny C program - not very portable, but that doesn't matter, it does the job:

#include <stdio.h>
#include <fenv.h>

int main( void )
{
    fenv_t  fenv;

    __asm fnop;
    fegetenv( &fenv );
    printf( "Last FP EIP: %08x\n", fenv.ins_ptr_offset );
    return( 0 );
}

The program prints something along the lines "Last FP EIP: 00401018" which is the address of the FNOP instruction. The result is identical on my Windows Server 2003 host and inside a Windows XP guest.

The address 0x77C4FA29 looks very much like an address inside a DLL. You should find out what is at that address. My guess is that the debugger you are using is broken and upsets the results. I strongly suggest that rather than using a debugger, you print the last instruction pointer value from the fstenv buffer.

[Franjo Stipanovic]

michaln, thanks for the feedback, I'll do the test and get back at you.

[Frank Mehnert]

If you can still reproduce this issue, please attach a VBox.log file from such a VM session so we can have a look at the VM configuration.

[Franjo Stipanovic]

michaln, frank, sorry for false alarm. When executed w/o debugger the IP is correct, but when in debugger - it's completely wrong. This is strange because debugger (OllyDbg) works on host OS and on other machines ...

Thanks for the response.