[Qemu-devel] [PATCH] Fix brk syscall error handling and mmap issues

[Richard Purdie]

When running certain memory heavy programs under QEMU ARM user emulation
I've noticed a QEMU segfault.

The segfault is occurring just after an mmap call on the host system.
The reason is that the target application tries to enlarge the data
segment over the top of the memory address where the qemu binary is
loaded using a brk syscall. The brk syscall is translated into an mmap
call which unmaps the qemu binary from memory after which it
unsurprisingly segfaults.

Initially I tested this by changing the load address of the QEMU binary
but that fix is less than ideal since its still possible to conflict.
I've attached a better which where the mmap code checks the call doesn't
touch "reserved" pages and returns an error if it does instead of
calling mmap. 

brk syscalls usually occur from within malloc within glibc. If they
fail, malloc should start using mmap to request memory instead. The
above error should therefore not be fatal and the application should
just stop making brk calls and use mmap but this isn't what was
observed, it continued to segfault somewhere else in the program.

I found that the error return values for the brk syscall were incorrect
and were causing this secondary segfault. In case of an error, the brk
syscall should return the original brk value, not any kind of error
code.

The attached patch fixes both these issues.

Cheers,

Richard

fix_brk.patch
Description: Text Data