Re: [Qemu-devel] dynamically linked binaries under sparc-linux-user

[Artyom Tarasenko]

On Sat, Jun 4, 2011 at 10:30 AM, Blue Swirl <address@hidden> wrote:
> On Sun, May 29, 2011 at 2:32 AM, Artyom Tarasenko <address@hidden> wrote:
>> On Thu, May 26, 2011 at 8:45 PM, Blue Swirl <address@hidden> wrote:
>>> On Tue, May 24, 2011 at 10:42 PM, Artyom Tarasenko <address@hidden> wrote:
>>>> Should it be possible to use dynamically linked binaries under
>>>> sparc*-linux-user?
>>>> Under qemu-system-sparc the Debian 4.08r1 initrd works fine, but:
>>>>
>>>> master$ sparc-linux-user/qemu-sparc -strace -L
>>>> ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>>>> 14004 uname(0x409ffbae) = 0
>>>> 14004 brk(NULL) = 0x00063000
>>>> 14004 access("/etc/ld.so.nohwcap",F_OK) = -1 errno=2 (No such file or 
>>>> directory)
>>>> 14004 mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0)
>>>> = 0x40a2c000
>>>> 14004 access("/etc/ld.so.preload",R_OK) = -1 errno=2 (No such file or 
>>>> directory)
>>>> 14004 open("/etc/ld.so.cache",O_RDONLY) = 3
>>>> 14004 fstat64(3,0x409ff500) = 0
>>>> 14004 mmap(NULL,195479,PROT_READ,MAP_PRIVATE,3,0) = 0x40a2d000
>>>> 14004 close(3) = 0
>>>> Segmentation fault
>>>>
>>>> The strange thing here is that it loads ld.so.cache. The guest fs
>>>> doesn't have it, but the host does:
>>>>
>>>> master$  ll ../../debian-4.08r1-initrd/etc/ld.so.cache /etc/ld.so.cache
>>>> ls: cannot access ../../debian-4.08r1-initrd/etc/ld.so.cache: No such
>>>> file or directory
>>>> -rw-r--r--. 1 root root 195479 2011-03-17 13:48 /etc/ld.so.cache
>>>>
>>>> Isn't this wrong?
>>>
>>> I'm not sure.
>>
>> Right. On a second thought, qemu is probably doing what is expected:
>> the syscalls are emulated, so the host libraries must be loaded.
>> Then the problem must be elsewhere. Here is the backtrace:
>>
>> master$  gdb sparc-linux-user/qemu-sparc
>> GNU gdb (GDB) Fedora (7.0.1-50.fc12)
>> (gdb) run -L ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>> Starting program: sparc-linux-user/qemu-sparc -L
>> ../debian-4.08r1-initrd/ ../debian-4.08r1-initrd/bin/busybox
>> [Thread debugging using libthread_db enabled]
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00000000601c49c2 in static_code_gen_buffer ()
>> Missing separate debuginfos, use: debuginfo-install
>> glibc-2.11.2-3.x86_64 libattr-2.4.44-1.fc12.x86_64
>> nspr-4.8.6-1.fc12.x86_64 nss-3.12.8-2.fc12.x86_64
>> nss-util-3.12.8-1.fc12.x86_64 zlib-1.2.3-23.fc12.x86_64
>> (gdb) bt
>> #0  0x00000000601c49c2 in static_code_gen_buffer ()
>> #1  0x00007fffffffd684 in ?? ()
>> #2  0x00007ffff4bc8728 in ?? ()
>> #3  0x00000000ffffffff in ?? ()
>> #4  0x0000000060029083 in tb_alloc_page (tb=0x40a2bc00, phys_pc=<value
>> optimized out>, phys_page2=1084406920) at exec.c:1214
>> #5  tb_link_page (tb=0x40a2bc00, phys_pc=<value optimized out>,
>> phys_page2=1084406920) at exec.c:1278
>> #6  0x000000006002a037 in tb_gen_code (env=0x6223f390, pc=1084305880,
>> cs_base=<value optimized out>, flags=<value optimized out>,
>> cflags=<value optimized out>)
>>    at exec.c:1004
>> #7  0x000000006002afe8 in cpu_sparc_exec (env1=<value optimized out>)
>> at cpu-exec.c:636
>> #8  0x0000000060005d50 in cpu_loop (env=0x6223f390) at linux-user/main.c:1008
>> #9  0x00000000600069d9 in main (argc=1646342192, argv=<value optimized
>> out>, envp=<value optimized out>) at linux-user/main.c:3533
>> (gdb)
>>
>> Any ideas?
>
> The logs would probably show the crashing instruction.

$ sparc-linux-user/qemu-sparc -g 1234 -L ../debian-4.08r1-initrd/
../debian-4.08r1-initrd/bin/busybox &
$ gdb-sparc
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x40a02c80 in ?? ()
(gdb) c
Continuing.

Breakpoint 1, 0x40a09a2c in ?? ()
(gdb) disas $pc-0x20,$pc+0x20
Dump of assembler code from 0x40a09a0c to 0x40a09a4c:
   0x40a09a0c:  srl  %l3, 0x1f, %g2
   0x40a09a10:  add  %g2, %l3, %g2
   0x40a09a14:  add  %l6, %g1, %l5
   0x40a09a18:  sra  %g2, 1, %l0
   0x40a09a1c:  add  %l0, %l0, %l1
   0x40a09a20:  add  %l1, %l0, %g1
   0x40a09a24:  sll  %g1, 2, %g1
   0x40a09a28:  add  %g1, %l6, %g1
=> 0x40a09a2c:  ld  [ %g1 + 0x14 ], %g2
   0x40a09a30:  cmp  %l4, %g2
   0x40a09a34:  bleu  0x40a09968
   0x40a09a38:  sethi  %hi(0x400), %g1
   0x40a09a3c:  clr  %l2
   0x40a09a40:  or  %g1, 0x2ac, %g1
   0x40a09a44:  b  0x40a09a84
   0x40a09a48:  ld  [ %l7 + %g1 ], %i3
End of assembler dump.
(gdb) info registers g1
g1             0xe0d8cff4       -522661900
(gdb) si

Program received signal SIGSEGV, Segmentation fault.
0x40a09a2c in ?? ()

I went through the arithmetical operations and got the same result for
%g1: 0xe0d8cff4 .

Is the user-emulation memory layout somewhere documented? It would be
interesting to know what are the address spaces for the libraries and
the user program itself .

>>> It could be possible to construct a blacklist of host
>>> files that may not be accessible or visible to the guest but that
>>> wouldn't very robust either. Chrooting into a 100% guest architecture
>>> system should work better.
>>
>> You mean some sort of mixed chrooting? At least some host libraries
>> must be visible to the guest as if they were native.
>
> The simplest way is to build a statically linked emulator, chroot to
> the guest system and execute the emulator. Then the only host
> executable should be QEMU itself. Maybe binfmt_misc can also be used.

Great idea, indeed! It works, thanks!  The breakpoint at 40a09a2c is
never reached in this case hit though, so I can't directly see where
the dynamically linked qemu derails.

And it looks that some functionality is missing:

#    chroot ../debian-4.08r1-initrd /qemu-sparc  -L . /bin/busybox ash
BusyBox v1.1.3 (Debian 1:1.1.3-4) Built-in shell (ash)
Enter 'help' for a list of built-in commands.

~ # echo 123
123
~ # ls
HELPME: master/target-sparc/cpu.h:625
ash: ls: Exec format error

And here it hangs. The comment says /* FIXME: Do we also need to clear
CF?  */ . I see no further references of CF in cpu.h - do yo know what
is meant here?

ls is the link to the same busybox, and it works when executed directly.

-- 
Regards,
Artyom Tarasenko

solaris/sparc under qemu blog: http://tyom.blogspot.com/