Re: Debugging coredumps in different computers

[Paul Pluzhnikov]

"Bruno Gonzalez (STenyaK)" <stenyak@gmail.com> writes:

> I'm new to debugging using core dumps. I've managed to get core dumps
>  + symbols using g++ and gdb under linux.
>  I compile using g++ -g -dH, and i debug using  gdb executableFile -c
>  coredumpFile.
>
> This is a test program backtrace in the original computer (a gentoo
...
>  #0  0xffffe410 in __kernel_vsyscall ()
>  #1  0x4017a101 in raise () from /lib/libc.so.6
>  #2  0x4017b8e8 in abort () from /lib/libc.so.6
>  #3  0x400ed794 in __gnu_cxx::__verbose_terminate_handler () from /
> usr/
>  lib/gcc/i686-pc-linux-gnu/4.1.2/libstdc++.so.6
>  #4  0x400eb1b5 in ?? () from /usr/lib/gcc/i686-pc-linux-gnu/4.1.2/
>  libstdc++.so.6
>  #5  0x400eb1f2 in std::terminate () from /usr/lib/gcc/i686-pc-linux-
>  gnu/4.1.2/libstdc++.so.6
>  #6  0x400eb2aa in __cxa_rethrow () from /usr/lib/gcc/i686-pc-linux-
> gnu/
>  4.1.2/libstdc++.so.6
>  #7  0x08049047 in main () at main.cpp:46

This core dump came from uncaught exception (but you probably
know that).

> And this is what i get after i scp the exe and dump to a different
>  computer (a kubuntu box):
>  (gdb) bt
>  #0  0xffffe410 in __kernel_vsyscall ()
>  #1  0x4017a101 in modff () from /lib/libc.so.6
>  #2  0x4017b8e8 in sigorset () from /lib/libc.so.6
>  #3  0x400ed794 in ?? ()
...
> As you can see, the backtraces differ, and i lose almost all symbols.
>  What am i doing wrong?

You aren't necessarily doing anything wrong.

The core dump contains memory image of the various writable (data)
segments of the process, but it does not contain copies of the
executable code (read-only sections), to save space [1].

Because of that, gdb backtrace requires access to the *exact*
version of all executable sections that were used when the core
dump was produced. This means that you need to tell gdb where to
find the main executable *and the shared libraries* which were used
by the running process, so it can extract symbol table from them,
and decode addresses in the core-dump into symbol names.

Without matching shared libraries, gdb just uses whatever libraries
are installed on the current system, if these don't match what's
in the dump, you get "garbage".

So, copy all the shared libraries (use gdb "info shared" to find
what they are) from the "original" computer to a subdirectory on
the second one (DO NOT overwrite /lib/libc.so.6 or any other libs
on the second computer -- that will likely render it unbootable),
and tell gdb to use them with gdb "solib-absolute-prefix". From
"info gdb":

`set solib-absolute-prefix PATH'
     If this variable is set, PATH will be used as a prefix for any
     absolute shared library paths; many runtime loaders store the
     absolute paths to the shared library in the target program's
     memory.  If you use `solib-absolute-prefix' to find shared
     libraries, they need to be laid out in the same way that they are
     on the target, with e.g. a `/usr/lib' hierarchy under PATH.

For example, if "info shared" says:

    From        To          Syms Read   Shared Object Library
    0x004880c0  0x0048d590  Yes         /lib/tls/librt.so.1
    0x00b74890  0x00c56f48  Yes         /lib/tls/libc.so.6
    0x00daa2f0  0x00db2608  Yes         /lib/tls/libpthread.so.0
    0x00b477a0  0x00b58a47  Yes         /lib/ld-linux.so.2

then do this:

  # on "original" computer:

    cd / 
    tar czf /tmp/to-copy.tgz \
       path/to/exe path/to/core lib/tls/librt.so.1 \
       lib/tls/libc.so.6 lib/tls/libpthread.so.0 lib/ld-linux.so.2

  # on the "second" computer:
  
    mkdir /tmp/libs.orig; cd /tmp/libs.orig
    tar xvzf to-copy.tgz

    gdb path/to/exe
    set solib-absolute-prefix /tmp/libs.orig
    core path/to/core
    where                # should give reasonable back trace

Cheers,

[1] Solaris allows inclusion of executable sections into core
with coreadm(1), but I don't believe such capability exists in linux.
-- 
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.