Re: [Qemu-devel] [Bug 893208] Re: qemu on ARM hosts can't boot i386 image

[Laszlo Ersek]

On 09/19/15 12:54, Marina Kovalevna wrote:
> Hello boyos,
> 
> I got myself an Rpi2 recently and have been reading up on qemu.
> 
> Does this mean there's a problem booting x86 images on Arm or just the
> ones from that particular source?
> 

The outlandishness of this use case (-> buy an underpowered toy, run x86
programs on it via *emulation*) is so exceptional that it tickled my
fancy and I looked into it.

I'm CCing Peter and Dave; I can see in the LP comments from 2011 that
they looked at this in 2011-2012. We're going to have a good chuckle
here I promise.

So, Dave was correct in comment #1
<https://bugs.launchpad.net/qemu-linaro/+bug/893208/comments/1> where he
wrote,

"it falls with a triple fault from a divide instruction not long after a
load of rdtsc stuff".

I booted the same Debian i386 Squeeze (and Wheezy) "standard" images
from Aurelien's website as everyone else, in TCG mode, both on an x86_64
host, and -- "brace for impact" -- on an aarch64 host (APM Mustang). The
command line was simple,

$ qemu-system-i386 -hda debian_squeeze_i386_standard.qcow2

The symptoms reproduced on the aarch64 host, and didn't on the x86_64 host.

Then I added

  -d in_asm,op,int,exec,cpu,mmu,cpu_reset,ioport,unimp,guest_errors

to capture the TCG logs, up to the point where grub rebooted (vs. didn't
reboot, on the x86_64 host), and then diffed the logs between each
other. (This wasn't so fast, on the aarch64 host, approx. 530 MB of log
was written before the reboot.)

Looking at the logs, I can confirm Dave's analysis from 2011 -- there's
a CPUID, then an RDTSC, then a division by zero.

So if you look at GRUB's code, you find the calibrate_tsc() function in
"grub-core/kern/i386/tsc.c". (We're old friends with that function.) It
calls grub_get_tsc() -- same file --, which explains both CPUID and
RDTSC. (CPUID is only used for serialization, ie. for preventing the CPU
from executing RDTSC out-of-order. RDTSCP would be an alternative, which
combines both, but that's not as widely available.)

Where does the division by zero come from then? Well grub fetches and
stashes the TSC, then programs the PIT to sleep for some time, then
re-fetches the TSC, and uses the TSC difference as denominator when
calculating the "TSC rate". (It has a solid idea of the real time
passed, due to the PIT frequency being a given.)

Let's see where the TSC values come from in QEMU / TCG:

helper_rdtsc()             [target-i386/misc_helper.c]
  cpu_get_tsc()            [hw/i386/pc.c]
    cpu_get_ticks()        [cpus.c]
      cpu_get_real_ticks() [include/qemu/timer.h]

Now, the cpu_get_real_ticks() implementation is *host* specific. You can
find it implemented for a bunch of host architectures in
"include/qemu/timer.h".

Neither ARM nor AARCH64 qualify though; for those, the following
pearlescent fallback gets built:

> /* The host CPU doesn't have an easily accessible cycle counter.
>    Just return a monotonically increasing value.  This will be
>    totally wrong, but hopefully better than nothing.  */
> static inline int64_t cpu_get_real_ticks (void)
> {
>     static int64_t ticks = 0;
>     return ticks++;
> }

Note that this code dates back to the following commit (year 2006):

commit 46152182100e68f7f8aa4954af1bf91160bb3d15
Author: pbrook <address@hidden>
Date:   Sun Jul 30 19:16:29 2006 +0000

    Rewrite Arm host support.

So... the frequency of the PIT is 1193182 per second (see PIT_FREQ in
QEMU, and GRUB_SPEAKER_PIT_FREQUENCY in GRUB). Grub sleeps for 65535
such cycles in calibrate_tsc(), between the two TSC reads. That's
approximately 55 milliseconds. And for that long period, grub finds that
the TSC has incremented by ... one.

(Side remark: you'll find that recent grub versions don't choke on this.
See <http://git.savannah.gnu.org/cgit/grub.git/commit/?id=2e62352b>,
from January 2015.)

I applied the following extremely sophisticated patch (with the motto
"it cannot get more wronger"):

> diff --git a/include/qemu/timer.h b/include/qemu/timer.h
> index 9939246..def22de 100644
> --- a/include/qemu/timer.h
> +++ b/include/qemu/timer.h
> @@ -1003,8 +1003,7 @@ static inline int64_t cpu_get_real_ticks(void)
>     totally wrong, but hopefully better than nothing.  */
>  static inline int64_t cpu_get_real_ticks (void)
>  {
> -    static int64_t ticks = 0;
> -    return ticks++;
> +    return get_clock();
>  }
>  #endif
>

get_clock() is CLOCK_MONOTONIC based, has (theoretical) nanosecond
resolution, and a nice flat int64_t encoding that should suffice for
approx. 329 years. This should provide grub with a larger denominator.

This "fix" allowed me to boot the i386 Debian image on the AARCH64 host.

For a real fix... I think on AARCH64 hosts at least, a "real" cycle
counter should be available, and someone who knows AARCH64 could write a
function that fetches it.

For 32-bit ARM, I presume the Raspberry Pi 2 and the Odroid C1 are
advanced enough for a similar cycle counter reading function.

I wonder though if ARM platforms remain in existence for which the 2006
patch captures the right hardware capability. For those (and perhaps as
a general fallback) I think my "patch" above would be an improvement.

Peter?

Thanks
Laszlo