Re: [Qemu-devel] [PATCH 4/9] hbitmap: store / restore

[John Snow]

On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote:
> Functions to store / restore HBitmap. HBitmap should be saved to linear
> bitmap format independently of endianess.
>
> Because of restoring in several steps, every step writes only the last
> level of the bitmap. All other levels are restored by
> hbitmap_restore_finish as a last step of restoring. So, HBitmap is
> inconsistent while restoring.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
> ---
>   include/qemu/hbitmap.h | 49 +++++++++++++++++++++++++++++
>   util/hbitmap.c         | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 133 insertions(+)
>
> diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
> index b645cfc..e57b610 100644
> --- a/include/qemu/hbitmap.h
> +++ b/include/qemu/hbitmap.h
> @@ -126,6 +126,55 @@ void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t 
> count);
>   bool hbitmap_get(const HBitmap *hb, uint64_t item);
>
>   /**
> + * hbitmap_data_size:
> + * @hb: HBitmap to operate on.
> + * @count: Number of bits
> + *
> + * Return amount of bytes hbitmap_store_data needs
> + */
> +uint64_t hbitmap_data_size(const HBitmap *hb, uint64_t count);
> +
> +/**
> + * hbitmap_store_data
> + * @hb: HBitmap to oprate on.
> + * @buf: Buffer to store bitmap data.
> + * @start: First bit to store.
> + * @count: Number of bits to store.
> + *
> + * Stores HBitmap data corresponding to given region. The format of saved data
> + * is linear sequence of bits, so it can be used by hbitmap_restore_data
> + * independently of endianess
> + */
> +void hbitmap_store_data(const HBitmap *hb, uint8_t *buf,
> +                        uint64_t start, uint64_t count);
> +
> +/**
> + * hbitmap_restore_data
> + * @hb: HBitmap to oprate on.
> + * @buf: Buffer to restore bitmap data from.
> + * @start: First bit to restore.
> + * @count: Number of bits to restore.
> + *
> + * Retores HBitmap data corresponding to given region. The format is the same
> + * as for hbitmap_store_data.
> + *
> + * ! The bitmap becomes inconsistent after this operation.
> + * hbitmap_restore_finish should be called before using the bitmap after
> + * data restoring.
> + */
> +void hbitmap_restore_data(HBitmap *hb, uint8_t *buf,
> +                          uint64_t start, uint64_t count);
> +

I guess by nature of how bitmap migration has been implemented, we 
cannot help but restore parts of the bitmap piece by piece, which 
requires us to force the bitmap into an inconsistent state.

Yuck.

It might be "nice" to turn on a disable bit inside the hbitmap that 
disallows its use until it is made consistent again, but I don't know 
what the performance hit of the extra conditionals everywhere would be like.

> +/**
> + * hbitmap_restore_finish
> + * @hb: HBitmap to operate on.
> + *
> + * Repair HBitmap after calling hbitmap_restore_data. Actuall all HBitmap
> + * layers are restore here.
> + */
> +void hbitmap_restore_finish(HBitmap *hb);
> +
> +/**
>    * hbitmap_free:
>    * @hb: HBitmap to operate on.
>    *

These are just biased opinions:

- It might be nice to name the store/restore functions "serialize" and 
"deserialize," to describe exactly what they are doing.

- I might refer to "restore_finish" as "post_load" or "post_restore" or 
something similar to mimic how device migration functions are named. I 
think hbitmap_restore_data_finalize would also be fine; the key part 
here is clearly naming it relative to "restore_data."

> diff --git a/util/hbitmap.c b/util/hbitmap.c
> index 8aa7406..ac0323f 100644
> --- a/util/hbitmap.c
> +++ b/util/hbitmap.c
> @@ -362,6 +362,90 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item)
>       return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0;
>   }
>
> +uint64_t hbitmap_data_size(const HBitmap *hb, uint64_t count)
> +{
> +    uint64_t size;
> +
> +    if (count == 0) {
> +        return 0;
> +    }
> +
> +    size = (((count - 1) >> hb->granularity) >> BITS_PER_LEVEL) + 1;
> +
> +    return size * sizeof(unsigned long);
> +}
> +

This seems flawed to me: number of bits without an offset can't be 
mapped to a number of real bytes, because "two bits" may or may not 
cross a granularity boundary, depending on *WHERE* you start counting.

e.g.

granularity = 1 (i.e. 2^1 = 2 virtual bits per 1 real bit)
virtual: 001100
real:     0 1 0

The amount of space required to hold "two bits" here could be as little 
as one bit, if the offset is k={0,2,4} but it could be as much as two 
bits if the offset is k={1,3}.

You may never use the function in this way, but mapping virtual bits to 
an implementation byte-size seems like it is inviting an inconsistency.

> +void hbitmap_store_data(const HBitmap *hb, uint8_t *buf,
> +                        uint64_t start, uint64_t count)
> +{
> +    uint64_t last = start + count - 1;
> +    unsigned long *out = (unsigned long *)buf;
> +
> +    if (count == 0) {
> +        return;
> +    }
> +
> +    start = (start >> hb->granularity) >> BITS_PER_LEVEL;
> +    last = (last >> hb->granularity) >> BITS_PER_LEVEL;
> +    count = last - start + 1;
> +
> +#ifdef __BIG_ENDIAN_BITFIELD
> +    for (i = start; i <= last; ++i) {
> +        unsigned long el = hb->levels[HBITMAP_LEVELS - 1][i];
> +        out[i] = (BITS_PER_LONG == 32 ? cpu_to_le32(el) : cpu_to_le64(el));
> +    }
> +#else
> +    memcpy(out, &hb->levels[HBITMAP_LEVELS - 1][start],
> +           count * sizeof(unsigned long));
> +#endif
> +}
> +

I suppose the ifdefs are trying to take an optimization by using memcpy 
if at all possible, without a conversion.

Why are the conversions to little endian, though? Shouldn't we be 
serializing to a Big Endian format?

> +void hbitmap_restore_data(HBitmap *hb, uint8_t *buf,
> +                          uint64_t start, uint64_t count)
> +{
> +    uint64_t last = start + count - 1;
> +    unsigned long *in = (unsigned long *)buf;
> +
> +    if (count == 0) {
> +        return;
> +    }
> +
> +    start = (start >> hb->granularity) >> BITS_PER_LEVEL;
> +    last = (last >> hb->granularity) >> BITS_PER_LEVEL;
> +    count = last - start + 1;
> +
> +#ifdef __BIG_ENDIAN_BITFIELD
> +    for (i = start; i <= last; ++i) {
> +        hb->levels[HBITMAP_LEVELS - 1][i] =
> +            (BITS_PER_LONG == 32 ? be32_to_cpu(in[i]) : be64_to_cpu(in[i]));
> +    }
> +#else
> +    memcpy(&hb->levels[HBITMAP_LEVELS - 1][start], in,
> +           count * sizeof(unsigned long));
> +#endif
> +}
> +

...? We're storing as LE but restoring from BE? I'm confused.

I'm also not clear on the __BIG_ENDIAN_BITFIELD macro. Why do we want to 
pack differently based on how C-bitfields are packed by the compiler? I 
don't think that has any impact on how longs are stored (always in the 
host native format.)

I would rather see the serialize/deserialize always convert to and from 
BE explicitly with cpu_to_be[32|64] and be[32|64]_to_cpu. I think trying 
to optimize the pack/unpack in the no-op condition with a # define and a 
memcpy is inviting portability problems.

> +void hbitmap_restore_finish(HBitmap *bitmap)
> +{
> +    int64_t i, size;
> +    int lev, j;
> +
> +    /* restore levels starting from penultimate to zero level, assuming
> +     * that the last one is ok */
> +    size = MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
> +    for (lev = HBITMAP_LEVELS - 1; lev-- > 0; ) {
> +        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
> +        for (i = 0; i < size; ++i) {
> +            bitmap->levels[lev][i] = 0;
> +            for (j = 0; j < BITS_PER_LONG; ++j) {
> +                if (bitmap->levels[lev+1][(i << BITS_PER_LEVEL) + j]) {
> +                    bitmap->levels[lev][i] |=  (1 << j);
> +                }
> +            }

Just a musing: Should we cache the size of each level? I know we can 
keep recalculating it, but... why bother? We recalculate it in so many 
places now.

> +        }
> +    }
> +}
> +
>   void hbitmap_free(HBitmap *hb)
>   {
>       unsigned i;