Re: [Qemu-devel] [PATCH] configure: Add support for jemalloc

[Paolo Bonzini]

On 19/06/2015 12:56, Alexandre Derumier wrote:
> This adds "--enable-jemalloc" and "--disable-jemalloc" to allow linking
> to jemalloc memory allocator.
> 
> We have already tcmalloc support,
> but it seem to not working well with a lot of iothreads/disks.
> 
> The main problem is that tcmalloc use a shared thread cache of 16MB
> by default.
> With more threads, this cache is shared, and some bad garbage collections
> can occur if the cache is too low.
> 
> It's possible to tcmalloc cache increase it with a env var:
> TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB

It's also possible to do

MallocExtension_SetNumericProperty("tcmalloc.max_total_thread_cache_bytes",
                                   num_io_threads << 24);

What is the peak memory usage of jemalloc and tcmalloc?

Paolo

> With default 16MB, performances are  really bad with more than 2 disks.
> Increasing to 256MB, it's helping but still have problem with 16 
> disks/iothreads.
> 
> Jemalloc don't have performance problem with default configuration.
> 
> Here the benchmark results in iops of 1 qemu vm randread 4K iodepth=32,
> with rbd block backend (librbd is doing a lot of memory allocation),
> 1 iothread by disk
> 
> glibc malloc
> ------------
> 
> 1 disk      29052
> 2 disks     55878
> 4 disks     127899
> 8 disks     240566
> 15 disks    269976
> 
> jemalloc
> --------
> 
> 1 disk      41278
> 2 disks     75781
> 4 disks     195351
> 8 disks     294241
> 15 disks    298199
> 
> tcmalloc 2.2.1 default 16M cache
> --------------------------------
> 
> 1 disk   37911
> 2 disks  67698
> 4 disks  41076
> 8 disks  43312
> 15 disks 37569
> 
> tcmalloc : 256M cache
> ---------------------------
> 
> 1 disk     33914
> 2 disks    58839
> 4 disks    148205
> 8 disks    213298
> 15 disks   218383
> 
> Signed-off-by: Alexandre Derumier <address@hidden>
> ---
>  configure | 29 +++++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/configure b/configure
> index 222694f..2fe1e05 100755
> --- a/configure
> +++ b/configure
> @@ -336,6 +336,7 @@ vhdx=""
>  quorum=""
>  numa=""
>  tcmalloc="no"
> +jemalloc="no"
>  
>  # parse CC options first
>  for opt do
> @@ -1147,6 +1148,10 @@ for opt do
>    ;;
>    --enable-tcmalloc) tcmalloc="yes"
>    ;;
> +  --disable-jemalloc) jemalloc="no"
> +  ;;
> +  --enable-jemalloc) jemalloc="yes"
> +  ;;
>    *)
>        echo "ERROR: unknown option $opt"
>        echo "Try '$0 --help' for more information"
> @@ -1420,6 +1425,8 @@ Advanced options (experts only):
>    --enable-numa            enable libnuma support
>    --disable-tcmalloc       disable tcmalloc support
>    --enable-tcmalloc        enable tcmalloc support
> +  --disable-jemalloc       disable jemalloc support
> +  --enable-jemalloc        enable jemalloc support
>  
>  NOTE: The object files are built at the place where configure is launched
>  EOF
> @@ -3344,6 +3351,11 @@ EOF
>    fi
>  fi
>  
> +if test "$tcmalloc" = "yes" && test "$jemalloc" = "yes" ; then
> +    echo "ERROR: tcmalloc && jemalloc can't be used at the same time"
> +    exit 1
> +fi
> +
>  ##########################################
>  # tcmalloc probe
>  
> @@ -3361,6 +3373,22 @@ EOF
>  fi
>  
>  ##########################################
> +# jemalloc probe
> +
> +if test "$jemalloc" = "yes" ; then
> +  cat > $TMPC << EOF
> +#include <stdlib.h>
> +int main(void) { malloc(1); return 0; }
> +EOF
> +
> +  if compile_prog "" "-ljemalloc" ; then
> +    LIBS="-ljemalloc $LIBS"
> +  else
> +    feature_not_found "jemalloc" "install jemalloc devel"
> +  fi
> +fi
> +
> +##########################################
>  # signalfd probe
>  signalfd="no"
>  cat > $TMPC << EOF
> @@ -4499,6 +4527,7 @@ echo "snappy support    $snappy"
>  echo "bzip2 support     $bzip2"
>  echo "NUMA host support $numa"
>  echo "tcmalloc support  $tcmalloc"
> +echo "jemalloc support  $jemalloc"
>  
>  if test "$sdl_too_old" = "yes"; then
>  echo "-> Your SDL version is too old - please upgrade to have SDL support"
>