Re: [Qemu-devel] [PATCH] rcu: reduce half heap memory size by malloc_trim()

[Zhong Yang]

On Mon, Nov 20, 2017 at 02:14:50PM +0000, Daniel P. Berrange wrote:
> On Mon, Nov 20, 2017 at 04:54:42PM +0800, Zhong Yang wrote:
> > On Fri, Nov 17, 2017 at 02:06:20PM +0000, Daniel P. Berrange wrote:
> > > On Fri, Nov 17, 2017 at 01:54:09PM +0000, Stefan Hajnoczi wrote:
> > > > On Fri, Nov 17, 2017 at 02:23:34PM +0800, Yang Zhong wrote:
> > > > > diff --git a/util/rcu.c b/util/rcu.c
> > > > > index ca5a63e..8d491a6 100644
> > > > > --- a/util/rcu.c
> > > > > +++ b/util/rcu.c
> > > > > @@ -26,6 +26,7 @@
> > > > >   * IBM's contributions to this file may be relicensed under LGPLv2 
> > > > > or later.
> > > > >   */
> > > > >  
> > > > > +#include <malloc.h>
> > > > 
> > > > This header file is not mentioned in the C99 standard or POSIX.  It is
> > > > probably not available on all host OSes that QEMU supports.  Please use
> > > > #ifdef CONFIG_LINUX.
> > > > 
> > > > >  #include "qemu/osdep.h"
> > > > >  #include "qemu-common.h"
> > > > >  #include "qemu/rcu.h"
> > > > > @@ -272,6 +273,9 @@ static void *call_rcu_thread(void *opaque)
> > > > >              node->func(node);
> > > > >          }
> > > > >          qemu_mutex_unlock_iothread();
> > > > > +#ifdef CONFIG_LINUX
> > > > > +        malloc_trim(0);
> > > > > +#endif
> > > > 
> > > > It is important that the rcu thread isn't overzealous in minimizing heap
> > > > size if that means ordinary malloc(3) calls will experience latency
> > > > spikes.  Please leave a few MB free so that malloc(3) doesn't take the
> > > > slow path.
> > > 
> > > If you pass '0' the docs say that the minimum amount is left in the
> > > heap, per M_TOP_PAD, which is 128kb.
> > > 
> > > Strangely the mallopt(3) man page suggests, that free() should 
> > > automatically
> > > trim the heap when its size exceeds M_TOP_TRIM, which is again 128kb by
> > > default.  So I'm puzzelled by malloc_trim() would be needed unless there
> > > are scenarios in which free() won't trim, that aren't mentioned in the
> > > manpage.
> > 
> >   In fact, i firstly adopted mallopt() solution to optimize the heap memory,
> >   but i found this function is NOT useful, which are difference with MAN's
> >   description, so i had to swith to use malloc_trim().
> 
> 
> > > Also, how does malloc_trim interact with tcmalloc.so that people often
> > > use in preference to glibc's built in malloc ?
> >   Thanks, you reminded me to consider tcmalloc or jemalloc.
> >  
> >   Whether below code is more suitable? thanks!
> >   #if defined(CONFIG_LINUX) && defined(__GLIBC__)
> >          malloc_trim(0);
> >   #endif
> 
> Both of those macro symbols will still be defined even when tcmalloc/jemalloc
> are in use.
  
  Hello Daniel,

  If those two macro are NOT useful for this scenario, how about below changes?

  In configure file, 

  if test "$tcmalloc" = "yes" ; then
      echo "CONFIG__TCMALLOC=y" >> $config_host_mak
  ........ 

  then use !CONFIG_TCMALLOC macro to check malloc_trim()? thanks!

  Regards,

  Yang

 
> I wonder if tcmalloc/jemalloc even suffer from the same problem that libc's
> builtin malloc has ?
 
  Hello Xulei,

  Did you compiled Qemu with tcmalloc or jemalloc ? If you did , would you 
  please share those datas with different malloc compile option?  Many thanks!

  Regards,

  Yang


> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|