[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH] misc: introduce strim-memory qapi to support free memory tri
From: |
Daniel P . Berrangé |
Subject: |
Re: [PATCH] misc: introduce strim-memory qapi to support free memory trimming |
Date: |
Thu, 1 Aug 2024 14:12:31 +0100 |
User-agent: |
Mutt/2.2.12 (2023-09-09) |
On Sat, Jul 27, 2024 at 01:18:32PM +0800, Guoyi Tu wrote:
> On 2024/7/25 19:57, Daniel P. Berrangé wrote:
> > On Thu, Jul 25, 2024 at 01:35:21PM +0200, Markus Armbruster wrote:
> > > Guoyi Tu <tugy@chinatelecom.cn> writes:
> > >
> > > > In the test environment, we conducted IO stress tests on all storage
> > > > disks
> > > > within a virtual machine that had five storage devices mounted.During
> > > > testing,
> > > > we found that the qemu process allocated a large amount of memory
> > > > (~800MB)
> > > > to handle these IO operations.
> > > >
> > > > When the test ended, although qemu called free() to release the
> > > > allocated
> > > > memory, the memory was not actually returned to the operating system, as
> > > > observed via the top command.
> > > >
> > > > Upon researching the glibc memory management mechanism, we found that
> > > > when
> > > > small chunks of memory are allocated in user space and then released
> > > > with
> > > > free(), the glibc memory management mechanism does not necessarily
> > > > return
> > > > this memory to the operating system. Instead, it retains the memory
> > > > until
> > > > certain conditions are met for release.
> > >
> > > Yes.
> >
> > Looking at mallopt(3) man page, the M_TRIM_THRESHOLD is said to control
> > when glibc releases the top of the heap back to the OS. It is said to
> > default to 128 kb.
> Yes, the M_TRIM_THRESHOLD option can control glibc to release the free
> memory at the top of the heap, but glibc will not release the free
> memory in the middle of the heap.
>
> > I'm curious how we get from that default, to 800 MB of unused memory > Is
> > it related to the number of distinct malloc arenas that are in use ?
>
> At least 600MB of memory is free, and this memory might be in the middle of
> the heap and cannot be automatically released.
>
> > I'm curious what malloc_stats() would report before & after malloc_trim
> > when QEMU is in this situation with lots of wasted memory.
> Here is the test case:
snip
That looks like an artifical reproducer, rather than the real world
QEMU scenario.
What's the actual I/O stress test scenario you use to reproduce the
problem in QEMU, and how have you configured QEMU (ie what CLI args) ?
I'm fairly inclined to suggest that having such a huge amount of
freed memory is a glibc bug, but to escalate this to glibc requires
us to provide them better real world examples of the problems.
> > The above usage is automatic, while this proposal requires that
> > an external mgmt app monitor QEMU and tell it to free memory.
> > I'm wondering if the latter is really desirable, or whether QEMU
> > can call this itself when reasonable ?
>
> Yes, I have also considered implementing an automatic memory release
> function within qemu. This approach would require qemu to periodically
> monitor the IO load of all backend storage, and if the IO load is very
> low over a period of time, it would proactively release memory.
I would note that in systemd they have logic which is monitoring
either /proc/pressure/memory or $CGROUP/memory.pressure, and in
response to events on that, it will call malloc_trim
https://github.com/systemd/systemd/blob/main/docs/MEMORY_PRESSURE.md
https://docs.kernel.org/accounting/psi.html
Something like that might be better, as it lets us hide the specific
design & impl choices inside QEMU, letting us change/evolve them at
will without impacting public API designs.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- Re: [PATCH] misc: introduce strim-memory qapi to support free memory trimming,
Daniel P . Berrangé <=