Re: [lwip-users] mem_malloc(): memory fragmentation

[Jonathan Larmour]

Goldschmidt Simon wrote:
> > > One good solution if using 2^n sized pools is to use a buddy 
> > > allocator[1] to divide up larger contiguous space, so it may not be
> as 
> > > wasteful as you think. One difference with a normal buddy allocator
> is 
> > > that a normal one would normally e.g. return a 2Kbyte buffer if you 
> > > request 1025 bytes. An lwIP implementation could work for maximum 
> > > efficiency instead and allocate that as a 1024 byte buffer plus a 64
>
> > > byte buffer (or whatever the lowest granularity would be) chained
> together.
> > I think that would definitely be the only way such a change 
> > would be acceptable given that lwIP tries to have a low 
> > memory footprint, and indeed it's the only way I'd even 
> > though of it being done.
>
> I don't really understand what you mean with the buddy allocator... Do
> you mean chaining 2 pbufs together?

Two or more. A buddy allocator is a simple, quick and efficient way of 
allocating chunks of memory of size 2^^N for a range of N.

> The way it's done in the wikipedia
> example does not really suppress fragmentation, or does it?

Since you're able to chain pbufs together then it resists fragmentation.

> Chaining a
> pbuf from two fragments would be easy (implementing pools), but I don't
> know if it would be less speedy or causing any other problem.

A buddy allocator should be pretty speedy. It would have some interesting 
effects on the application though - netbufs would be far more likely to 
consist of multiple pbufs.

> > > But all this would be a non-trivial bit of coding so I'm sure people
>
> > > would be grateful if you have time to do it
>
> Don't know about the triviality... I implemented 5 pools of different
> sizes (using the memp.c interface, just added some pools). Normal calls
> to mem_malloc() get the size they want (maybe too big, but only DHCP,
> SNMP & loopif are using that and they could be reprogrammed using
> memp_malloc, I suggest). Pbuf_alloc(PBUF_RAM) would construct a
> pbuf-chain of sizes just like you suggested.

It's the "maybe too big" bit that I think wants to be solved. What sizes 
are the packets in your pools?

> About code size, I'm not sure, but you certainly can better calculate
> the amount of data RAM you need when suppressing (external)
> fragmentation (and thus save space compared to the current heap), and
> that should make up for the bigger code size. Last but not least you get
> a _much_ better feeling running lwIP applications for some years without
> rebooting (at least I do).

ROM is usually much cheaper than RAM anyway. So adding more code if it 
saves RAM is usually good.

> I would be open to fragmentation tests, too (like Christiaan suggested),
> but I'm not sure that this makes much sense. Since the memory usage is
> almost always caused by clients contacting my board, fragmentation would
> possibly depend on the network traffic my app is running on... So I'd
> rather theoretically think about the fragmentation issue instead of
> trying to prove it in examples.

It absolutely would depend on network traffic, and especially if you send 
packets of wildly different sizes. Things like adding a DHCP renewal in the 
middle may have the potential to fragment the space a bit as that uses 
small but persistant allocations. The upcoming SNMP agent would have a 
major effect on fragmentation as it makes lots of mem_malloc calls of 
various sizes.

Jifl
--
eCosCentric    http://www.eCosCentric.com/    The eCos and RedBoot experts
------["The best things in life aren't things."]------      Opinions==mine