RE: [Ltib] 3 ways to get uncompressed image size(s)..

[Stuart Hughes]

Hi Peter,

These are interesting questions.  First the easy ones:

As far as a minimal system goes, there are things called 'profiles' in
LTIB. In this context, it just means a way of selecting a number of
packages without changing platform specific options, so from your LTIB
instance you can say:

$ ./ltib --profile config/profile/min.config --batch

This will make you a small root filesystem.

You can then go in if you like and do:

$ ./ltib --configure

and select uClibc as your C library if you want something really small.
A minimal image for a uClibc build is about 850 KB compressed. 

As far as the deployment missing removing the docs/html files goes, this
is a bug.  I've added it as a todo item.  The simplest thing is probably
to run a 'find usr/share -name \*.html -exec rm {}\;'

Now the more difficult question is why is the size calculated on the
host so out with respect to the actual image.  I'm not totally sure, but
here are some points of guidance:

* -l is used in the calculation (du) because some filesystems don't
correctly cope with hard links and so this allows a conservative
estimate to be made without knowing the type of image that is later on
made.  This could be improved.  I even forget which ones can't handle
hard links, maybe it was cramfs (or jffs2 can't recall).

* Some space is added even when you do add your own extra padding (KBs):
    $fs_size += $fs_size >= 20000 ? 16384 : 2400;

This is to give a small amount of free space for system logs etc.

* I recall that the block size of the actual filesystem (on the host) is
bigger sometimes than some of the target image filesystem.  So when du
calculates the total size, some extra space is inappropriately estimated
for the target.  For example, each inode may be 4K on the host (can't
recall), but less on the target.  When you have a lot of symlinks I
think this may artificially push up the estimate.

If you can get some insight into this and come up with something more
accurate that would be great.  Any ideas/patches warmly received.

Regards, Stuart


On Mon, 2008-03-17 at 16:50 -0400, Stephens, Peter A. wrote:
> Thanks for the patience guys.  When people ask me, "how big will the
> image be ..." they look at me with dismay when I ask them, "What
> packages do you need?".  Most of the time they don't know because they
> haven't thought that far ahead (conscientious, up-front, systems work is
> rare).  
> 
> So, I want a way to build an image, determine the size, and list the
> packages.  That way I can do this for several configurations that seem
> to make sense and give them these numbers with the configurations.  LTIB
> can list the packages, so all I need now is the size(s).  I don't mind
> using the build numbers, but I want to understand them so I have been
> doing this investigation.  
> 
> My biggest concern at this point is that the build numbers don't match
> what I get when I decompress the image, mount it, and use "du -cks"
> (which matches "df") on it.  The build numbers are about 150% of what I
> get with these methods.  Really not that bad for an estimate, but why
> the large difference?  Is it because the "l" option of "du" counts hard
> links multiple times?
> 
> Another concern is that some of the man pages and docs (mostly html
> files) don't seem to have been stripped.  I thought I saw a
> configuration setting that indicated it would delete these, but perhaps
> I misinterpreted it, or there are packages that don't honor this yet
> (gtk and libxml2?), etc...  Anyway, I was able to reduce the image size
> to about half by deleting these.
> 
> By "default", I meant that I just took what the configuration was when I
> installed LTIB.  I am not really good enough with small Linux systems to
> know what is a "minimal" build yet.  That will probably be the subject
> of my next research topic, but "minimal" is relative anyway (back to the
> need for conscientious, up-front, systems work).
> 
> Peter A. Stephens
> Advanced Engineering
> address@hidden
> www.delphi.com
> 765-451-7074
> 
> -----Original Message-----
> From: Stuart Hughes [mailto:address@hidden 
> Sent: Monday, March 17, 2008 1:24 PM
> To: Stephens, Peter A.
> Cc: address@hidden; address@hidden
> Subject: Re: [Ltib] 3 ways to get uncompressed image size(s)..
> 
> Hi Peter,
> 
> What default do you mean, I can see gtk in your output, which is not
> very minimal.
> 
> Padding is default by zero, it can be added in the deployment section
> when you configure.  There is a small amount of extra space allocated on
> the filesystem depending on the size, this is needed to make sure there
> is space for system created logs/files etc.  Take a look at
> bin/Ltibutils.pm:sub mk_fs_image for the details, it looks like this:
> 
>     my $fs_count = 0;
>     find( sub { $fs_count++ }, $stage);
>     my ($fs_size) =  split(/\s+/, `LANG=C du -slk $stage`);
>     $fs_size += $fs_size >= 20000 ? 16384 : 2400;
>     $fs_size += $pcf->{DEPLOYMENT_PADDING_KB}
> if$pcf->{DEPLOYMENT_PADDING_KB};
> 
> Don't try to calculate the size from the rootfs directory as this
> included un-stripped and files that get removed by default (man pages
> etc).  As you can see in ltib, it calculates it size by running
>   $ du -slk rootfs.tmp
> after it has stripped out unwanted files/directories.
> 
> What are you trying to achieve? if you want a really small rootfs, you
> need to cut back your packages and if it is appropriate select a uClibc
> toolchain.  Images under 10MB can benefit greatly by using uClibc,
> however be warned that for some platforms this doesn't give the best
> optimised performance in some areas (no NPTL, sometimes the math is only
> soft etc).
> 
> Regards, Stuart
> 
> 
> 
> On Mon, 2008-03-17 at 11:55 -0400, Stephens, Peter A. wrote:
> > I built an image pretty close to the minimal, default, setup.  I used 
> > 3 methods to get the uncompressed sizes.
> > 
> >     Using "df" after mounting the new file system           32 Mb
> > reported
> >     Using "du . -ck" after mounting the new file system     32 Mb
> reported
> >     Using the build output
> > 47 Mb reported
> > 
> > The build output says, "including padding".  What does that mean?  
> > What is the padding for?
> > Is the padding enough that it makes up the difference between 47.5Mb 
> > and the 32Mb reported using the other methods? Can the padding be 
> > adjusted or removed?
> > 
> > Pete
> > 
> > ================================= Numbers for Minimal image 
> > ================================
> > 
> > -------------------------------- Compressed image file size
> > --------------------------------
> > 
> > ~/ltib> ls -l rootfs.*
> > -rw-rw-r-- 1 hzqmrk hzqmrk  8776723 2008-03-14 15:48 rootfs.ext2.gz
> > 
> > Approximately 8.8Mb.  Pretty good.
> > 
> > -- Method 1 ----------- Using "df" after mounting the new file system
> > -----------------------
> > 
> > ~/ltib/tmp2> gunzip rootfs.ext2.gz
> > ~/ltib/tmp2> sudo mount -o loop rootfs.ext2 /mnt/iso/ ~/ltib/tmp2> cd 
> > /mnt/iso/ /mnt/iso> df -l .
> > Filesystem 1K-blocks Used  Available Use% Mounted on
> > /home/hzqmrk/proj/ltib/tmp2/rootfs.ext2
> >            43429     31749 9305  78% /mnt/iso
> > 
> > OK, so this reports ~ 32 Mb after mounting the image.
> > 
> > -- Method 2 --------- Using "du . -ck" after mounting the new file 
> > system --------------------
> > 
> > ~/ltib/tmp2> gunzip rootfs.ext2.gz
> > ~/ltib/tmp2> sudo mount -o loop rootfs.ext2 /mnt/iso/ ~/ltib/tmp2> cd 
> > /mnt/iso/ /mnt/iso> du . -ck | sort -rn | head -n 10
> > du: cannot read directory `./lost+found': Permission denied
> > du: cannot read directory `./var/run/usb': Permission denied
> > 31749   total
> > 31749   .
> > 20817   ./usr
> > 13834   ./usr/share
> > 9133    ./usr/share/gtk-doc
> > 9132    ./usr/share/gtk-doc/html
> > 5033    ./usr/lib
> > 5028    ./lib
> > 3785    ./usr/share/gtk-doc/html/glib
> > 3245    ./usr/share/doc
> > 
> > I also created a file that contained all the file sizes from the iso 
> > using
> > 
> >     /mnt/iso> find . -type f -name "*" -exec ls -l {} \; > ~/tmp.txt
> > 
> > and the total size for the files was 29MB.  The difference may be 
> > "blocks used" versus "actual" file sizes, but it is so close to the 
> > 32MB reported with this method that I think this immaterial.  What is 
> > of interest is that I found a lot of documentation in 
> > "/usr/share/doc", "/usr/share/gtk-doc", and "/usr/man".  After 
> > deleting those, the uncompressed file total was 16.9Mb implying a 
> > compressed file of
> > (possibly) less than
> > 5Mb for this minimal setup.
> > 
> > -- Method 3 ----------------------- Using build output
> > --------------------------------------
> > 
> > Filesystem stats, including padding:
> >     Total size            = 47504k
> >     Total number of files = 1102
> > Your ramdisk exceeds the old default size of 4096k, you may need to 
> > set the command line argument for ramdisk_size in your bootloader 
> > allowing 10% free this gives 52254k .  For instance, for u-boot:
> > 
> > setenv bootargs root=/dev/ram rw ramdisk_size=52254
> > 
> > creating an ext2 compressed filesystem image: rootfs.ext2.gz
> > 
> > Started: Fri Mar 14 15:43:44 2008
> > Ended:   Fri Mar 14 15:48:18 2008
> > Elapsed: 274 seconds
> > 
> > Build Succeeded
> > 
> > 
> > 
> > Peter A. Stephens
> > Advanced Engineering
> > address@hidden
> > www.delphi.com
> > 765-451-7074
> > 
> > -----Original Message-----
> > From: Erik Peterson [mailto:address@hidden
> > Sent: Friday, March 14, 2008 3:08 PM
> > To: Stephens, Peter A.
> > Cc: address@hidden; address@hidden
> > Subject: Re: [Ltib] Way to calculate uncompressed image size..
> > 
> > You could always use your cross-compile-"size"
> > for example :
> > address@hidden bin]$ ./powerpc-linux-size /auto/freescale/ltib/ltib- 
> > mpc8544ds-20071122/rootfs/boot/vmlinux
> >     text       data     bss     dec     hex filename
> > 4004373      198408  191484 4394265  430d19
> /auto/freescale/ltib/ltib- 
> > mpc8544ds-20071122/rootfs/boot/vmlinux
> > 
> > 
> > On Mar 14, 2008, at 11:17 AM, Stephens, Peter A. wrote:
> > 
> > >
> > > If I want to calculate the uncompressed size of an image can I just 
> > > add the packages I want.  Build it.  Change directories to "rootfs" 
> > > and run "du . -ckS"?  Here is an example...
> > >
> > > address@hidden rootfs]$ du . -ckS | sort -rn | head -n 20
> > > 118524  total
> > > 33896   ./usr/lib
> > > 26076   ./usr/lib/gstreamer-0.10
> > > 15520   ./usr/bin
> > > 6804    ./boot
> > > 4904    ./lib
> > > 3984    ./usr/share/gtk-doc/html/glib
> > > 2676    ./usr/share/gtk-doc/html/libxml2
> > > 2660    ./usr/info
> > > 2540    ./usr/share/doc/libxml2-2.6.28/html/html
> > > 1980    ./usr/share/gtk-doc/html/gobject
> > > 1772    ./sbin
> > > 1600    ./bin
> > > 1252    ./usr/share/sounds/alsa
> > > 976     ./usr/share/gtk-doc/html/liboil
> > > 808     ./usr/share/man/man3
> > > 692     ./usr/share/man/man1
> > > 580     ./usr/sbin
> > > 528     ./usr/share/doc/libxml2-2.6.28/html
> > > 508     ./usr/include/gstreamer-0.10/gst
> > >
> > > Peter A. Stephens
> > > address@hidden <mailto:address@hidden>
> > > 765-451-7074
> > >
> > >
> > >
> > **********************************************************************
> > **
> > ****************
> > >
> > > Note:  If the reader of this message is not the intended recipient, 
> > > or an employee or agent responsible for delivering this message to 
> > > the intended recipient, you are hereby notified that any 
> > > dissemination, distribution or copying of this communication is 
> > > strictly prohibited. If you have received this communication in 
> > > error, please notify us immediately by replying to the message and 
> > > deleting it from your computer. Thank you.
> > >
> > >
> > **********************************************************************
> > **
> > ****************
> > >
> > >
> > > _______________________________________________
> > > LTIB home page: http://bitshrine.org
> > >
> > > Ltib mailing list
> > > address@hidden
> > > http://lists.nongnu.org/mailman/listinfo/ltib
> > 
> > 
> > **********************************************************************
> > ******************
> > 
> > Note:  If the reader of this message is not the intended recipient, or
> an employee or agent responsible for delivering this message to the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited. If
> you have received this communication in error, please notify us
> immediately by replying to the message and deleting it from your
> computer. Thank you. 
> > 
> > **********************************************************************
> > ******************
> > 
> > 
> > _______________________________________________
> > LTIB home page: http://bitshrine.org
> > 
> > Ltib mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/ltib
> 
> 
> ****************************************************************************************
> 
> Note:  If the reader of this message is not the intended recipient, or an 
> employee or agent responsible for delivering this message to the intended 
> recipient, you are hereby notified that any dissemination, distribution or 
> copying of this communication is strictly prohibited. If you have received 
> this communication in error, please notify us immediately by replying to the 
> message and deleting it from your computer. Thank you. 
> 
> ****************************************************************************************
> 
> 
> _______________________________________________
> LTIB home page: http://bitshrine.org
> 
> Ltib mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/ltib