Re: Performance of the man page database generation

[Maxim Cournoyer]

Hello!

address@hidden (Ludovic Courtès) writes:

> Hello,
>
> Maxim Cournoyer <address@hidden> skribis:
>
>> myglc2 <address@hidden> writes:
>
> [...]
>
>>> It is taking more like 50 seconds on my 3.4 Ghz server for 33 packages.
>>>
>>>> I wonder why the odd package count mismatch (23 vs 22) ?
>>>
>>> That's my fault. I cut and pasted from two different runs. SORRY! ;-)
>>>
>>
>> OK! No problem; thanks for clarifying it!
>>
>>>> I don't recall observing this while testing (maybe it only happens
>>>> when using the -m option) ?
>>>
>>> No, I get the same effect w/ 'guix package -r foo.
>>
>> 50 seconds for 33 packages on such a powerful machine seems abnormal,
>> unless you have a specific package(s) which would install an unusually
>> large amount of manual pages (which would take more time to be indexed).
>>
>> Ludovic had a way to time the generation of the manual-database
>> derivation; he was using something like [0]:
>>
>> time guix build --check 
>> /gnu/store/rkri628apz2a2i2jvav11ylv2736fvv3-manual-database.drv
>>
>> Notice that you need the derivation (.drv) of manual-database rather
>> than the store item ending by manual-database. Unfortunately I'm not
>> aware of how we can easily find the derivation of the corresponding
>> manual-database store item used by the profile (as can be found by: guix
>> gc -R $(realpath $HOME/.guix-profile) | grep manual-database).
>>
>> Maybe Ludovic can enlight us?
>
> You can see those .drv names when building the profile.  So you just
> need to copy/paste them and run “guix build --check” above.
>
> If you have the profile but not its .drv, you can find out what the .drv
> for that profile was but there’s no command-line interface for that (you
> have to use ‘valid-derivers’ from (guix store).)
>
> HTH,
> Ludo’.

Thanks Ludovic; that was helpful.

I did a small test and it takes about 40 s on my system, with 60
packages or so in my profile. I removed the "--quiet" flag of mandb in
the profile hook (guix profile) to see the output of mandb while it was
indexing the manual pages:

--8<---------------cut here---------------start------------->8---
time guix build --check 
/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv
@ build-started /gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv 
- x86_64-linux 
/var/log/guix/drvs/vk//10zmd5nm8kppd0f655fraradr019fq-manual-database.drv.bz2
creating manual page database for 62 packages...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
$MANPATH set, ignoring 
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/etc/man_db.conf
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man7'. Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection...
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man3'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_jit_stack_alloc(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_jit_stack_alloc(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_jit_stack_free(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_jit_stack_free(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_utf32_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_copy_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_copy_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_dfa_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_dfa_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_utf16_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_get_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_get_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_config(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_config(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/5h50xmsvxkxj1xjl81vh1shxwprz8gna-pcre-8.40/share/man/man3/pcredemo.3.gz:
 whatis parse for pcredemo(3) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_free_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_free_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_version(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_version(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_refcount(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_refcount(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_get_stringnumber(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_get_stringnumber(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_fullinfo(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_fullinfo(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_jit_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_jit_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_get_stringtable_entries(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_get_stringtable_entries(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_free_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_free_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_pattern_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_pattern_to_host_byte_order(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_free_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_free_study(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_get_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_get_substring_list(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_compile2(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_compile2(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_exec(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_compile(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_compile(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_maketables(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_maketables(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_assign_jit_stack(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_assign_jit_stack(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_copy_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_copy_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre32_get_named_substring(3)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for pcre16_get_named_substring(3)
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man5'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for gimprc-2.8(5)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man5/gdbinit.5.gz:
 whatis parse for gdbinit(5) failed
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man4'. Wait...
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man8'. Wait...
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/man1'. Wait...
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for dot(1)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for gimp-console-2.8(1)
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/g++.1.gz: 
whatis parse for g++(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/55qb28n3hwvmyn60hfqwpqyxgd09d0mp-gcc-6.3.0/share/man/man1/gcc.1.gz: 
whatis parse for gcc(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gcore.1.gz:
 whatis parse for gcore(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdbserver.1.gz:
 whatis parse for gdbserver(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
/gnu/store/2wmc5gz15w6hznyzgn99z87bwqf63vs5-gdb-7.12.1/share/man/man1/gdb.1.gz: 
whatis parse for gdb(1) failed
/gnu/store/3j7fyxghacpzk655m54zk20p8s5w928s-man-db-2.7.5/bin/mandb: warning: 
failed to store entry for python3(1)
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/ja/man1'. Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/ja...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.KOI8-R/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.KOI8-R...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.ISO8859-1/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.ISO8859-1...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/it.ISO8859-1/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/it.ISO8859-1...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/fr/man1'. Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/fr...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/it/man1'. Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/it...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.ISO8859-2/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.ISO8859-2...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/pl/man1'. Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/pl...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.UTF-8/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/ru.UTF-8...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.UTF-8/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/fr.UTF-8...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/it.UTF-8/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/it.UTF-8...
done.
Updating index cache for path 
`/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.UTF-8/man1'. 
Wait...
Processing manual pages under 
/tmp/guix-build-manual-database.drv-0/manpages-collection/pl.UTF-8...
done.
18 man subdirectories contained newer manual pages.
3943 manual pages were added.
find-files: 
/gnu/store/5mk4lg1rdzq2p3148idjd4z0bvif0nf4-gimp-2.8.18/share/man/man1/gimp-console.1:
 No such file or directory
guix build: error: build failed: derivation 
`/gnu/store/vk10zmd5nm8kppd0f655fraradr019fq-manual-database.drv' may not be 
deterministic: output 
`/gnu/store/lmf9rfzj565m31xzhhlam3lk91bjbqsm-manual-database' differs

real    0m39.890s
user    0m0.356s
sys     0m0.040s
--8<---------------cut here---------------end--------------->8---

As you can see, mandb stumbled more than once with some of our packages'
manpages. Those failures are probably attributable to bad packages
manual pages. For example, the gcc manual pages are known to be broken
for example. [0]

By looking at the output as it was generated, it seemed as if those
failures caused mandb to take more time.

Another reason could be that mandb is just inefficient. There's an old
bug about mandb's use of processes being less than optimal ("strace
indicates that mandb is forking on the order of four processes per
page") [1]. It seems that Colin Watson had a good idea of how to improve
things but didn't get around to implementing it yet.

[0] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24069
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=630799

signature.asc
Description: PGP signature