[Duplicity-talk] passwd/group inefficiencies

[Martin Pool]

I am using Jacob's CVS package on debian
 
  duplicity-0.4.1-0cvs20030425
  python2.2-2.2.3-3

While I was experimenting with it, I noticed that it seems to spend an
enormous amount of effort pointlessly re-reading the user and password
files.  strace -e trace=open shows

open("/home/mbp/work/linux/linux-2.5.65-vmware/sound/ppc", 
O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7
open("/etc/passwd", O_RDONLY)           = 7
open("/etc/group", O_RDONLY)            = 7

(This is for an incremental backup, so it is stating other files in
between.)

This seems to come from the calls into the grp and pwd modules in
tarfile.py.

It seems to be a safe assumption that not too many users will be
created in the course of a backup run.  Perhaps it would be worth
inserting a read-only cache in that file -- just read both files up
front and stick them in dictionaries?

-- 
Martin