[Wp-mirror-list] Attempting to mirror on laptop

[wp mirror]

Dear Benjamin,


0) Preamble

Thank you for providing output from `wp-mirror --text'.  When
WP-MIRROR runs in monitor mode (--text, --screen, or --gui) it
collects state information by forking shell processes that run MySQL
as well as certain BASH commands.  This suggests that MySQL problems
have been solved, and that file system permissions are in order.

Thank you for the chown error messages.  It appears that WP-MIRROR is
not locating the image files that were downloaded when the i-chunks
were processed.  You should have over 50,000 images occupying about
40G.  So where are the image files?

1) images

Downloaded image files should be stored in a directory tree under
/var/lib/mediawiki/images/[0-9a-f]/.  Resized image files should be
stored under /var/lib/mediawiki/images/thumb/.  Mathematical
expressions should have been converted to PNG image files and stored
under /var/lib/mediawiki/images/math/.

To that end, please let me see your output for the following commands:

root-shell> ls -l /var/lib/mediawiki/images/
total 6235
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 0
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 1
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 2
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 3
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 4
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 5
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 6
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 7
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 8
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 9
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 a
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 b
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 bad-images
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 c
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 d
-rw-r--r--  1 root     root     1507005 Aug 12 03:09 download.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 e
-rw-r--r--  1 root     root      278194 Aug 12 03:06 exists.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 f
-rw-r--r--  1 root     root     4527367 Aug 12 03:10 failed.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 math
drwxrwxrwx  2 www-data www-data      48 Nov 27  2011 temp
drwxrwxrwx 18 www-data www-data     432 Nov 29  2011 thumb
drwxrwxrwx  2 www-data www-data      48 Aug 12 09:54 tmp
drwxr-xr-x  2 www-data www-data   62936 Aug 14 23:37 wp-mirror

root-shell> du --human --total /var/lib/mediawiki/images/[0-9a-f] | tail -n 1
45G     total

root-shell> du --human --total /var/lib/mediawiki/images/thumb | tail -n 1
1.1G    total

root-shell> du --human --total /var/lib/mediawiki/images/math | tail -n 1
16M     total

2) work files

WP-MIRROR stores its work files (dump files, x-chunks, i-chunks, etc.)
under the directory /var/lib/mediawiki/images/wp-mirror/.  I would
like to see if files are there, and if the i-chunks are properly
formed.

To that end, please let me see your output for the following commands.

root-shell> du --human --total /var/lib/mediawiki/images/wp-mirror/ | tail -n 1
1.5G    total

root-shell> ls 
/var/lib/mediawiki/images/wp-mirror/simplewiki-latest-md5sums.txt*
| wc -l
2

root-shell> ls 
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles*
| wc -l
315

root-shell> ls /var/lib/mediawiki/images/wp-mirror/[0-9a-f]
ls: cannot access /var/lib/mediawiki/images/wp-mirror/[0-9a-f]: No
such file or directory

root-shell> cat
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles-p000000000-c000001000.sh
| head -n 33
#!/bin/sh

IMAGE=
OUTPUT=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/

/bin/mkdir -p $OUTPUT./thumb
/bin/chmod 777 $OUTPUT./thumb
/bin/mkdir -p $OUTPUT./temp
/bin/chmod 777 $OUTPUT./temp
/bin/mkdir -p $OUTPUT./tmp
/bin/chmod 777 $OUTPUT./tmp

if [ -a $IMAGE./c/c7/PB050006.JPG ]; then
        echo c/c7/PB050006.JPG already exists >> exists.log
else
        curl --retry 0 -f -O $COMMONSPATH./c/c7/PB050006.JPG
        if [ -a $IMAGE./PB050006.JPG ]; then
                /bin/mkdir -p $OUTPUT./c/c7/
                /bin/mv ./PB050006.JPG $OUTPUT./c/c7/
                echo ./c/c7/PB050006.JPG downloaded >> download.log
        else
                curl --retry 0 -f -O $IMAGEPATH./c/c7/PB050006.JPG
                if [ -a $IMAGE./PB050006.JPG ]; then
                        /bin/mkdir -p $OUTPUT./c/c7/
                        /bin/mv ./PB050006.JPG $OUTPUT./c/c7/
                        echo ./c/c7/PB050006.JPG downloaded >> download.log
                else
                        echo ./c/c7/PB050006.JPG failed >> failed.log
                fi
        fi
fi

3) MediaWiki

Please let me know if you can browse <simple.mediawiki.site>.

To that end, please open a web browser to:

<http://simple.mediawiki.site/index.php/Main_Page>

Please let me know if you see images on this page:

<http://simple.mediawiki.site/index.php/Dinosaur>

Please let me know if you see nicely formatted math symbols on this page:

<http://simple.mediawiki.site/index.php/Pythagorean_theorem>

Sincerely Yours,
Kent