[Wp-mirror-list] Attempting to mirror on laptop

[Dr. Kent L. Miller]

---------- Forwarded message ----------
Date: Sun, 19 Aug 2012 14:12:00 -0400
From: Benjamin Goldsmith <address@hidden>
To: wp mirror <address@hidden>
Subject: RE: Attempting to mirror on laptop


1) File system permissions

address@hidden:~# du --human --total /var/lib/mediawiki/images/ | tail -n 1
29G     total
address@hidden:~# ls -l /var/lib/mediawiki/ | grep images
drwx------ 8 www-data www-data 2134016 Aug 11 18:09 images
address@hidden:~# ls -l /var/lib/mediawiki/images/*log
-rw-r--r-- 1 root root 3960329 Aug 11 17:59
/var/lib/mediawiki/images/failed.log

2) Image files
address@hidden:~# gm identify /var/lib/mediawiki/images/Arc_en_ciel.png
/var/lib/mediawiki/images/Arc_en_ciel.png PNG 150x75+0+0 DirectClass 8-bit
227 0.000u 0:01

3) Experiment
address@hidden:~# chown --recursive www-data:www-data
/var/lib/mediawiki/images

address@hidden:~# chmod 700 /var/lib/mediawiki/images

address@hidden:~# cd /var/lib/mediawiki/images/

address@hidden:/var/lib/mediawiki/images#  cp -a
wp-mirror/simplewiki-20120805-pages-articles-p000000000-c000001000.sh .

address@hidden:/var/lib/mediawiki/images# 
./simplewiki-20120805-pages-articles-p000000000-c000001000.sh
<snip>

During the run I saw warnings regarding -a:

[: 15773: -a: unexpected operator
  % Total    % Received % Xferd  Average Speed   Time    Time     Time 
Current
                                 Dload  Upload   Total   Spent    Left 
Speed
100 44649  100 44649    0     0  70411      0 --:--:-- --:--:-- --:--:--
87205
[: 15773: -a: unexpected operator
  % Total    % Received % Xferd  Average Speed   Time    Time     Time 
Current
                                 Dload  Upload   Total   Spent    Left 
Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
0
curl: (22) The requested URL returned error: 404

Here is the code from
simplewiki-20120805-pages-articles-p000000000-c000001000.sh from those
lines:

if [ -a $IMAGE./2/28/Hw\-augustus.jpg ]; then
        echo 2/28/Hw\-augustus.jpg already exists >> exists.log
else
        curl --retry 0 -f -O $COMMONSPATH./2/28/Hw\-augustus.jpg
        if [ -a $IMAGE./Hw\-augustus.jpg ]; then
                /bin/mkdir -p $OUTPUT./2/28/
                /bin/mv ./Hw\-augustus.jpg $OUTPUT./2/28/
                echo ./2/28/Hw\-augustus.jpg downloaded >> download.log
        else
                curl --retry 0 -f -O $IMAGEPATH./2/28/Hw\-augustus.jpg
                if [ -a $IMAGE./Hw\-augustus.jpg ]; then
                        /bin/mkdir -p $OUTPUT./2/28/
                        /bin/mv ./Hw\-augustus.jpg $OUTPUT./2/28/
                        echo ./2/28/Hw\-augustus.jpg downloaded >>
download.log
                else
                        echo ./2/28/Hw\-augustus.jpg failed >> failed.log
                fi
        fi
fi <-- line 15773

Only the failed log exists:
address@hidden:/var/lib/mediawiki/images# ls *.log
failed.log

address@hidden:/var/lib/mediawiki/images# du --human --total [0-9a-f] | tail
-n 1
du: cannot access `[0-9a-f]': No such file or directory
0       total

Looking at simplewiki-20120805-pages-articles-p000000000-c000001000.sh, I
see that:
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/
OUTPUT=
IMAGE=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/


4) wp-mirror
Since the experiment above did not create the [0-9a-f] directory tree, I did
not perform this step.

As always, I greatly appreciate your continued assistance.

Best,
Ben