wp-mirror-list
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Wp-mirror-list] Attempting to mirror on laptop


From: Dr. Kent L. Miller
Subject: [Wp-mirror-list] Attempting to mirror on laptop
Date: Wed, 15 Aug 2012 19:02:55 -0400 (EDT)
User-agent: Alpine 2.00 (DEB 1167 2008-08-23)

---------- Forwarded message ----------
Date: Wed, 15 Aug 2012 17:50:59 -0400
From: Benjamin Goldsmith <address@hidden>
To: wp mirror <address@hidden>
Subject: RE: Attempting to mirror on laptop


Thank you for your continued assistance with wp-mirror, Kent.

Here is the information you requested:

1)images

I will not post the results of:
root-shell> ls -l /var/lib/mediawiki/images/

The list included tens of thousands of image files:

address@hidden:~# ls -l /var/lib/mediawiki/images/ | wc -l
40356

Here is a list of the directories ONLY.  You'll note that the [a-f][0-9]
subdirs are missing:

address@hidden:~# ls -l /var/lib/mediawiki/images/ | egrep '^d'
drwxr-xr-x 18 root root      4096 Jul 11 00:03 bad-images
drwxr-xr-x 18 root root      4096 Aug 11 18:11 math
drwxrwxrwx  2 root root      4096 Jul 12 19:08 temp
drwxrwxrwx  2 root root      4096 Jul 12 19:08 thumb
drwxrwxrwx  2 root root      4096 Aug 12 13:50 tmp
drwxr-xr-x  2 root root     36864 Aug 12 13:50 wp-mirror


address@hidden:~# du --human --total /var/lib/mediawiki/images/[0-9a-f] | tail
-n 1
du: cannot access `/var/lib/mediawiki/images/[0-9a-f]': No such file or
directory
0       total


address@hidden:~# du --human --total /var/lib/mediawiki/images/thumb | tail -n
1
4.0K    total


address@hidden:~# du --human --total /var/lib/mediawiki/images/math | tail -n
1
18M     total


2) work files

address@hidden:~# du --human --total /var/lib/mediawiki/images/wp-mirror/ |
tail -n 1
1.1G    total


address@hidden:~# ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-latest-md5sums.txt* | wc -l
2


address@hidden:~# ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles* | wc
-l
315


address@hidden:~# ls /var/lib/mediawiki/images/wp-mirror/[0-9a-f]
ls: cannot access /var/lib/mediawiki/images/wp-mirror/[0-9a-f]: No such file
or directory


#!/bin/sh

IMAGE=
OUTPUT=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/

/bin/mkdir -p $OUTPUT./thumb
/bin/chmod 777 $OUTPUT./thumb
/bin/mkdir -p $OUTPUT./temp
/bin/chmod 777 $OUTPUT./temp
/bin/mkdir -p $OUTPUT./tmp
/bin/chmod 777 $OUTPUT./tmp

if [ -a $IMAGE./0/0f/Dibuix_de_Leo.png ]; then
        echo 0/0f/Dibuix_de_Leo.png already exists >> exists.log
else
        curl --retry 0 -f -O $COMMONSPATH./0/0f/Dibuix_de_Leo.png
        if [ -a $IMAGE./Dibuix_de_Leo.png ]; then
                /bin/mkdir -p $OUTPUT./0/0f/
                /bin/mv ./Dibuix_de_Leo.png $OUTPUT./0/0f/
                echo ./0/0f/Dibuix_de_Leo.png downloaded >> download.log
        else
                curl --retry 0 -f -O $IMAGEPATH./0/0f/Dibuix_de_Leo.png
                if [ -a $IMAGE./Dibuix_de_Leo.png ]; then
                        /bin/mkdir -p $OUTPUT./0/0f/
                        /bin/mv ./Dibuix_de_Leo.png $OUTPUT./0/0f/
                        echo ./0/0f/Dibuix_de_Leo.png downloaded >>
download.log
                else
                        echo ./0/0f/Dibuix_de_Leo.png failed >> failed.log
                fi
        fi
fi


3) MediaWiki

To that end, please open a web browser to:

<http://simple.mediawiki.site/index.php/Main_Page>

Worked but with issues.  I saw messages such as:
a) Error: image is invalid or non-existent
b) {{:Main Page/Article Division by zero}}
c) File:Gnome-applications-science.svg

Please let me know if you see images on this page:

<http://simple.mediawiki.site/index.php/Dinosaur>

The page loaded but without images.

Please let me know if you see nicely formatted math symbols on this page:

<http://simple.mediawiki.site/index.php/Pythagorean_theorem>

The math symbols appear and look great.

Thank you for your continued assistance.  I really appreciate it.

Best,
-Ben


-----Original Message-----
From: wp mirror [mailto:address@hidden
Sent: Wed 8/15/2012 5:21 AM
To: Benjamin Goldsmith
Cc: address@hidden
Subject: Attempting to mirror on laptop

Dear Benjamin,


0) Preamble

Thank you for providing output from `wp-mirror --text'.  When
WP-MIRROR runs in monitor mode (--text, --screen, or --gui) it
collects state information by forking shell processes that run MySQL
as well as certain BASH commands.  This suggests that MySQL problems
have been solved, and that file system permissions are in order.

Thank you for the chown error messages.  It appears that WP-MIRROR is
not locating the image files that were downloaded when the i-chunks
were processed.  You should have over 50,000 images occupying about
40G.  So where are the image files?

1) images

Downloaded image files should be stored in a directory tree under
/var/lib/mediawiki/images/[0-9a-f]/.  Resized image files should be
stored under /var/lib/mediawiki/images/thumb/.  Mathematical
expressions should have been converted to PNG image files and stored
under /var/lib/mediawiki/images/math/.

To that end, please let me see your output for the following commands:

root-shell> ls -l /var/lib/mediawiki/images/
total 6235
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 0
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 1
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 2
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 3
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 4
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 5
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 6
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 7
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 8
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 9
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 a
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 b
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 bad-images
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 c
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 d
-rw-r--r--  1 root     root     1507005 Aug 12 03:09 download.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 e
-rw-r--r--  1 root     root      278194 Aug 12 03:06 exists.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 f
-rw-r--r--  1 root     root     4527367 Aug 12 03:10 failed.log
drwxr-xr-x 18 www-data www-data     432 Nov 27  2011 math
drwxrwxrwx  2 www-data www-data      48 Nov 27  2011 temp
drwxrwxrwx 18 www-data www-data     432 Nov 29  2011 thumb
drwxrwxrwx  2 www-data www-data      48 Aug 12 09:54 tmp
drwxr-xr-x  2 www-data www-data   62936 Aug 14 23:37 wp-mirror

root-shell> du --human --total /var/lib/mediawiki/images/[0-9a-f] | tail -n
1
45G     total

root-shell> du --human --total /var/lib/mediawiki/images/thumb | tail -n 1
1.1G    total

root-shell> du --human --total /var/lib/mediawiki/images/math | tail -n 1
16M     total

2) work files

WP-MIRROR stores its work files (dump files, x-chunks, i-chunks, etc.)
under the directory /var/lib/mediawiki/images/wp-mirror/.  I would
like to see if files are there, and if the i-chunks are properly
formed.

To that end, please let me see your output for the following commands.

root-shell> du --human --total /var/lib/mediawiki/images/wp-mirror/ | tail
-n 1
1.5G    total

root-shell> ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-latest-md5sums.txt*
| wc -l
2

root-shell> ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles*
| wc -l
315

root-shell> ls /var/lib/mediawiki/images/wp-mirror/[0-9a-f]
ls: cannot access /var/lib/mediawiki/images/wp-mirror/[0-9a-f]: No
such file or directory

root-shell> cat
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles-p000
000000-c000001000.sh
| head -n 33
#!/bin/sh

IMAGE=
OUTPUT=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/

/bin/mkdir -p $OUTPUT./thumb
/bin/chmod 777 $OUTPUT./thumb
/bin/mkdir -p $OUTPUT./temp
/bin/chmod 777 $OUTPUT./temp
/bin/mkdir -p $OUTPUT./tmp
/bin/chmod 777 $OUTPUT./tmp

if [ -a $IMAGE./c/c7/PB050006.JPG ]; then
        echo c/c7/PB050006.JPG already exists >> exists.log
else
        curl --retry 0 -f -O $COMMONSPATH./c/c7/PB050006.JPG
        if [ -a $IMAGE./PB050006.JPG ]; then
                /bin/mkdir -p $OUTPUT./c/c7/
                /bin/mv ./PB050006.JPG $OUTPUT./c/c7/
                echo ./c/c7/PB050006.JPG downloaded >> download.log
        else
                curl --retry 0 -f -O $IMAGEPATH./c/c7/PB050006.JPG
                if [ -a $IMAGE./PB050006.JPG ]; then
                        /bin/mkdir -p $OUTPUT./c/c7/
                        /bin/mv ./PB050006.JPG $OUTPUT./c/c7/
                        echo ./c/c7/PB050006.JPG downloaded >> download.log
                else
                        echo ./c/c7/PB050006.JPG failed >> failed.log
                fi
        fi
fi

3) MediaWiki

Please let me know if you can browse <simple.mediawiki.site>.

To that end, please open a web browser to:

<http://simple.mediawiki.site/index.php/Main_Page>

Please let me know if you see images on this page:

<http://simple.mediawiki.site/index.php/Dinosaur>

Please let me know if you see nicely formatted math symbols on this page:

<http://simple.mediawiki.site/index.php/Pythagorean_theorem>

Sincerely Yours,
Kent


reply via email to

[Prev in Thread] Current Thread [Next in Thread]