[Wp-mirror-list] Attempting to mirror on laptop

[wp mirror]

Dear Benjamin,

Thank you for your reply.  The situation you describe is something of a puzzle.

0) Preamble

I-chunks, which are the files with names like

simplewiki-20120805-pages-articles-p000000000-c000001000.sh
simplewiki-20120805-pages-articles-p000001000-c000001000.sh
etc.

are stored under /var/lib/mediawiki/images/wp-mirror/.  They are large
BASH scripts which execute commands such as:

curl - which downloads image files to /var/lib/mediawiki/images/,
mkdir - which creates the [0-9a-f] directory tree,
mv - which moves the downloaded files into the [0-9a-f] directory tree, and
echo - which emits messages to log files.

What is strange about your case, is that apparently CURL was invoked
to download the image files, yet MKDIR and MV were not invoked to
store them in the [0-9a-f] directory tree.  If you ran WP-MIRROR from
a root-shell, this should not have happened.

I am guessing that file system permissions are involved.

So, let us collect a bit more diagnostic information, and then try an
experiment.

1) File system permissions

Please let me see your output for the following commands:

root-shell> du --human --total /var/lib/mediawiki/images/ | tail -n 1
47G     total

root-shell> ls -l /var/lib/mediawiki/ | grep images
drwx------ 24 www-data www-data 688 Aug 16 04:47 images

root-shell> ls -l /var/lib/mediawiki/images/*log
-rw-r--r-- 1 root root 1507005 Aug 12 03:09
/var/lib/mediawiki/images/download.log
-rw-r--r-- 1 root root  278194 Aug 12 03:06 /var/lib/mediawiki/images/exists.log
-rw-r--r-- 1 root root 4527367 Aug 12 03:10 /var/lib/mediawiki/images/failed.log

2) Image files

Let us determine if the image files were downloaded without corruption.

To that end, please show me your output for the following command:

root-shell> gm identify /var/lib/mediawiki/images/Arc_en_ciel.png
/var/lib/mediawiki/images/Arc_en_ciel.png PNG 150x75+0+0 DirectClass
8-bit 227 0.000u 0:01

If Arc_en_ciel.png was not downloaded, please feel free to pick any
other image file that was downloaded.

3) Experiment

Let us manually fix permissions and ownerships, and then manually
execute one of the i-chunks.

To that end, please show me your output for the following commands:

root-shell> chown --recursive www-data:www-data /var/lib/mediawiki/images

root-shell> chmod 700 /var/lib/mediawiki/images

root-shell> cd /var/lib/mediawiki/images/

root-shell> cp -a
wp-mirror/simplewiki-20120805-pages-articles-p000000000-c000001000.sh
.

root-shell> ./simplewiki-20120805-pages-articles-p000000000-c000001000.sh
<snip>

root-shell> du --human --total [0-9a-f] | tail -n 1
45G    total

4) wp-mirror

If the experiment above created the [0-9a-f] directory tree and
populated it with image files, then please try running:

root-shell> wp-mirror --mirror

Sincerely Yours,
Kent