[Wp-mirror-list] Fwd: Issue getting the default configuration of wp-mirror 0.6 to work

[wp mirror]

---------- Forwarded message ----------
From: Guy Castagnoli <address@hidden>
Date: Tue, 12 Nov 2013 17:40:32 +0100
Subject: RE: Issue getting the default configuration of wp-mirror 0.6 to work
To: wp mirror <address@hidden>

Kent,


many thanks for your response. Have installed your update, and it
looks as if it's now working. wp-mirror is currently downloading 9.9GB
of commonswiki and has scheduled about 2 hours and a quarter for
that..


Just a remark, relating to the entry for *wikimedia-site-xdump* : once
again, the fsm had nothing to do running "wp-mirror --mirror" on the
default configuration - am attaching the log of a run of wp-mirror
--mirror --debug of that attempt.  From looking at the log, it seems
like curl isn't finding the file `rsync-dirlist-last-1-good.txt' that
you mentioned.

So, I gave it a try again with the ftp://ftpmirror.your.org entry by
uncommenting it in /etc/wp-mirror/local.conf and now it's working..


regards, and thanks again,


Guy


p.s.: shall certainly consult the tuning issues that you address in
the Experiments section of the wp-mirror manual!




-----Original message-----
From:wp mirror <address@hidden>
Sent:Tue 12-11-2013 10:40
Subject:Re: Issue getting the default configuration of wp-mirror 0.6 to work
To:Guy Castagnoli <address@hidden>;
CC:Wp-mirror-list <address@hidden>; wpmirrordev
<address@hidden>;
Dear Guy,

Thank you very much for your report, and especially for sending me the
debug log.  The objective is to have wp-mirror work out-of-the-box
with no user configuration.  So every bug report is valuable.

1) *wikimedia-site-xdump*

This should not be a problem.  Some sites have an `index.html' file
that prevents wp-mirror from listing the directory.  For this reason,
wp-mirror also looks for a file named `rsync-dirlist-last-1-good.txt'
which contains the same information.  The default URL
<http://ftpmirror.your.org/pub/wikimedia/dumps/> is such a case.
wp-mirror should be able to find the desired dump files from the
default URL.

2) fsm-database-create

I located the bug that prevented creating database tables from
`database_farm.sql'. I fixed the bug, tested it on my platform, and
posted the fixed DEB package.

3) Testing

While I have tested the fixed DEB on my platform, I would like to know
if it also works on yours. Please try the following test and let me
know the outcome.

3.1) Download DEB package.

(shell)$ wget 
http://download.savannah.gnu.org/releases/wp-mirror/wp-mirror_0.6-1_all.deb

3.2) Install DEB package.

(root-shell)# dpkg --install wp-mirror_0.6-1_all.deb

3.3) Drop databases

(shell)$ mysql --host=localhost --user=root --password
Enter password: ********
...
mysql> show database;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| commonswiki        |
| mysql              |
| performance_schema |
| simplewiki          |
| simplewiktionary      |
| wikidb             |
| wpmirror           |
+--------------------+
8 rows in set (0.00 sec)
mysql> drop database commonswiki;
mysql> drop database simplewiki;
mysql> drop database simplewiktionary;
mysql> drop database wikidb;
mysql> drop database wpmirror;
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
mysql> quit;

3.4) Build mirror

(root-shell)# wp-mirror --mirror

4) Documentation

Given your background in mathematics and computer science, you may be
interested in the database performance experiments that I ran.  Please
take a look at Figure G-1. InnoDB Experiments.  The default
configuration of MySQL does not permit mirrors of the top ten
wikipedias to build in a reasonable amount of time.  Hence the
experiments.  The configuration files
`/usr/share/doc/wp-mirror/examples/wp-mirror_desktop.cnf' and
`/usr/share/doc/wp-mirror/examples/wp-mirror_laptop.cnf' are the
result of these performance studies.

Sincerely Yours,
Kent


On 11/10/13, Guy Castagnoli <address@hidden> wrote:
> Dear Kent,
>
>
> My name is Guy Castagnoli. I’m a mathematician/programmer in Zurich,
> Switzerland and I would like to get to know the technologies behind
> wikipedia by using with the English simple wiki.
>
> I've have successfully downloaded and installed wp-mirror 0.6 on a Debian
> 7.2.0 (wheezy) system. However, I can't get the default configuration
> (simple wiki) to work.
>
>
> First, it seems the default download source for xml-files,
> http://ftpmirror.your.org/pub/wikimedia/pub/dumps/ doesn’t work anymore
>  because wikipedia seems to have inserted an extra page under that URL (it
> gets redirected to the following page: http://dumps.wikimedia.your.org/
> <http://dumps.wikimedia.your.org/> which some announcements but has no
> sources for download). Therefore, I tried the other options for
> wikimedia-site-xdump given in /etc/wp-mirror/local.conf by uncommenting the
> corresponding lines.
>
>
> With the ftp and the rsync  lines under Wikimedia-site-xdump activated, the
> fsm does non-trivial action. The commonswiki, simplewiki, and
> simplewiktinary databases are created, but the fsm doesn't evolve beyond
> fsm-database-create. wp-mirror.log ends with a [fail]fsm-database-create
> line for all three databases. Their state in wpmirror.profile is 'fail'. The
> log says, that the three created databases have 0 tables, which looks
> suspicious. Turning on the mysql general log confirms that database_farm.sql
> generated no "CREATE TABLE" statements. Also, the mailing lists and the
> "What's new" sections give no hint about the significance of this.
>
>
> Since mechanically, my wp-mirror installation seems to be working correctly
> and since I'm unsure of what to expect from the source URLs configurable by
> uncommenting in local.conf, I tought I would turn to you to see whether I’m
> basically on track or whether there is a known issue. My next step would be
> to try to inspect wp-mirror's network interactions to make sure wp-mirror is
> receiving all the data it's expecting. However, since it may be clear to you
> whether this could be the source of the issue, I'd appreciate any hint,
> before going further - especially if it's that the downloadable packages on
> the internet have changed and you're already working on this.. Perhaps to
> start out, it would be more reasonable to study the wp-mirror guide and
> manually perform the steps to establish a mirror of the simple wiki?
>
>
> With kind regards, and already now, many thanks for your response,
>
>
> Guy
>
>
> p.s.: am attaching the output of "wp-mirror --mirror --debug" after a
> "wp-mirror --recreate" and setting  *wikimedia-site-xdump*  =
>  "ftp://ftpmirror.your.org/pub/wikimedia/dumps/";
>
>
>
>

wp-mirror.log.2_plus_debug
Description: Binary data