Re: [Wp-mirror-list] Issue getting the default configuration of wp-mirror 0.6 to work

[wp mirror]

Dear Guy,

Thank you very much for your report, and especially for sending me the
debug log.  The objective is to have wp-mirror work out-of-the-box
with no user configuration.  So every bug report is valuable.

1) *wikimedia-site-xdump*

This should not be a problem.  Some sites have an `index.html' file
that prevents wp-mirror from listing the directory.  For this reason,
wp-mirror also looks for a file named `rsync-dirlist-last-1-good.txt'
which contains the same information.  The default URL
<http://ftpmirror.your.org/pub/wikimedia/dumps/> is such a case.
wp-mirror should be able to find the desired dump files from the
default URL.

2) fsm-database-create

I located the bug that prevented creating database tables from
`database_farm.sql'. I fixed the bug, tested it on my platform, and
posted the fixed DEB package.

3) Testing

While I have tested the fixed DEB on my platform, I would like to know
if it also works on yours. Please try the following test and let me
know the outcome.

3.1) Download DEB package.

(shell)$ wget 
http://download.savannah.gnu.org/releases/wp-mirror/wp-mirror_0.6-1_all.deb

3.2) Install DEB package.

(root-shell)# dpkg --install wp-mirror_0.6-1_all.deb

3.3) Drop databases

(shell)$ mysql --host=localhost --user=root --password
Enter password: ********
...
mysql> show database;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| commonswiki        |
| mysql              |
| performance_schema |
| simplewiki          |
| simplewiktionary      |
| wikidb             |
| wpmirror           |
+--------------------+
8 rows in set (0.00 sec)
mysql> drop database commonswiki;
mysql> drop database simplewiki;
mysql> drop database simplewiktionary;
mysql> drop database wikidb;
mysql> drop database wpmirror;
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
mysql> quit;

3.4) Build mirror

(root-shell)# wp-mirror --mirror

4) Documentation

Given your background in mathematics and computer science, you may be
interested in the database performance experiments that I ran.  Please
take a look at Figure G-1. InnoDB Experiments.  The default
configuration of MySQL does not permit mirrors of the top ten
wikipedias to build in a reasonable amount of time.  Hence the
experiments.  The configuration files
`/usr/share/doc/wp-mirror/examples/wp-mirror_desktop.cnf' and
`/usr/share/doc/wp-mirror/examples/wp-mirror_laptop.cnf' are the
result of these performance studies.

Sincerely Yours,
Kent


On 11/10/13, Guy Castagnoli <address@hidden> wrote:
> Dear Kent,
>
>
> My name is Guy Castagnoli. I’m a mathematician/programmer in Zurich,
> Switzerland and I would like to get to know the technologies behind
> wikipedia by using with the English simple wiki.
>
> I've have successfully downloaded and installed wp-mirror 0.6 on a Debian
> 7.2.0 (wheezy) system. However, I can't get the default configuration
> (simple wiki) to work.
>
>
> First, it seems the default download source for xml-files,
> http://ftpmirror.your.org/pub/wikimedia/pub/dumps/ doesn’t work anymore
>  because wikipedia seems to have inserted an extra page under that URL (it
> gets redirected to the following page: http://dumps.wikimedia.your.org/
> <http://dumps.wikimedia.your.org/> which some announcements but has no
> sources for download). Therefore, I tried the other options for
> wikimedia-site-xdump given in /etc/wp-mirror/local.conf by uncommenting the
> corresponding lines.
>
>
> With the ftp and the rsync  lines under Wikimedia-site-xdump activated, the
> fsm does non-trivial action. The commonswiki, simplewiki, and
> simplewiktinary databases are created, but the fsm doesn't evolve beyond
> fsm-database-create. wp-mirror.log ends with a [fail]fsm-database-create
> line for all three databases. Their state in wpmirror.profile is 'fail'. The
> log says, that the three created databases have 0 tables, which looks
> suspicious. Turning on the mysql general log confirms that database_farm.sql
> generated no "CREATE TABLE" statements. Also, the mailing lists and the
> "What's new" sections give no hint about the significance of this.
>
>
> Since mechanically, my wp-mirror installation seems to be working correctly
> and since I'm unsure of what to expect from the source URLs configurable by
> uncommenting in local.conf, I tought I would turn to you to see whether I’m
> basically on track or whether there is a known issue. My next step would be
> to try to inspect wp-mirror's network interactions to make sure wp-mirror is
> receiving all the data it's expecting. However, since it may be clear to you
> whether this could be the source of the issue, I'd appreciate any hint,
> before going further - especially if it's that the downloadable packages on
> the internet have changed and you're already working on this.. Perhaps to
> start out, it would be more reasonable to study the wp-mirror guide and
> manually perform the steps to establish a mirror of the simple wiki?
>
>
> With kind regards, and already now, many thanks for your response,
>
>
> Guy
>
>
> p.s.: am attaching the output of "wp-mirror --mirror --debug" after a
> "wp-mirror --recreate" and setting  *wikimedia-site-xdump*  =
>  "ftp://ftpmirror.your.org/pub/wikimedia/dumps/";
>
>
>
>