[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Wp-mirror-list] Fwd: Issue getting the default configuration of wp-mirr
From: |
wp mirror |
Subject: |
[Wp-mirror-list] Fwd: Issue getting the default configuration of wp-mirror 0.6 to work |
Date: |
Tue, 12 Nov 2013 12:15:07 -0500 |
---------- Forwarded message ----------
From: Guy Castagnoli <address@hidden>
Date: Tue, 12 Nov 2013 17:40:32 +0100
Subject: RE: Issue getting the default configuration of wp-mirror 0.6 to work
To: wp mirror <address@hidden>
Kent,
many thanks for your response. Have installed your update, and it
looks as if it's now working. wp-mirror is currently downloading 9.9GB
of commonswiki and has scheduled about 2 hours and a quarter for
that..
Just a remark, relating to the entry for *wikimedia-site-xdump* : once
again, the fsm had nothing to do running "wp-mirror --mirror" on the
default configuration - am attaching the log of a run of wp-mirror
--mirror --debug of that attempt. From looking at the log, it seems
like curl isn't finding the file `rsync-dirlist-last-1-good.txt' that
you mentioned.
So, I gave it a try again with the ftp://ftpmirror.your.org entry by
uncommenting it in /etc/wp-mirror/local.conf and now it's working..
regards, and thanks again,
Guy
p.s.: shall certainly consult the tuning issues that you address in
the Experiments section of the wp-mirror manual!
-----Original message-----
From:wp mirror <address@hidden>
Sent:Tue 12-11-2013 10:40
Subject:Re: Issue getting the default configuration of wp-mirror 0.6 to work
To:Guy Castagnoli <address@hidden>;
CC:Wp-mirror-list <address@hidden>; wpmirrordev
<address@hidden>;
Dear Guy,
Thank you very much for your report, and especially for sending me the
debug log. The objective is to have wp-mirror work out-of-the-box
with no user configuration. So every bug report is valuable.
1) *wikimedia-site-xdump*
This should not be a problem. Some sites have an `index.html' file
that prevents wp-mirror from listing the directory. For this reason,
wp-mirror also looks for a file named `rsync-dirlist-last-1-good.txt'
which contains the same information. The default URL
<http://ftpmirror.your.org/pub/wikimedia/dumps/> is such a case.
wp-mirror should be able to find the desired dump files from the
default URL.
2) fsm-database-create
I located the bug that prevented creating database tables from
`database_farm.sql'. I fixed the bug, tested it on my platform, and
posted the fixed DEB package.
3) Testing
While I have tested the fixed DEB on my platform, I would like to know
if it also works on yours. Please try the following test and let me
know the outcome.
3.1) Download DEB package.
(shell)$ wget
http://download.savannah.gnu.org/releases/wp-mirror/wp-mirror_0.6-1_all.deb
3.2) Install DEB package.
(root-shell)# dpkg --install wp-mirror_0.6-1_all.deb
3.3) Drop databases
(shell)$ mysql --host=localhost --user=root --password
Enter password: ********
...
mysql> show database;
+--------------------+
| Database |
+--------------------+
| information_schema |
| commonswiki |
| mysql |
| performance_schema |
| simplewiki |
| simplewiktionary |
| wikidb |
| wpmirror |
+--------------------+
8 rows in set (0.00 sec)
mysql> drop database commonswiki;
mysql> drop database simplewiki;
mysql> drop database simplewiktionary;
mysql> drop database wikidb;
mysql> drop database wpmirror;
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)
mysql> quit;
3.4) Build mirror
(root-shell)# wp-mirror --mirror
4) Documentation
Given your background in mathematics and computer science, you may be
interested in the database performance experiments that I ran. Please
take a look at Figure G-1. InnoDB Experiments. The default
configuration of MySQL does not permit mirrors of the top ten
wikipedias to build in a reasonable amount of time. Hence the
experiments. The configuration files
`/usr/share/doc/wp-mirror/examples/wp-mirror_desktop.cnf' and
`/usr/share/doc/wp-mirror/examples/wp-mirror_laptop.cnf' are the
result of these performance studies.
Sincerely Yours,
Kent
On 11/10/13, Guy Castagnoli <address@hidden> wrote:
> Dear Kent,
>
>
> My name is Guy Castagnoli. I’m a mathematician/programmer in Zurich,
> Switzerland and I would like to get to know the technologies behind
> wikipedia by using with the English simple wiki.
>
> I've have successfully downloaded and installed wp-mirror 0.6 on a Debian
> 7.2.0 (wheezy) system. However, I can't get the default configuration
> (simple wiki) to work.
>
>
> First, it seems the default download source for xml-files,
> http://ftpmirror.your.org/pub/wikimedia/pub/dumps/ doesn’t work anymore
> because wikipedia seems to have inserted an extra page under that URL (it
> gets redirected to the following page: http://dumps.wikimedia.your.org/
> <http://dumps.wikimedia.your.org/> which some announcements but has no
> sources for download). Therefore, I tried the other options for
> wikimedia-site-xdump given in /etc/wp-mirror/local.conf by uncommenting the
> corresponding lines.
>
>
> With the ftp and the rsync lines under Wikimedia-site-xdump activated, the
> fsm does non-trivial action. The commonswiki, simplewiki, and
> simplewiktinary databases are created, but the fsm doesn't evolve beyond
> fsm-database-create. wp-mirror.log ends with a [fail]fsm-database-create
> line for all three databases. Their state in wpmirror.profile is 'fail'. The
> log says, that the three created databases have 0 tables, which looks
> suspicious. Turning on the mysql general log confirms that database_farm.sql
> generated no "CREATE TABLE" statements. Also, the mailing lists and the
> "What's new" sections give no hint about the significance of this.
>
>
> Since mechanically, my wp-mirror installation seems to be working correctly
> and since I'm unsure of what to expect from the source URLs configurable by
> uncommenting in local.conf, I tought I would turn to you to see whether I’m
> basically on track or whether there is a known issue. My next step would be
> to try to inspect wp-mirror's network interactions to make sure wp-mirror is
> receiving all the data it's expecting. However, since it may be clear to you
> whether this could be the source of the issue, I'd appreciate any hint,
> before going further - especially if it's that the downloadable packages on
> the internet have changed and you're already working on this.. Perhaps to
> start out, it would be more reasonable to study the wp-mirror guide and
> manually perform the steps to establish a mirror of the simple wiki?
>
>
> With kind regards, and already now, many thanks for your response,
>
>
> Guy
>
>
> p.s.: am attaching the output of "wp-mirror --mirror --debug" after a
> "wp-mirror --recreate" and setting *wikimedia-site-xdump* =
> "ftp://ftpmirror.your.org/pub/wikimedia/dumps/"
>
>
>
>
wp-mirror.log.2_plus_debug
Description: Binary data
Message not available