Re: [Qemu-devel] Moving release tarballs to a CDN

[Stefan Hajnoczi]

On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote:
> Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> > Hi Mike and Jeff,
> > qemu.org's bandwidth usage is dominated by release tarball downloads.
> > This puts qemu.org bandwidth usage in the 2+ TB/month range.
> 
> Hi Stefan,
> 
> how much of this traffic is caused by web spiders?
> 
> From my own binaries I know that the bots of the
> different search engines cause most of the traffic,
> if they are allowed to do so.
> 
> Usually they respect robots.txt. There is no
> https://www.qemu.org/robots.txt currently.
> Nor is there a https://download.qemu.org/robots.txt.
> Adding both would reduce the downloads, maybe
> enough to fix the problem.
> 
> Or do you see an advantage from bots which download
> QEMU tarballs? robots.txt can also block only
> selected bots.
> 
> Regards
> Stefan
> 
> PS. There is a https://git.qemu.org/robots.txt.

Great idea!  It's an easy to try adding a robots.txt and check how
bandwidth uses changes over the next month.

Jeff: Want to try this?

Stefan

signature.asc
Description: PGP signature