[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Qemu-devel] Moving release tarballs to a CDN
From: |
Stefan Hajnoczi |
Subject: |
Re: [Qemu-devel] Moving release tarballs to a CDN |
Date: |
Wed, 8 Nov 2017 17:11:20 +0000 |
User-agent: |
Mutt/1.9.1 (2017-09-22) |
On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote:
> Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> > Hi Mike and Jeff,
> > qemu.org's bandwidth usage is dominated by release tarball downloads.
> > This puts qemu.org bandwidth usage in the 2+ TB/month range.
>
> Hi Stefan,
>
> how much of this traffic is caused by web spiders?
>
> From my own binaries I know that the bots of the
> different search engines cause most of the traffic,
> if they are allowed to do so.
>
> Usually they respect robots.txt. There is no
> https://www.qemu.org/robots.txt currently.
> Nor is there a https://download.qemu.org/robots.txt.
> Adding both would reduce the downloads, maybe
> enough to fix the problem.
>
> Or do you see an advantage from bots which download
> QEMU tarballs? robots.txt can also block only
> selected bots.
>
> Regards
> Stefan
>
> PS. There is a https://git.qemu.org/robots.txt.
Great idea! It's an easy to try adding a robots.txt and check how
bandwidth uses changes over the next month.
Jeff: Want to try this?
Stefan
signature.asc
Description: PGP signature