Re: [Qemu-devel] Moving release tarballs to a CDN

[Jeff Cody]

On Wed, Nov 08, 2017 at 05:11:20PM +0000, Stefan Hajnoczi wrote:
> On Wed, Nov 08, 2017 at 05:19:25PM +0100, Stefan Weil wrote:
> > Am 08.11.2017 um 16:33 schrieb Stefan Hajnoczi:
> > > Hi Mike and Jeff,
> > > qemu.org's bandwidth usage is dominated by release tarball downloads.
> > > This puts qemu.org bandwidth usage in the 2+ TB/month range.
> > 
> > Hi Stefan,
> > 
> > how much of this traffic is caused by web spiders?
> > 
> > From my own binaries I know that the bots of the
> > different search engines cause most of the traffic,
> > if they are allowed to do so.
> > 
> > Usually they respect robots.txt. There is no
> > https://www.qemu.org/robots.txt currently.
> > Nor is there a https://download.qemu.org/robots.txt.
> > Adding both would reduce the downloads, maybe
> > enough to fix the problem.
> > 
> > Or do you see an advantage from bots which download
> > QEMU tarballs? robots.txt can also block only
> > selected bots.
> > 
> > Regards
> > Stefan
> > 
> > PS. There is a https://git.qemu.org/robots.txt.
> 
> Great idea!  It's an easy to try adding a robots.txt and check how
> bandwidth uses changes over the next month.
> 
> Jeff: Want to try this?
> 
> Stefan

Yes, sure - I added a robots.txt to exclude .bz2 and .xz files, and we can
see how that affects bandwidth.  Right now, with our current hosting provider,
we are not near any bandwidth limit, but it makes sense to conserve
resources (unless there is any benefit we see to allowing bots to index
download.qemu.org binaries).


Thanks,

Jeff