CDN performance

[Ludovic Courtès]

Hello Chris,

Chris Marusich <address@hidden> skribis:

> Regarding DNS, it would be nice if we could use an official GNU
> subdomain.  If we can't use a GNU subdomain, we should at least make
> sure we have some kind of DNS auto-renewal set up so that nobody can
> poach our domain names.  And the operators should take appropriate
> precautions when sharing any credentials used for managing it all.

Agreed.

Regarding the GNU sub-domain, as I replied to Meiyo, I’m in favor of it,
all we need is someone to champion setting it up.

> Regarding CDNs, I definitely think it's worth a try!  Even Debian is
> using CloudFront (cloudfront.debian.net).  In fact, email correspondence
> suggests that as of 2013, Amazon may even have been paying for it!
>
> https://lists.debian.org/debian-cloud/2013/05/msg00071.html

(Note that debian.net is not Debian, and “there’s no cloud, only other
people’s computer” as the FSFE puts it.)

> Although I don't doubt that a CDN will perform better than what we have
> now, I do think it would be good to measure the performance so that we
> know for sure the money spent is actually providing a benefit.  It would
> be nice to have some data before and after to measure how availability
> and performance have changed.  Apart from anecdotes, what data do we
> have to determine whether performance has improved after introducing a
> CDN?  For example, the following information could be useful:
>
>   * Network load on the origin server(s)
>   * Clients' latency to (the addresses pointed to by) ci.guix.info
>   * Clients' throughput while downloading substitutes from ci.guix.info

Note that performance is one aspect; another one is availability (we’ve
seen that with the recent hydra.gnu.org outage!).  We know we’ll always
win in terms of availability by having a CDN in front of our servers.

That said, measuring performance is very useful and it’s great that we
can benefit from your expertise here!

> Here, it took 0.459667 - 0.254210 = 0.205457 seconds (about 205 ms) to
> establish the TCP connection after the DNS lookup.  The average
> throughput was 1924285 bytes per second (about 40 megabits per second,
> where 1 megabit = 10^6 bits).  It seems my connection to berlin is
> already pretty good!

Indeed.  The bandwidth problem on berlin is when you’re the first to
download a nar and it’s not been cached by nginx yet.  In that case, you
get very low bandwidth (like 10 times less than when the item is cached
by nginx.)  I’ve looked into it, went as far as strace’ing nginx, but
couldn’t find the reason of this.

Do you any idea?

> Establishing the TCP connection took about 21 ms (which matches the mtr
> output), and the throughput was about 79 megabits per second.  (On this
> machine, 100 Mbps is the current link speed, according to dmesg output.)
> This means that in my case, when using CloudFront the latency is 10x
> lower, and the throughput (for a cache hit) is 2x higher, than using
> berlin.guixsd.org directly!

Impressive.

> It would be interesting to see what the performance is for others.

I’ve tried this from home (in France, with FTTH):

--8<---------------cut here---------------start------------->8---
$ measure_get 
https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
  2>/dev/null
url_effective: 
https://berlin.guixsd.org/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 141.80.181.40
remote_port: 443
size_download: 69899433 B
speed_download: 1522001.000 B/s
time_appconnect: 0.178892 s
time_connect: 0.049649 s
time_namelookup: 0.000422 s
time_pretransfer: 0.178934 s
time_redirect: 0.000000 s
time_starttransfer: 0.278312 s
time_total: 45.926021 s
$ measure_get 
https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
 2>/dev/null
url_effective: 
https://berlin-mirror.marusich.info/nar/gzip/1bq783rbkzv9z9zdhivbvfzhsz2s5yac-linux-libre-4.19
http_code: 200
num_connects: 1
num_redirects: 0
remote_ip: 2600:9000:2116:6e00:c:49d4:5a80:93a1
remote_port: 443
size_download: 69899433 B
speed_download: 20803402.000 B/s
time_appconnect: 0.552008 s
time_connect: 0.482477 s
time_namelookup: 0.467598 s
time_pretransfer: 0.552157 s
time_redirect: 0.000000 s
time_starttransfer: 0.735758 s
time_total: 3.360500 s
--8<---------------cut here---------------end--------------->8---

Wall-clock time is less than a tenth; woow.

Thanks for sharing your insight and scripts!

Ludo’.