[Savannah-hackers-public] "Tokyo datacenter" for FSF

[Jing Luo]

Hello Ian, Michael, Bob, Corwin, Amin,

First of all, my condolences to Ian's and Michael's livers. No one should have 
to work an all-nighter, so that's why I'm here hoping I can help.

As I proposed on IRC, I can offer my VMs (& my expertise) to FSF so that 
hopefully they can provide fail-over or load-balancing. I have a cluster of 
servers that consists of various different spec, some are freedom-friendly and 
some are less so. So here is a detailed list of machines I own that can host 
VMs and each of their pros and cons (in my opinion) for your consideration 
(also for future documenting purpose). Please let me know what you think and 
what they can provide for FSF (e.g. DNS, MariaDB cluster, web etc.). 
Suggestions are very welcome.

Geolocation: Tokyo, Japan
Electricity: 100% renewable energy.
Internet (not a machine but worth mentioning):
ISP1:
 10Gbps residential (unshared), unmetered and unlimited, 1 static IPv4, a /56 
block of IPv6.
 Pros: high speed and unmetered. Serving 5TB/day on average now, peak traffic 
55TB/day.
 Cons: the ISP doesn't support rDNS.
ISP2:
 1Gbps semi-business-residential (fiber shared with the apartment building), 
data caps at 1TB per month, a /28 block of IPv4 and a /56 block of IPv6.
 Pros: the ISP supports rDNS with IPv4. Suitable for name servers.
 Cons: speed severely throttles after 1TB/month.

Hostname: h12ssl-nt
CPU: AMD EPYC 75F3, 32 cores
RAM: 8x64=512GB DDR4 ECC RDIMM
Motherboard: Supermicro H12SSL-NT
Storage:
 boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an 
Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
 general purpose storage: 10x 7.68TB Kioxia CD6-R, PCIe Gen 4 NVNe SSD, ZFS 
RAIDz2, with an Intel Optane P1600X 58GB, PCIe Gen3 NVMe SSD as SLOG
Description:
 This server runs an installation of Proxmox VE in an ATX PC case. Most of the 
components were bought used from the second hand market, except for the two 
Intel Optane SSDs. It provides many services important to the community and 
essential to my digital life, such as:
  - 1 node of repo.jing.rocks (the largest free software mirror in Japan, no 
kidding)
  - 1 authoritative name server and 2 recursive DNS servers (pihole, with DHCP)
  - 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
  - 1 node of invidious.jing.rocks (luckily has not been banned by google)
  - mastodon.jing.lgbt (1 node for load-balancing)
  - multiple web servers in LXC containers
  - 1 Proxmox Mail Gateway cluster node
 It's also worth mentioning that I provide {web,name,mail} servers for Dragora 
GNU/Linux-libre.
Pros:
 It's very stable, high performance, is also used as a NAS and a build machine.
Cons:
 Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free 
IPMI but not connected to the internet. Built this machine when I didn't really 
know free software. Also it's at risk of running out of RAM because of ZFS 
filesystem caching (which Bob disagrees)

Hostname: rome2d16-2t
CPU: 2 sockets: AMD EPYC 7773X, 2x64=128 cores
RAM: 16x64=1024GB DDR4 ECC RDIMM
Motherboard: Asrock ROME2D16-2T
Storage:
 boot drives: 2x 1.6TB Kioxia CM6-V, PCIe Gen 4 NVMe SSD, ZFS RAID1, with an 
Intel Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
 general purpose storage:
  - 11x 16TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB 
SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an Intel 
Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
  - 11x 18TB SATA spinning hard drives, ZFS RAIDz3, with an L2ARC of 3.84TB 
SATA SSD, with a three-way mirrored special metadata vdev 1.92TB, with an Intel 
Optane P1600X 118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
  - 2x 8TB SATA spinning hard drives, ZFS RAID1, with an Intel Optane P1600X 
118GB, PCIe Gen3 NVMe SSD as SLOG (shared)
  - 2x 3.84TB SATA SSD, ZFS RAID1, with an Intel Optane P1600X 118GB, PCIe Gen3 
NVMe SSD as SLOG (shared), reserved for gcc compile farm
Description:
 This server runs an installation of Proxmox VE in an EATX PC case. Many 
components were bought used. It mainly provides these services:
  - 3 VMs for gcc compile farm [1]: {cfarm420,cfarm421,cfarm422}.cfarm.net (I 
specifically asked for those host names :)
  - 1 node of repo.jing.rocks
  - 1 authoritative name server and 1 internal DNS server (pihole, with DHCP)
  - 1 vcs server that runs a forgejo instance and a savane instance (testing 
only)
  - 1 MariaDB cluster node, 2 Postgresql cluster nodes for various services
  - mastodon.jing.lgbt (1 node for load-balancing)
  - 1 Proxmox Mail Gateway cluster node
Pros:
 Suitable for highly parallel workload. The second ZFS pool has about 50TB of 
space available, while others are at about 75~80% capacity.
Cons:
 Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free 
IPMI but not connected to the internet. Built this machine when I didn't really 
know free software. Also it's at risk of running out of RAM because of ZFS 
filesystem caching (which Bob disagrees).

[1] https://portal.cfarm.net/machines/list/

Hostname: z490
CPU: Intel Core i9-10900K
RAM: 4x32=128GB DDR4 UDIMM
Description:
 It also runs Proxmox VE, but mainly for more "not highly parallel" services, 
like jitsi, nextcloud, invidious. About to be turned into a second-level L3 
switch. It has Intel ME.

Hostname: x570d4u-2l2t
CPU: AMD Ryzen 9 5950X, 32 cores.
RAM: 4x32=128GB DDR4 ECC UDIMM
Motherboard: Asrock X570D4U-2L2T
Description:
 It also runs Proxmox VE, but mainly functions as a router/core switch. It's 
super stable, has a long uptime. It has 10 10GbE ports in total. It runs 2 
OpenWRT VMs, 1 ZNC IRC bouncer, 1 Proxmox Mail Gateway cluster node, and 1 
reverse proxy/load-balancer for all web servers. Has a 4x960GB SATA SSD ZFS 
RAID0 that isn't storing anything...
Pros: It boots really fast.
Cons:
 Has the AMD equivalent of Intel ME (forgot the name...). Has onboard non-free 
IPMI but not connected to the internet. The boot drives are about to be 
replaced with ZFS RAID1, so a reinstall is coming.

Hostname: altrad8ud-1l2t
CPU: Ampere Altra Max Q128-30 engineering sample, 128 cores.
RAM: 8x16=128GB for now, upgrading to 512GB
Motherboard: Asrock ALTRAD8UD-1L2T
Description:
 A new machine that is not x86! It runs a version of Proxmox VE that I patched 
and built from source then ported to arm64. It runs Trisquel perfectly in my 
testing. Currently not running any service. About to be reinstalled, replacing 
btrfs raid1 with zfs raid1.
Pros: 100% free. Suitable for highly parallel workload, preparing to offer VMs 
to Trisquel build farm and gcc compile farm. Runs really cool.
Cons: Asrock ships a non-free UEFI/BIOS and a non-free distribution of OpenBMC 
(not connected to internet).

Hostname: (none)
CPU: Rockchip RK3588
RAM: 32GB LPDDR4
Storage: 4TB NVMe SSD, consumer grade, and onboard eMMC and/or microSD cards.
Description:
 I also have 3 rockchip rk3588 based SBCs. They have my patched Proxmox VE 
arm64 installed, with a custom kernel build. They all can run linux-libre 
without framebuffer support (I gave up on those a long time ago). They are 
under powered for compiling jobs, but suitable for server use.
Pros: Stable and power efficient.
Cons:
 Has a fatal flaw that requires a non-free blob to boot. The DDR init blob must 
be inserted into u-boot, or else it doesn't boot. Currently using a custom 
u-boot build. I hope maybe someone can reverse engineer it someday...

Some useful URLs:
https://stats.jing.rocks/
https://munin.jing.rocks/+(add "munin" or "munin/", want to avoid scraping bots 
here)
https://git.jing.rocks/cgit/home-config.git/
https://goaccess.jing.rocks/

Thanks for reading, looking forward to your opinions. I'm going back to patch 
proxmox and linux...

-- 
Jing Luo
About me: https://jing.rocks/about/
GPG Fingerprint: 4E09 8D19 00AA 3F72 1899 2614 09B3 316E 13A1 1EFC

signature.asc
Description: OpenPGP digital signature