Re: [cks-devl] My own keyserver...

[V Alex Brennen]

I'm actually having allot of problems with resources right now 
because of postgres.  I detail bellow.

On 7 May 2002, Adrian von Bidder wrote:

> Yo!
> 
> I'm just wondering: how much resources does a public keyserver take?

I don't know yet.

I'm currently running on a PIII 450mhz w/ an 80GB Ultra 133 IDE drive.
The system runs Linux, RH6.2.  The system is fast and responsive to 
key id, long key id, and fingerprint queries but times out causing the
web server to return a 500 error on regex searches against the UID 
strings.

To correct this, I think I am going to try storing all of the UID
strings concurrently in a Berkeley DB instance and write my own 
regex code in C to run against the Berkeley DB.  Then I'll pull the
key data from postgres.  Obviously this sucks because the DBs can
get out of sync.  Eventually,  I'll try and support both postgres 
and Berkeley DB as complete separate backends (with hooks for DB2 
and oracle etc) with the keyserver code and people running very 
large keyservers (ie public ones) will likely need to use a 
Berkeley DB backend until the postgres code improves.

This will take a while to implement anyway due to all of the code
that needs to be written/changed, so for right now I'm not sure
how much hardware is needed to support a public keyserver with CKS,
but it's allot.


> Disk? CPU? Network? (During normal operation? During sync/bulk
> transfers?)

Disk: 7-20GB
CPU:  Negligible
Network: Negligible
Sync:    Negligible
Bulk Sync (initial load):  ~2GB of data transfer

With 1.7 million records, I'm using about 7GB of disk space.  With
Postgres you need to run a command to reclaim disk space and optimize
table indexes.  This command is called "vacuum analyze". In the 
version of postgres that I'm running vacuum analyze uses about 1.25 times
as much disk space as the database - so another 9-12 GB.  So you need
about 20GB right now.  If your disk fills during vacuum analyze, your
database will be crashed, so you should try and have allot of extra
space (hence my 80GB drive).

A newer version of postgres which doesn't reorder tables on disk and 
therefor doesn't use the additional 9-12GB is available, but it broke
compatibility with the older version's database files.  In order to 
upgrade, you need to export all your data and reimport it. Unfortunately,
there appears to be a bug in tar where it cannot write more that 2.7GB of 
data to a single tar archive.  Postgres can only export it's database 
files to a tar archive if they contain BLOBs. So, I can't upgrade 
postgres on gnv.us.ks.cryptnet.net until I write my own back up 
script and just reimport all the keys or fix the tar bug.

Since there isn't a differential backup mechanism for postgres, I'm
throttling my IDE drive doing my nightly backups which is causing
the keyserver to be unresponsive between like 12PM and 3AM.  I'm
working on some kind of transaction logging or something to 
resolve this problem.

Once I finish a backup program, I'll upgrade postgres and have a 
better idea of disk requirements.

 
> How often does gnv.us.ks.cryptnet.net sync with other cryptnet nodes
> (are there any?) and with the horowitz net (that would be
> wwwkeys.pgp.net?)

There are no other nodes of the cryptnet keyserver network right now
because the software needs to mature.

 
> I have a machine that will be pretty much unused by the end of this
> year, I might be interested in running a keyserver on it.

Hopefully, by that time I'll have all of the problems resolved and 
it will be easy to set up a node with minimal resources.  Right now
I don't really recommend that anyone run a node with CKS unless they're
planning on doing development work on CKS.  When CKS reaches version 
0.2.5 that will change.

Postgres and CKS are maturing in parallel, so CKS isn't moving as 
fast as I would hope.  But, CKS will start moving much faster once
I enable support for other databases.


        - VAB
-- 
V. Alex Brennen
Senior Systems Engineer
IBM Certified Specialist
e-TechServices.com
IBM Business Partner
Bus: 352.246.8553
Fax: 770.216.1877
address@hidden
http://www.e-techservices.com/people/vab/