bug#23064: herd has incomplete status reporting, not so helpful

[Danny Milosavljevic]

So there was a postgresql update where the data format was incompatible.

That happens regularily and is ... acceptable, I guess (even more acceptable in 
guix because I can easily switch back to the old release).

However, in order to find that out, I had to:

$ psql
... something about a socket not being reachable

$ sudo herd status postgres
Status of postgres:
  It is started.
  Running value is 18438.
  It is enabled.
  Provides (postgres).
  Requires (user-processes loopback).
  Conflicts with ().
  Will be respawned.

So it claimed that everything was just peachy.

$ sudo herd stop postgres
$ sudo herd start postgres

... which reported nothing out of the ordinary either.

$ ps -ef |grep postgres
[xxxx] <defunct>

Aha! finally an indication that something is amiss.

So I tried to find the log entries, to no avail so far.
(I found: 
/gnu/store/b073csgl56g2wnq7azj4wrswb6azsvc0-postgresql-9.5.1/bin/pg_ctl -D 
/var/lib/postgresql/data -l logfile start, not sure how that helps)
$ man pg_ctl
No manual entry for pg_ctl

In the mean time, I decided I can just start up the postgres server process 
manually and it will tell me.

$ postgres --config-file=/etc/postgresql.conf -D /var/lib/postgresql/data
FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.

$ sudo -i
# su - postgres
Password:

errr... what password? 

# passwd postgres
...
# su - postgres
Password:
su: Authentication failure

# passwd -u postgres
# su - postgres
Password:
This account is currently not available.

Hmmm...

$ sudo -u postgres postgres --config-file=/etc/postgresql.conf -D 
/var/lib/postgresql/data
FATAL:  database files are incompatible with server
DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is 
not compatible with this version 9.5.1.
$ 

Aha! It says so right there in the standard output (or error?). Where did that 
message go in the shepherd case?

$ cd /var/log
$ grep -r incompatible .
$ 

???

Anyway, it works again now but we shouldn't have to do this kind of digging.