Re: [Tsp-devel] dtest: a distributed test framework

[Frederik Deweerdt]

On Wed, Apr 25, 2007 at 04:29:24PM +0200, Eric Noulard wrote:
> 2007/4/25, Frederik Deweerdt <address@hidden>:
> >> then you have one (or several) "local" python scripts which
> >> "only" handles ssh connection whose stdin/stdout/stderr are
> >> directly controlled by the local python scripts.
> >> Thoses scripts may barrier/ok locally.
> >I agree that the test results could come back through stdout, specially
> >if we use TAP, which only defines an output format.
> >However, I don't see how you'll get barriers from ssh: we want the
> >barriers to be inside the running code
> 
> It depends on what you consider to be
> 
> 1) the test scripts
> 
[...]
> 
> 2) the tested program
|...]
Yes, I had unit tests written in python calling TSP functions in
mind. That explains most of the misunderstandings.
> 
> >that is:
> >client                  server
> >wait_for_barrier(1)
> >                        do some setup
> >                        wait for connections
> >                        wait_for_barrier(1)
> >========== Do the actual testing ============
> >
> >That's also why XMLRPC is interesting, is that you can report the test
> >results and use the barriers from whatever language you like.
> 
> Unless I miss your point your client and server program should
> be able to do xmlrpc calls (in Python, C, etc...) in order to implement
> barrier.
> 
> But we could simply think that barrier are implemented via
> stdout/stdin monitoring:
> 
> wait barrier:  print "barrier wait <barrierID>"
> then block until stdin receive "barrier ack <barrierID>"
Ahh, I didn't got that at first. OK for me then. It's better than XMLRPC,
and easier to setup (I was afraid for the authentication, but conch at
least seems to provide the needed tools).
> This scheme is interesting but it is not the first one I may
> want to do, I would rather do:
> 
> server
> export STRACE_DEBUG=xxx
> launch tsp_stub_server
> OK(1)
> barrier(1)
> scan stdout for tsp_stdout connection
> barrier(2)
> terminate tsp_stub_server
> OK(3)
> barrier(end)
> 
> client
> barrier(1)
> launch tsp_stdout_client -n 10
> scan stdout for ERROR
> OK(2)
> barrier(2)
> barrier(end)
> 
> For implementing the last scheme we dont need to touch tsp_stdout_client
> and or tsp_stub_server code, we should only write python tests scripts
> and ssh connection handling (this is the big part).
Allright, ok then. This will be quicker to get done indeed. And the test
code is already written :)
> barrier may be implement locally because we need to synchronize
> "the tests scripts" and not "the tested program".
That is, unless you want to get the samples in a step-by-step fashion
for example. But we may not need so fine grained tests. And we could
integrate into dtest whatever language binding is needed to get this in
our C files at some point in the future.
> I think both scheme are valuable we may implements both
> beginning with what we (respectively) need first.

Agreed, and as long as OK/barrier is easy enough to implement (and the
stdin/stdout method falls in that category), both will be doable. This is
just a matter of being able to provide an ok() and barrier() method to the
test. The test can in turn launch a TSP executable or call TSP functions.

So, I'm anxiously waiting for a first ssh based distributed test framework
then :).

Fred