Re: [Myexperiment-discuss] proposal: a "test and controls" section for experiments in myExperiment

[Paolo Missier]

Hi all,
 the need for sanity checks of myExp workflows has been raised before, 
and not only in the context of the so-called "workflow decay monitor", 
and indeed I have long been an advocate of "packing" all the necessary 
test data along with the workflow.
I think this _is_ a topic for myExperiment, rather than for Taverna, 
because it involves making the published workflow self-consistent for 
multiple reasons:

- to clarify its intent. Not been an e-scientist myself, I can only 
guess that I/O examples are at least as informative to bioinformaticians 
as a narrative description

- to verify that its current functionality is intact, in view of 
possible changes in the external services that the workflow depends on. 
Regression testing after changes (either intentional or not) has long 
been a key feature of SE practices.

- to verify functional correctness (i.e., no bugs). Notice that this may 
be subtle: a correct T1 experiment may translate in some strange way to 
a T2 workflow (there are, as we know, backward compatibility issues) 
that is no longer "semantically" correct.

I don't see many other options. Adding exception management to SW 
components is also good SW eng practice in general, but we can't expect 
users to do that systematically by adding new processors, and as you 
(Paul) point out this may reduce the readability of the process. On the 
other hand, provider-supplied error messages may not be informative enough.
Also, a feature/limitation of Taverna has always been its weak data 
typing. Strong typing  is used in programming languages to perform 
sanity checks on input values, amongst other things. Without it, the 
ability for a third party to verify functionality by testing on author's 
supplied I/O data becomes essential.

As a simple example of how the intended behaviour of a workflow may be 
hidden in the guts of the process, here is a bit of a recent private 
email exchange with Alan and others on a myexp-published workflow:
> regarding this workflow that I wrote about to Alan earlier:
> >  http://www.myexperiment.org/workflows/166 (Retrieve SNPs from 
> > regions around known genes)
> > it works fine but not with the suggested input, which is an old 
> > geneID that is now archived.
> > I am running it with a list of two geneIDs:
> > ENSG00000139618
> > ENSG00000083093
>
> I am not sure the dbSNP processor is wired correctly -- shouldn't it 
> use a dot iteration strategy? as it is it appears to pick up all the 
> possible combinations of chromosome name
> one for each input gene,  start positions (again one for each input 
> gene) and end positions (same again). In the end it outputs a complex 
> nested list that I don't understand.
>
> I changed the cross to dot product in dbSNP and now I get a much more 
> reasonable 800+ SNPs, organized into two simple lists that make sense 
> to me (I can send them if needed)
>
> it also only takes 30 secs to run now... 
it seems to me that there was a bug in the workflow (but I am not sure), 
which only came up because the output "didn't look right" to me (and I 
barely understand SNPs etc.).
Having made test I/O available would have (a) spotted this and (b) 
avoided having me bother people to verify this.

So I believe this is indeed an important issue...

--Paolo


Paul Fisher wrote:
> I see your point, but, this will in most cases be handeled directly by 
> the service provider. An example is to supply a gene identifier, when 
> a protein identifier is needed. The service would recognise that you 
> have put the wrong id in, as it simply wont return any results, or 
> return an error stating that the input was incorrect. This would be a 
> service-side means of error checking. On the other hand you COULD add 
> in error checks for the entire workflow, but for an example of:
>
> http://www.myexperiment.org/workflows/72
>
> you can quickly see that the size of the workflow will be incredibly 
> large, if each input is to be checked before it is passed to the next 
> service.. Given that people choose to re-use not only on "if it works" 
> but also on the size of the workflow, and many other things, then this 
> may result in workflow no longer being used as they are too big to 
> understand.
>
> Perhaps this discussion should move to the Taverna-Users list instead, 
> as a feature for Taverna or a workflow best practice thought?!?!?!
>
> I do know that a workflow decay monitor has been in production, and 
> there may be plans for its' integration into other projects, such as 
> BioCatalogue. Not that I want to speculate here,
>
> regards,
> Paul.