nodalpoint.org - best practices - Comments

Taverna

mart1nus — Wed, 11 Apr 2007 16:35:06 -0400

I don't get it. The diagram is too complex? Of course it is complex. But everything is pointing towards a workflow (SOA) world. Why does bioinformatics deny this turn? Oracle, SAP, IBM and so on all rewrite all their applications so they are able to use BPEL as graphical buil environment for integration processess. So stick in the 90s and keep using scripts, but within a few years we all will use workflows in a SOA world. I really believe this will happen, IBM believes in it so does the whole IT world. Let's see who's right.

A hack (not too kludgy)

pjw — Wed, 28 Mar 2007 09:02:33 -0400

Designate one of the multiple output as a representative file (say foo.psl), touch the other files at the end of the commands, and have the rule: foo.log foo.whatever ...: foo.psl

Doesn't change a thing

Antonio Piccolboni — Fri, 23 Mar 2007 00:22:26 -0400

What you wrote unfortunately is equivalent to what I wrote. It's like a shorthand notation for two separate rules, I should have explained it right away. It is going to execute the command twice. What we need is a way to express that a command has multiple outputs, and rules with multiple targets don't accomplish that.

makeovers

Ian Holmes — Wed, 21 Mar 2007 12:26:55 -0400

We certainly would want the replacement to look almost (if not exactly) like make. It should definitely be as simple as make -- that's probably the primary design goal: that you can cut and paste from the command line to a pipeline description file, with minimal extra typing. I'm fully aware of the dangers of excessive re-engineering...

Thanks for the other links... we'll certainly look into them, and post discussion summaries on biowiki.

http://biowiki.org/IanHolmes

Why not scripting?

Duncan — Wed, 21 Mar 2007 04:20:40 -0400

Scripting is great, a powerful tool that lets you achieve world peace in three lines of PERL/Python/Ruby. But what if people don't want to hack scripts? According to Grady Booch, the history of software engineering is one of increasing levels of abstraction, which is where Taverna and workflows are trying to go. Admittedly, we're not quite there yet, sometimes the abstractions leak, and as Stew says, you end up hacking BeanShell, thats not really a problem with Taverna or workflows, its an inherent problem in bioinformatics data, a flat-file legacy nightmare, that means we'll be forced to hack scripting languages for a long time to come, whether we like it or not.

Lets agree to disagree

Duncan — Wed, 21 Mar 2007 04:17:27 -0400

Many taverna users I've been in contact with find the visualisations very useful. Clearly you're not one of these people, so lets agree to disagree.

make alternatives

maximilianh — Tue, 20 Mar 2007 19:34:27 -0400

There's also a page on freshmeat with a collection of make alternatives. For Python fans: Scons.org

make alternatives: makepp?

maximilianh — Tue, 20 Mar 2007 19:17:47 -0400

Great, these wiki pages. My web-searching-reflexes still aren't good enough. I haven't searched for "bioinformatics pipelines make" before submitting the post.

Has anyone of you Perl coders tried makepp ? Though it doesn't address many issues raised, it might be 1) a step into the right direction while maintaining compatibility (a cherished concept for us) and 2) Might serve as a base one day, as I guess you prefer to modify rather Perl code than C code.
I just hope you won't embark into the LISP/Scheme/Prolog direction. Whereas it might be tempting to write a completely new make system, something that keeps at least some superficial compatibility can lure many more people into trying it out than a new system that we would have to learn from scratch.

Would you mind posting some results from your discussion to the frontpage of biowiki or here, for those people that don't live in the bay area?

model multiple outputs in make

maximilianh — Tue, 20 Mar 2007 18:38:32 -0400

hm... I might have missed something, but why don't you write:
f.psl f.log : f.fasta
blat > f.fasta 2> f.log

made guys

Ian Holmes — Tue, 20 Mar 2007 12:40:18 -0400

Thanks for that Chris Lee paper... I've been using 'make' for pipelines since 1996 (I actually built my entire thesis using makefiles, from analysis to latex to postscript). However it has serious limitations, as documented by Andrew Uzilov on our wiki.

As Jason pointed out, we are offering a Summer of Code proposal (applications this week!) to boost 'make' past these limitations. This largely grew out of discussions with Chris Mungall on his biomake project (which also has a biowiki page detailing some of the high-level design goals).

I think Chris has done as good a job as anyone of defining the high-level goals for such a tool: declarative structure, shell script hooks, flexible dependency tracking (MD5 etc rather than just timestamps), facility to build database tables rather than just files, advanced pattern-matching (not just one wildcard per rule, as in 'make'), parallel execution on a cluster and (ideally) a Turing-complete functional programming syntax so that you can start to do low-intensity computation within the pipeline language itself.

Sadly, with Chris now doing more ontology work than genome annotation, biomake has stalled somewhat. There are some practical alternatives, e.g. Perl modules such as Shengqiang Shu's SAPS modules (used by the Berkeley Drosophila Genome Project for their pipeline); and then there are some pie-in-the-sky (but theoretically appealing) functional language-based alternatives, like Erlang (based on Prolog) or Termite (a dialect of Scheme/Lisp).

In the meantime, there are several versions of make that can use GNU make's remote stubs feature to do parallel execution on a cluster. These include distmake, qmake and omake (which also has MD5-based dependency tracking). And of course there are things like Apache Ant, but then you're moving too far away from the command line for my liking, personally ;-)

As you might gather from that last sentence I'm not exactly a proponent of the Taverna-style approach. I like what they're doing but I completely agree with the previous commenter that graphical editors tend to become an end in themselves. I think that there is quite enough to do with developing a workable domain-specific declarative programming language for pipelines without trying to build Yahoo Pipes at the same time. But that's just me, I'm a born-in-the-20th-century fogey; what can I say.

If anyone reading this is in the Bay Area at noon on Wednesday April 4th 2007, btw, we're having a lab meeting to discuss exactly this issue (make and successors). Come to my lab, 425 Hearst Mining Building on the Berkeley campus, and meet some other "made guys".

http://biowiki.org/IanHolmes

diagrams

maximilianh — Tue, 20 Mar 2007 11:25:06 -0400

The workflow diagram that you provide is for me not a real diagram. It's too complicated. I don't see a lot of value in it.

If a diagram is getting complicated it's taking too much space in you paper, isn't it? I'd rather write the _details_ as a text.

If a diagram is simple, I can also draw it myself and optimize the layout in a more human-friendly way than graphviz can.

We know the problem from software engineering, right: I don't believen in the added value of UML-generating tools. You end up with thousands of diagrams that no one can read anymore. If diagrams are created automatically they loose their value as a means to simplify and over-simplify the system.

Biologists are using textual protocols for a reason: They are much more compact and as everything is linear for them, there's no real need for the expressive power of flowcharts: complex branches or repetitions. I don't have these neither in my pipelines as they are rather linear in structure.

diagrams and pipelines.

dalloliogm — Tue, 20 Mar 2007 10:31:41 -0400

Too bad you don't use diagrams to describe your work.
It's very useful to show how your pipeline works to other people, and to yourself.. it helps you in writing the right test units for your programs and to see which improvements/changes you can do to them.
Also, it's easier to compare two experiments when they are described with a diagram produced with the same syntax.
I wonder why wet biology scientits don't use diagrams to describe their experiments when writing papers, too (bioinformaticits will be very happy then).
If you try to draw the diagram of the pipelin you've described in your post, I will understand it easily.

I agree with your sentiment

Greg — Tue, 20 Mar 2007 10:28:31 -0400

I agree with your sentiment regarding generalized 'ubertools'. So much effort creating is expended creating a generalized workflow editor, that the editor becomes and end in itself.

Another thing that gets me about graphical workflow editors is that for anything other than the most common tasks, you need to revert to scripting to plug outputs into inputs. So why not scripting in the first place ? Maybe I'm not the right audience for Taverna...

powerpoint vs graphviz diagrams

Duncan — Tue, 20 Mar 2007 09:50:50 -0400

You could draw them in PowerPoint, but if they start getting complicated, it is time consuming. Having a standard way of drawing pipelines/workflows is a handy tool for quickly communicating what your analysis does graphically. Taverna uses GraphViz to do this, see for example this workflow diagram. I dunno about you, but I wouldn't fancy drawing that figure manually in PowerPoint, when GraphViz can do most of the hard work for me.

taverna papers

Duncan — Tue, 20 Mar 2007 09:43:23 -0400

Not sure what you mean by "applying taverna for papers", but here is one example of what I think you might be interested in, the successful application of taverna workflows to a difficult problem.