nodalpoint.org - The End of Relational Databases? - Comments

RDF Databases

cjm — Fri, 27 Jan 2006 22:39:38 -0500

I too am excited about the possibility of using RDF databases but disappointed by current performance issues. There's some obvious optimisations they could be doing wrt transitive closures, hopefully that will come soon.

You may want to check out chado - http://www.gmod.org/schema

In some ways it's a hybrid schema between a truly generic RDF type model and a more traditional relation schema. I think it's quite a nice middle ground, but then I would say that.

Trolls: My remark was to the

Greg — Wed, 25 Jan 2006 06:04:36 -0500

Trolls: My remark was to the article in BioIT-World - your coverage is perfectly adequate.

Thank you for clarifying.

The problem is not so much the suitability of relational databases for storing data, but in terms of their flexibilty for integrating data. Every data warehouse implementation (SRS, Discovery Link) requires maintenance of the various wrappers used to pull-in data from external data sources. Each one of those wrappers is ad-hoc as the underlying relational framework does not facilitate that kind of diverse data integration.

RDF goes some way towards solving the flexibilty problem, but does have performance issues (not to mention higher level semantic issues). I have also had the opportunity to talk to Kei Hoi Cheung (also of YeastHub) who expressed similar concerns about using RDF databases (in their case Sesame) for large scale data storage. Even given the limitations I will still be pushing RDF as a solution locally as I see this as a research opportunity for myself (let's see how quickly that gets crushed by the powers that be and the LIMS vendors).

At the end of the day, technology choices are hard.

Alternative trolls

Spitshine — Sat, 21 Jan 2006 12:01:14 -0500

Trolls: My remark was to the article in BioIT-World - your coverage is perfectly adequate.

In 1998, Lion Bioscience, my employer back then, commercialized the SRS system (the current EBI installation), which was targeted to replace relational databases in the life sciences. Thus, I've heard the requiem before. However, SRS could only only serve the biological knowledge side, not the transactional side that you need for a LIMS.
Nowadays, you can connect SRS to relational databases in every which way, using Oracles Gateway or IBMs Discovery Link and many other things. But SRS is only one of many ways to tackle to problem.
All of the modern relational databases can store and query XML documents, thereby removing many limitations on rich data structures - I don't really see relational DBs coughing, let alone dying.

Alternative: RDF is all nice and shiny but seems to loose on the performance end. I talked to Andrew Smith (yeasthub) and you pay for the flexibilty by CPU time. I would be surprised to see a system which is performant on normalized and highly non-normalized, rich data structures at the same time. However, semantic web technologies appear very appealing for the biological knowledge representation.

Or do you think such RDF based extensions to Oracle can offer the speed and transactional safety of the highly optimized table based system that it relies upon?

industrially strengthen and rock solid ?

agbiotec — Fri, 20 Jan 2006 12:24:08 -0500

We are missing an important point here. From an academic point of view, of course we must think and work towards making our bioinformatics future better through more semantics and integration of our data. But how about data holding vaults (see GUS implemented in Oracle) for big institutes or companies in bioinformatics ? Sure AceDB holds sequence + expression data from 2-3 species. But how about 10 genomes (PATRIC) ? What I am advocating for, is that those kind of big-mama DBMS can hold tera-bytes of data, and they are industry rated for they maintenance capabilities. Their DBMS server is rock-solid, and you can build whatever you want on it, without being afraid that it will break on large volumes of data.

But they cost big bucks (including the certified DBAs to manage those complex systems). And true, modeling biological data in relational schemas can be tough. I believe that Myhrvold has got a point in his writing, but the big change will come when those BioIT - specific database systems (developed by academic labs, usually pushed by the industry) reach some maturity. An by maturity I mean all the above I mentioned for those big-mama DBMS above.

Another point is that open-minded people working in BioIT (there are some still doing it with Excel) should be trailing along with those BioIT-specific DB systems as they mature. In this way those systems can end-up having the right objects for biological data, the right query mechanisms, plus semantic capabilities. Otherwise we'll be at the starting line again, trying to integrate the non-homogenous sources.

Me trolling ? Never. At the

Greg — Fri, 20 Jan 2006 02:38:22 -0500

Me trolling ? Never.

At the moment I'm dealing with two LIMS systems, one for microarray data and the other for proteomics, both are backed by relational databases. I know at some point I will be asked to 'mine data' from both. The issue for me is how easy will that be ? Assuming that both systems provide development libraries (object interfaces etc.) for the relational stores this will be reasonably straight forward after investigating both schemas and writing a few scripts. In this sense relational databases are 'adequate'.

These 'interfaces' (or APIs) are what Chris Dwan is referring to in his article Interfaces Will Save the World. Now I would argue that the interface technologies he suggests, SOAP/WSDL, are not likely to save the world, more likely it will be simple REST based interfaces like those provided by the NCBI (Eutils). Irrespective of what technology is used, SOAP or REST, APIs do not help you deal with data structure heterogeneity i.e. tables vs. tress vs. graphs. On top of data structure heterogeneity you then have to deal with data type heterogeneity.

The alternative that you missed is using a single data structure to eliminate the first level of heterogeneity. This is what the Resource Description Framework (RDF) is all about and why it is the underlying technology of the semantic web. Thus the direction we need to be looking is RDF databases, for example Kowari, Sesame, Oracle's semantic extensions and maybe Google's BigTable. Ontologies come into play at the next level, data type heterogeneity. Using ontologies, ontology mappings and automated reasoning to sift through data in an RDF database is where we want to be going. Not installing yet another LIMS with a relational database backend.

BioSQL

Neil — Thu, 19 Jan 2006 22:21:37 -0500

I've played around with it. It's not very mature and I don't think there's a lot of active development, judging by the CVS activity.

It does seem like a possible solution to local storage though. In theory you can take any of the common biological databases and just drop them into the schema. If you understand the schema, it should be trivial to hook up a CGI web frontend with Perl/PHP, extract sequences using scripts and DB bindings and so on. There's a guide to setting up and using BioSQL with a Swissprot example at this Bioteam web page.

I think relational databases tend to be slow when used in this context mostly because biological databases are so large. There are plenty of ways to optimise your databases though - check the mysql documentation. People are always telling me that postgresql performs a lot better but I've never looked into it.

Fixed.

Greg — Thu, 19 Jan 2006 22:06:26 -0500

Fixed.

To paraphrase Winston Churchill...

Stewb — Thu, 19 Jan 2006 06:06:03 -0500

It has been said that relational databases are the worst form of genomics data storage except all the others that have been tried.

Is there a viable alternative to established relational databases? Perhaps if development of AceDB started up again. Personally I find the hassle invovled in twisting table schemas to meet your needs gets outweighed in the long run by speed, ease of use and connectivity (try connecting to AceDB from, say, a .NET app).

Does anybody have experience with the BioSQL schema? Is it any good (I suppose it might depend on what you want to do with it)?

Broken link

karbak — Thu, 19 Jan 2006 05:36:40 -0500

There's a loose <br> tag at the end of the Bio-IT World link.

Did I miss the alternative?

Spitshine — Thu, 19 Jan 2006 05:33:42 -0500

If I would post the original story, people would laugh or accuse me of trolling. The problems we are facing in Proteomics research are not at all different from other data base systems, so relational solutions seem to be as adequate. Besides, when was the last time you put "knowledge" in a LIMS. Warehousing solutions (separating the transaction and the analysis elements on a larger scale) seem appropiate the last time I looked at it...

Anyway, I can only re-iterate your last statement - the real problem is lack of support and resources from stakeholders who are "not interested how it works". While the problem is abundant in IT, it is pretty severe in *omics research as you have to tackle complicated data structures that are not at all well studied yet (compared to address fields and parts lists).

The End of Relational Databases?

Greg — Wed, 18 Jan 2006 04:09:28 -0500

Nathan Myhrvold is predicting the end of relational databases for *omics data storage:

'heavy reliance on relational databases like Oracle, SQL, and DB2 within the field of proteomics and genomics is "completely boneheaded."'

This quote is from a short news piece on BioIT World. On the one hand it is quite a glib statement to make, "the end of X as we know it...", however there might be some truth to it...