nodalpoint.org - database - Comments

Yeah, need to update my

ejain — Tue, 22 Jan 2008 19:47:42 -0500

Yeah, need to update my blog...

700?

nuin — Tue, 22 Jan 2008 13:18:11 -0500

At 700 a pop for an Action Figure?

Dude, Where's My NAR?

Duncan — Tue, 22 Jan 2008 13:09:27 -0500

Hello Maximillian, I wasn't attacking NAR, was just wondering about what proportion of the data is dead. It is an interesting technical challenge to find this out. It would also be useful to measure the cost of gathering noisy, redundant and poorly understood data in terms of wasted resources (people, time, money, computers, false positives etc). Perhaps somebody has done something like this already? Especially with all the irresponsible sequencing just for sake of it that goes on...

As for the physicists, I mentioned them for comparison. Like you, I doubt they will use 100% of their data either, but will probably use much more of it. However, they keep delaying switching their big machine on, which means they are still waiting for the data. That is, unless when they finally flip the switch on the LHC, we all disappear into a black hole, tombs and all :)

It doesn't matter, they got published

lbbros — Tue, 22 Jan 2008 06:46:50 -0500

Does it matter that large quantities of this data will probably never be used?

Probably not, as their authors are already happy enough to have those published, and then they can ~~stop mantaining~~ move on to another ~~project~~ publication. It is indeed a sad state of affairs.

Swiss Prot Databases / Action Figures

Duncan — Mon, 21 Jan 2008 18:12:33 -0500

Hi Eric, yeah even more redundancy there. Talking of SWISS Prot people, I'm just wondering when your Amos Bairoch action figure will be available in shops?!?! What are you up to now that you've moved on from SWISS Prot to Seattle?

Why NAR

maximilianh — Sun, 20 Jan 2008 10:59:14 -0500

I add the usual defense of the NAR databases: At least they are indexed by pubmed, so as there are biologists that are not used to checking in google if there are websites about their subject, they will find them quicker via NAR. The data is not dead, websites can always be exported. (I've made the experience that for smaller databses it's usually quicker to scrape the data from an html page with something like HTTP::Recorder than writing to the person responsible for the data to send you an sql dump.) In addition, people get papers for their databases like this and other people can cite the database properly, so NAR makes the web citable and advances someone's career a little bit. It already eases the transition from a traditional paper-based science to a more web-orientated world. Databases are peer-reviewed, so they don't contain complete crap, a paper in NAR assures some minimal quality. And a write-only database is better than none at all, at least someone has collected something and you can scrape it from its tomb.

I would just love to see a minimal requirement for a publication in NAR: They should all offer some simple text-based export, e.g. tab-delimited flatfiles. That could save me a lot of time...

Of course, large quantities of these data are never used and never read. But, heck, this is research, right? 90% will not be used in the end. It's not too different from those 500 alignments algorithms, 200 genomic analyses, hundreds of papers that describe "new" cloning strategies, yet another new gene, etc. I don't believe that the physisists use 100% of their 1,5 GB / sec either.

Also, think of all the

ejain — Fri, 18 Jan 2008 20:12:16 -0500

Also, think of all the redundant effort that goes into setting up the basic technical infrastructure each database needs (data storage, user interfaces etc)...

Try X-Path it is pretty

paladinjack — Wed, 13 Jun 2007 14:52:31 -0400

Try X-Path it is pretty straightforward. You need to use libraries. Perl and Python i believe they all have X-path libraries.

XML is the hit!

paladinjack — Wed, 13 Jun 2007 14:51:20 -0400

However when you have a huge XML file X-Path could not be working as it needs to parse the entire document into tree structure for querying, and it could be slow. For a reasonably medium sized dataset, XML is a good substitution for database.

For splicing, python dictionaries could be better than databases

dalloliogm — Tue, 12 Jun 2007 12:00:56 -0400

Read this document:
- http://bioinfo.mbi.ucla.edu/pygr_0_5_0/seq-align.html#SECTION00011000000...

they say that databases could be complicated to use if you work with splicing or alternative splicing, or at least that you should look for some libraries.

Bye :)
--
http://genome.imim.es/~giovanni

Databases or XML? Invest in your future...

Duncan — Thu, 10 May 2007 06:50:01 -0400

There is quite a steep learning curve with both relational databases and XML. However, whichever you finally choose, they are both worth learning and will pay dividends in the long run.

I also thought about XML,

lbbros — Wed, 09 May 2007 16:26:30 -0400

I also thought about XML, but I found it quite complex, mostly because of the learning curve involved (which looked steep at the time I investigated). I tried to write something simple to parse Entrez Gene records (as a means to get a common ground for my datasets) in Python back then but I had to admit defeat due to lack of knowledge.

Thanks for the suggestions.

lbbros — Wed, 09 May 2007 16:20:51 -0400

Thanks for the suggestions. I thought about a database exactly because I had to infer relationships between my dataset and some others I obtained from the literature, and some of those relationships weren't exactly straightforward. I am into stage two at the moment, using SQLite as I don't want to pollute our MySQL database (we have an internal server with many dumps from the UCSC Genome Browser and other things).

I had to put things on hold right now (the deadline for the summary of the thesis is next month), but I'll resume soon. I'll give a look to UML for sure and decide if it's worth implementing something more complex.

Database or flat file? Could try XML

Duncan — Wed, 09 May 2007 13:26:37 -0400

Its horses for courses, as people have said. If its a one-off hack, that nobody (including you) will ever want to re-run or re-use, flat files are the thing. However, this is rarely the case. If you don't want to go through all the hassle of learning about relational databases, you could try XML. It doesn't force you to come up with complex schemas from the beginning, but gives you powerful mechanisms for querying and manipulating your data (XPath, XQuery and XSLT). You can still have simple flat-files of course, but you won't have to write your own parsers, since there are tons of XML parsers out there to choose from. Just my $0.02 ... hope it helps.

It does indeed depend

Neil — Wed, 09 May 2007 06:37:49 -0400

Greg has it about covered. It's appropriate to use a database when it's inappropriate to use a flat file. And vice versa :)

If you don't need (a) structure and (b) queries for the data in your current project, then stick with flat file for the sake of speed. However, it's well worth learning databases at some stage in your career. The best way to learn new stuff is to apply it to one of your projects, rather than just saying "well, I'll teach myself one day when I have a moment". You'll never have a moment and trying to learn something in an abstract sense is much harder than in a practical sense, where you can see the relevance to your data.