nodalpoint.org - Database or flat text file? - Comments

Try X-Path it is pretty

paladinjack — Wed, 13 Jun 2007 14:52:31 -0400

Try X-Path it is pretty straightforward. You need to use libraries. Perl and Python i believe they all have X-path libraries.

XML is the hit!

paladinjack — Wed, 13 Jun 2007 14:51:20 -0400

However when you have a huge XML file X-Path could not be working as it needs to parse the entire document into tree structure for querying, and it could be slow. For a reasonably medium sized dataset, XML is a good substitution for database.

For splicing, python dictionaries could be better than databases

dalloliogm — Tue, 12 Jun 2007 12:00:56 -0400

Read this document:
- http://bioinfo.mbi.ucla.edu/pygr_0_5_0/seq-align.html#SECTION00011000000...

they say that databases could be complicated to use if you work with splicing or alternative splicing, or at least that you should look for some libraries.

Bye :)
--
http://genome.imim.es/~giovanni

Databases or XML? Invest in your future...

Duncan — Thu, 10 May 2007 06:50:01 -0400

There is quite a steep learning curve with both relational databases and XML. However, whichever you finally choose, they are both worth learning and will pay dividends in the long run.

I also thought about XML,

lbbros — Wed, 09 May 2007 16:26:30 -0400

I also thought about XML, but I found it quite complex, mostly because of the learning curve involved (which looked steep at the time I investigated). I tried to write something simple to parse Entrez Gene records (as a means to get a common ground for my datasets) in Python back then but I had to admit defeat due to lack of knowledge.

Thanks for the suggestions.

lbbros — Wed, 09 May 2007 16:20:51 -0400

Thanks for the suggestions. I thought about a database exactly because I had to infer relationships between my dataset and some others I obtained from the literature, and some of those relationships weren't exactly straightforward. I am into stage two at the moment, using SQLite as I don't want to pollute our MySQL database (we have an internal server with many dumps from the UCSC Genome Browser and other things).

I had to put things on hold right now (the deadline for the summary of the thesis is next month), but I'll resume soon. I'll give a look to UML for sure and decide if it's worth implementing something more complex.

Database or flat file? Could try XML

Duncan — Wed, 09 May 2007 13:26:37 -0400

Its horses for courses, as people have said. If its a one-off hack, that nobody (including you) will ever want to re-run or re-use, flat files are the thing. However, this is rarely the case. If you don't want to go through all the hassle of learning about relational databases, you could try XML. It doesn't force you to come up with complex schemas from the beginning, but gives you powerful mechanisms for querying and manipulating your data (XPath, XQuery and XSLT). You can still have simple flat-files of course, but you won't have to write your own parsers, since there are tons of XML parsers out there to choose from. Just my $0.02 ... hope it helps.

It does indeed depend

Neil — Wed, 09 May 2007 06:37:49 -0400

Greg has it about covered. It's appropriate to use a database when it's inappropriate to use a flat file. And vice versa :)

If you don't need (a) structure and (b) queries for the data in your current project, then stick with flat file for the sake of speed. However, it's well worth learning databases at some stage in your career. The best way to learn new stuff is to apply it to one of your projects, rather than just saying "well, I'll teach myself one day when I have a moment". You'll never have a moment and trying to learn something in an abstract sense is much harder than in a practical sense, where you can see the relevance to your data.

There are three stages to

Greg — Tue, 08 May 2007 14:16:45 -0400

There are three stages to this. First you start out with a tab delimited text file. This is great for parsing using scripting languages, and writing simple queries, again using scripting languages. The problem is that at some point your intuition tells you that you should be doing more to structure your data, find relationships in your data etc. This then leads you to the never ending quest for the right database and the right schema. It is easy to dismiss relational databases at this point, when schema design and database choice side track you from your real goal.

My suggestion is to move on to stage two, use a relational database, but only for the added power that SQL gives you when querying a single table. It doesn't matter which one you use, MySQL is quite straight forward, and you don't need a complex schema. Just dump your existing flat file into a simple table that models the existing data types. Now you can use your scripting language's MySQL integration to pull out rows, query the table using conditions and limits etc. without having to write ad hoc query code in you analysis scripts.

If you think you can avoid the trap of finding the right schema/database, then moving on to stage three, and trying to model relationships in your data might be worth it. The setup time will cost you, but if you do it right, then you will have the full power of SQL to do your data mining. Some kind of visual modeling language can be helpful (e.g. UML) but relational database modeling is also an 'end in itself', try to avoid it at all costs, and focus on getting the best scientific value from your data.

database or flat file - it depends

dalloliogm — Tue, 08 May 2007 09:33:07 -0400

It depends on what are your skills with computers and databases.

Actually I'm using flat files, because of three reasons:
- I don't have too much data and most of the files I have are fasta sequences of gffs;
- I don't know very well how to work with databases, I think that sometimes it's easier to use grep/perl scripts from the unix command line instead of interrogating the database every time.
- I'm too lazy to ask the systems managers to create a database for me, and I don't want to bring my laptop every day at work :(

Anyway I believe that if you know how to handle data in a database you should do it, read for example this post:
- http://www.bioinformaticszen.com/2007/02/bioinformatics-use-a-database-f...

---

http://dalloliogm.wordpress.com

Database or flat text file?

lbbros — Wed, 02 May 2007 14:11:12 -0400

As the deadline for my Ph.D. thesis looms over me, I've been working recently with a certain quantity of data of the same type (basically a variant of microarray gene expression) but from different sources and/or software.