One of the projects that I worked on this year (not related to BitWise) involved storing and processing a decent amount of data in a PC application. I had sketched out on paper all sorts of elaborate structures with hash maps instead hash maps, sorted arrays, lists, and ways to link them all together. I was really dreading implementing all of this, especially writing the algorithms to join the data together. Then I got this nagging little thought in the back of my mind -- "Isn't there an embedded database that I can just plop in here and have it manage all the data?"
It didn't take me too long to find SQLite, which is the library that I remembered reading about. It become quickly obvious that this was the best choice for the application's data management needs. Suddenly, all of those ugly data structures and algorithms just disappeared, and I was writing elegant SQL queries to get the data that I needed. I hit the ground running and that project was a success.
Looking back, if we started a "from scratch" version of BitWise, I think the same approach could be used to manage things like the contact list, file transfer manager, preferences, etc. Let's just say that the underlying data structure and management for some of these are pretty involved! Using a database would also easily allow for persistence in the file transfer manager, which has been requested several times in the past. It's amazing how your data management problems just dissolve into nothing with the power of a database.
So that leads me to wonder: is the database the ultimate data structure? Who needs linked lists or hash maps when you can just use a database? The SQLite compiled library can be integrated into your application with less than half a meg of overhead, which is pretty insignifacnt considering what you get for the investment. You can even get encryption and compression modules, the former purchased for the project that I worked on this year.
So the next time I sit down to develop something new, I'm not going to think of hash maps, arrays and linked lists. I'm just going to think of a database.
I've used a MySQL database for my website for several years now, but nothing more complicated than a guestbook, news posting(blogging) and commenting systems.
At work I used an Access Database for slightly more complicated systems.
The benefit of databases is so great that once you use a database, it's hard to go back to any sort of other data storage. I've recently explored XML in class, and found it to be a pain to use compared to a database.
Unfortunatly there is a downside to databases, and that happens during distribution. If you want to provide client software with a database for storage, you must provide them with the database server.
In the future I see an integration between database servers and client software. Perhaps a system wide database where applications are free to write data into. Structured similar to current databases, but one that would be structured more like so: Database - Application - Table - Row - Field. Each application would be seperated from the others, allowing access priveleges as desired and no conflicts. Imagine this as different folders or different keys in the registry, but in a database format.
It would be impractical to have a database server installed for each application you wish to save data with.
I guess I didn't do a very good job describing SQLite, because the downside you talk about is exactly what SQLite solves. There is no database "server" -- the database engine is compiled directly into the program. And the overhead in the executable size is just 1/2 of a megabyte. :)
If you're developing for MacOS X, take a look at Core Data. Core Data does pretty much exactly what you're describing w/ SQLite out of the box. It takes about 5 minutes to create a simple SQLite-powered application -- i.e. address book.