Why do so many programs use relation databases instead of loading the data during startup and keeping it in memory? Especially for smaller datasets I would think, that a database adds unnecessary complexity and overhead. Also, a lot of data can be saved using modern RAM and when using an in-memory approach, optimized data structures can be utilised to further improve the performance
Edit: yes I meant relational databases
Because the data needs to be persistent?
Why do so many programs use rational databases instead of loading the data during startup and keeping it in memory?
I presume you’re referring to relational databases instead of rational.
The responsibility of a RDBMS is to implement a set of performance-focused data structures that help clients reliably get the data that they need in the fastest possible way, without having to reinvent the wheel.
More often than not, this data does not fit in the heap.
Also, in many usecases there is more than a single client.
Hope this helps.
First, persistency. You data lifecycle may not be directly proportional to your applications lifecycle. You may need it even after the app is shut down.
Second, RDBM systems provide a well defined memory/storage structure and API - “structured query language”. This enables you to easily query your data and acquire suitable views that are beneficial for your applications purposes.
Third, It’s always better to outsource your data layer to a battle tested, and trustworty database then trying to reinvent the wheel.
So this paves a road for you to focus on your business logic than splitting that focus for the data layer and business logic.
Relational database/RDBMS? It’s because the added complexity was necessary or desirable for some reason - relational databases are pretty good at managing data fairly quickly, often with features to deal with timing issues, concurrency, transactions, security, auditing, replication… They have theory going back decades and are still highly relevant. Same as any other software offering, you sometimes expect it to provide features you can’t, won’t or shouldn’t implement yourself.
The overhead is likely negligible, or was considered a fair tradeoff, or the database is actually better at its job in the given scenario. Hopefully. Sometimes people really do add shitloads of very unnecessary overhead by mistake, or overengineer their solutions terribly (cause it’s fun)