Hey all!
I’m seeking guidance. I have a lot of apps that use ORMs like Mongoose or Sequelize or Sqlalchemy, or even just init-db scripts with raw SQL. Point is a lot of apps have changes to the data layer. When we program and make apps they tend to describe how they need to see a database. Sometimes its no trivial feat to do an upgrade - and even if you have a CI/CD pipeline in place what does that look like to have something like even lemmy upgrade in a container.
If you have these apps in production how are you handling these apps? Both developer perspective and devops welcomed.
I see the devs offering insight into maybe how the backend recognizes database state, while the devops perhaps either describing process and implementation.
For me(programmer) it usually depends on the upgrade.
The system I am working on contains like 5-10 containers (depending on customer needs).
In generel I try to have as few connections between services as possible, Low Coupling, High Cohesion. No 2 containers share a database scheme, because that makea database updates way more difficult. The 2 services have to talk to each other through a welldefined API which can be versioned.
Application bugfix Sending out a bugfix usually only effects the actions of a single container. Then we upload the new docker image to the server, shuts down the old instance and start the new container, but reuses all the settings of the old container. This does result in like 10-20 seconds downtime, which is completly acceptable in my case.
Application feature When we implement a big new feature we usually also needs to update the database (new tables/columns). First take a backup of the database before the upgrade. Then like with the bugfix upload and start the new container. In our case we have the application push out the database update (C# dotnet EF core). If it is an even bigger update we may have multiple docker containers that is upgraded at the same time.
Database version upgrade We check the release note for the specific database on how to upgrade it, and follow those steps.
Other utility containers (Nginx, Elastic search, …) Same as database, check the release note for breaking changes to make sure the upgrade goes well.
Always always always make sure to test the update in a controlled environment that resembles the production environemnt as closely as possible. (maybe even a copy of production if I am lucky)
This also assumes you have adequate testing capabilities which are important so that you can evaluate unit capabilities, integrated capabilities, and performance characteristics.
On cloud you might have your testing infra be ephemeral and spin up whenever you need to perform the testing (as well as your IAC capabilities).
But this would partially evaluate any issues with the app coordinating with the database.
I would also assume DBA’s can write some test scripts to execute as well.