Tl;dr: Automatic updates on my home server caused 8 hours of downtime of all of renn.es’ docker services including email and public websites
Tl;dr: Automatic updates on my home server caused 8 hours of downtime of all of renn.es’ docker services including email and public websites
Blind automatic upgrades are a bad idea even for casual home users. You could run into a Linus Tech Tips “do as I say” scenario where it uninstalls half your system due to a dependency issue. Or it could accidentally uninstall part of your system that you don’t notice.
I’m not sure how stable Gentoo’s default branch is but I know that daily upgrades on Arch Linux is close to suicide - you have a higher chance of installing a buggy package before it’s fixed if you install every package version as it comes in.
I’m surprised this strategy was approved for a public server - it’s playing with a loaded revolver and it looks like you were finally shot.
The whole “do as I say” prompt existed to make sure that stuff doesn’t happen automatically. It’s rather unfortunate that people don’t read the warnings and just continue when something is clearly wrong, but with that problem fixed by just plain refusing to do some operations I think updates aren’t that big of a risk.
I wouldn’t run headless automatic updates on a server you’re not willing to spend a day on fixing at the worst moment you can imagine, but automatic updates don’t have to be bad. Debian/Ubuntu’s unattended-upgrades script have never caused me any trouble, especially with their stable release schedule. Maybe they cause a few minutes of performance issues and a few dropped connections, but the automatic security updates are worth the minor inconvenience.
The goal was to avoid getting hacked on a server that could have many vulnerable services (there are more than 20 services on there). When I set this up I was basically freaked out by the fact I hadn’t updated mastodon more than a week after the last critical vulnerability in it was found (arbitrary code execution on the server). The quantity of affected users, compared to the impact it would have if hacked, made me choose the option of auto-updates back then, even if I now agree it wasn’t clever (and I ended up shooting myself I’m the foot). These days I just do updates semi-regularly and I am subscribed to mailing lists like oss-security to know there’s a vulnerability as early as possible. Plus I am not the only person in charge anymore.
I’m not a real sysadmin so take it with a grain of salt, but in all reality this is probably why you would choose something like Debian for a server instead a bleeding-edge distro. Debian quickly backports security updates and fixes but otherwise keeps everything else stable and extremely well-tested, which pretty much 100% prevents serious bugs from reaching its Stable branch. You may still need to figure out an appropriate strategy for keeping your Mastodon container updated, but at least the rest of your system isn’t at risk of causing catastrophic errors like this. Also, Debian Stable does allow you to auto-upgrade security patches only, if you still want that functionality.
I totally agree. But I just wouldn’t necessarily say gentoo is a bleeding edge distro: it’s kinda up to the user. They are free to configure the package manager (portage) however they want and can even do updates manually. I just like the idea of having newer packages at the cost of stability, because I also use the server as a shell account host (with an isolated user ;-)) and need things like the latest neovim. These days I would know if an update failed because I would literally be in front of the process and test services are working after the updates, so I’d know if I have to rollback. This makes it basically like a stable distro IMO (even though the packages aren’t battle tested before being pushed as updates).
I don’t know to what extent you got molested by the prophets of immutable distros yet, but I can only recommend to join the cult. Install Fedora IoT (or CoreOS) and simply know that you’ll get a working container host (powered by podman) with every update. The whole discussion about which distro might survive whatever massacre the respective package manager commits next becomes superflous: You simply get the next image that was built upstream solely to serve containers. The whole package-udpating-shengiangs is done by other people for you, you only collect the sweet result. The only “downside” is that one has to become familiar with containers, but since you run docker already that should work out. Also for stuff like tinkering with the latest tools, just put those in a distrobox. That way they are indipendent from your solid container host, and you can mess them up in whatevery way you fancy and dispose them without any traces left behind.
Edit: To give one more example why this is awesome: It wouldn’t even matter which one you install, you can just rebase to the other (IoT lives in the
fedora-iotremote. silverblue, coreos and the others in thefedoraremote. Just for anybody who might be confused by only looking atostree remote refs fedora)