DevOps as a profession and software development for fun. Admin of lemmy.nrd.li and akkoma.nrd.li.

Filibuster vigilantly.

  • 0 Posts
  • 21 Comments
Joined 1 year ago
cake
Cake day: June 10th, 2023

help-circle
  • Laptops/desktopes: no real naming scheme, they use non-static DHCP leases anyway.

    Physical servers: NATO phonetic alphabet. If I run out of letters something has gone terribly wrong right.

    VMs: I don;t have many of these left, but they are named according to their function and then a digit in case I need more. e.g. docker1, k3s1. This does mean that I have some potential oddities like a k3s cluster with foxtrot, alpha, and k3s1 as members, but IMO that’s fine and lets me easily tell if something is physical or virtual. I am considering including the physical machine name in the VM name for new things as I no longer have things set up such that machines can migrate… though I haven’t made a new VM in some time.

    Network equipment: Named according to location and function. e,g, rack-router, rack-10g, rack-back-1g, rack-ap, upstairs-10g, upstairs-ap. If something moves or is repurposed it is likely getting reconfigured so renaming at that point makes sense.








  • I switched from Plex to Jellyfin several years ago and haven’t really looked back. Overall I just didn’t like the direction plex kept going (pushing shit streaming services, central auth, paywalling features), and dropped it even though I grabbed a lifetime plex pass back in the day. The only thing I miss about plex was the ease of developing a custom plugin for it since you could pretty much just drop python scripts in there and have it work, though their documentation for plugin development was terrible (and I think removed from their site entirely).








  • Be the change you want to see. Setting up an instance is surprisingly easy, it’s the admin stuff that will take much more time, and finding users that will probably be hard. Also scaling once you hit a certain level of size/traffic, but that’d be a good problem to have. To me the most beautiful part of the fediverse is that if you’re not finding the instance with rules/defederation/etc you want you can make that place exist.

    If you are interested in doing so I’d be more than happy to give what advice or help I can.


  • I expect the moderators of communities to do sufficient policing of their community to make sure it follows the rules of the instance it is on and the rules of that community. If those rules permit something you disagree with (or don’t permit something you do want to see) the power is in your hands as a user to not participate or even see that community. The only way for a user to guarantee they won’t interact with someone from instance X (whether that is exploding-heads or lemmygrad or whatever you don’t like) is to only interact with communities on instances that have them defederated. There are places you can get a more curated and aggressively moderated experience, and have been recommending places such as beehaw to anyone looking for that.

    I will take action against:

    • Local users harassing someone
    • Local users breaking local rules
    • Local users repeatedly breaking remote rules
    • Local communities that break local instance rules
    • Remote users harassing local users
    • Remote users repeatedly breaking local rules
    • Remote instances that repeatedly allow its users to break local rules
    • Remote instances that repeatedly allow its users to harass my users

    The first rule on my instance is a catch-all “Be welcoming”, that will be wielded to aggressively remove far more than just “racism, sexism, homophobia and transphobia”.

    As an admin I don’t have the time or desire to police:

    • Local users interacting on remote communities, so long as they are following remote rules
    • Remote communities
    • Remote users interacting with remote users/communities

    I do hope for a way to better curate (or just disable for now) the “All” feed, at the very least for anyone who isn’t logged in. Given the general rules above that feed may include disagreeable posts, and is not a good representation of my instance or the type of community most users there will experience.



  • Based on the hardware you have I would go with ZFS (using TrueNAS would probably be easiest). Generally with such large disks I would suggest using at least 2 parity disks, but seeing as you only have 4 that means you would lose half your storage to be able to survive 2 disk failures. Reason for the (at least) 2 parity disks is (especially with identical disks) the risk of failure during a rebuild after one failure is pretty high since there is so much data to rebuild and write to your new disk (like, it will probably take more than a day).

    Can’t talk much about backup as I just have very little data that I care enough about to backup, and just throw that into cloud object storage as well as onto my local high-reliability storage.

    I have tried many different solutions, so will give you a quick overview of my experiences, thoughts, and things I have heard/seen:

    Single Machine

    Do this unless you have to scale beyond one machine

    ZFS (on TrueNAS)

    • It’s great, with a few exceptions.
    • Uses data checksums so it can detect bitrot when performing a “scrub”.
    • Super simple to manage, especially with the FreeNAS TrueNAS GUI.
    • Can run some services via Jails and/or plugins
    • It only works on a single machine, which became a limiting factor for me.
    • It can’t add disks one at a time, you have to add an entire vdev (another set of drives in RAID-Z or whatever you choose).
    • You have to upgrade all disks in a vdev to use higher capacity disks.
    • Has lots of options for how to use disks in vdevs:
      • Stripe (basically RAID-0, no redundancy, only for max performance)
      • Mirror (basically RAID-1, same data on every disk in the vdev)
      • RAID-Zx (basically RAID-5, RAID-6, or <unnamed raid level better than 6>, uses x # of disks for parity, meaning that many disks can be lost)
    • ZFS send seems potentially neat for backups, though I have never used it

    MDADM

    • It’s RAID, just in your linux kernel.
    • Has been in the kernel for years, is quite reliable. (I’ve been using it for literally years on a few different boxes as ZFS on Linux was less mature at the time)
    • You can make LVM use it mostly transparently.
    • I would probably run ZFS for new installs instead.

    BTRFS

    • Can’t speak to this one with personal experience.
    • Have heard it works best on SSDs, not sure if that is the case any more.
    • The RAID offerings used to be questionable, pretty sure that isn’t the case any more.

    UnRaid

    • It’s a decently popular option.
    • It lets you mix disks of different capacity, and uses your largest disk for parity
    • Can just run docker containers, which is great.
    • Uses a custom solution for parity, so likely less battle-hardened and less eyes on it vs ZFS or MDAM.
    • Parity solution reminds me of RAID-4, which may mean higher wear on your parity drive in some situations/workloads.
    • I think they added support for more than one parity disk, so that’s neat.

    Raid card

    • Capabilities and reliability can vary by vendor
    • Must have battery backup if you are using write-back for performance gains
    • Seemingly have fallen out of favor to JBODs with software solutions (ZFS, BTRFS, UnRaid, MDADM)
    • I use the PERCs in my servers for making a RAID-10 pool out of local 2.5in disks on some of my servers. Works fine, no complaints.

    JBOD

    • Throwing this in here as it is still mostly one machine, and worth mentioning
    • You can buy basically a stripped down server (just a power supply and special SAS expander card) that you can put disks in, and lets you connect that shelf of storage to your actual server
    • May let you scale some of those “Single Machine” solutions beyond the number of drive bays you have.
    • Is putting a number of eggs in one basket as far as hardware goes if the host server dies, up to you to decide how you want to approach that.

    Multi-machine

    Ceph

    • Provides block (RBD), FS, and Object (S3) storage interfaces.
    • Used widely by cloud providers
    • Companies I’ve seen run it often have a whole (small) team just to build/run/maintain it
    • I had a bad experience with it
      • Really annoying to manage (even with cephadm)
      • Broke for unclear reasons, while appearing everything was working
      • I lost all the data I put into during testing
      • My experience may not be representative of what yours would be

    SeaweedFS

    • Really neat project
    • Combines some of the best properties of replication and erasure coding
      • Stores data in volume files of X size
      • Read/Write happens on replica volumes
      • Once a volume fills you can set it as read only and convert it to erasure coding for better space efficiency
      • This can make it harder to reclaim disk space, so depending on your workload may bot be right for you
    • Has lots of storage configuration options for volumes to tolerate machine/rack/row failures.
    • Can shift cold data to cloud storage and I think even can back itself up to cloud storage
    • Can provide S3, WebDAV, and FUSE storage natively
    • Very young project
    • Management story is not entirely figured out yet
    • I also lost data while testing this, though root cause there was unreliable hardware

    Tahoe LAFS

    • Very brief trial
    • Couldn’t wrap my head around it
    • Seems interesting
    • Seems mostly designed for storing things reliably on untrusted machines, so my use case was probably not ideal for it.

    MooseFS/LizardFS

    • Looked neat and had many of the features I want
    • Some of those features are only on (paid) MooseFS Pro or LizardFS (seemingly abandoned/unmaintained)

    Gluster

    • Disks can be put into many different volume configurations depending on your needs
      • Distributed (just choose a disk for each file, no redundancy)
      • Replicated (store every file on every disk, very redundant, wastes lots of space, only as much space as the smallest disk)
      • Distributed Replicated (Distributed across Replicated sets, add X disks as a Replicated set of disks, choose one of the replica sets and store the file on every disk in that set, is how you scale Replicated disks, each replica can only be as big as the smallest member disk, you must add X disks at a time)
      • Dispersed (store each file across every disk using X disks for parity, tolerates X disk failures, only as much space as the smallest disk * (number of disks - X), means you are only losing X disks worth of parity)
      • Distributed Dispersed (Distributed across Dispersed sets, add X disks as a Dispersed set of disks with Y parity, choose one of the disperse sets and store each file across its X disks using Y disks for parity, is how you scale Dispersed disks, each disperse only has as much space as the smallest disk * (X - Y), you must add X disks at a time)
    • Also gets used by enterprises
    • Anything but dispersed stores full files on a normal filesystem (vs Ceph using its own special filesystem, vs Seaweed that stores things in volume files) meaning in a worst case recovery scenario you can read the disks directly.
    • Very easy to configure
    • I am using it now, my testing of it went well
      • Jury is still out on what maintenance looks like

    Kubernetes-native

    Consider these if you’re using k8s. You can use any of the single-machine options (or most of the multi-machine options) and find a way to use them in k8s (natively for gluster and some others, or via NFS). I had a lot of luck just using NFS from my TrueNAS storage server in my k8s cluster.

    Rook

    • Uses Ceph under the hood
    • Used it very briefly and it seemed fine.
    • I have heard good things, but am skeptical given my previous experience with Ceph

    Longhorn

    • Project by the folks at Rancher/SUSE
    • Replicates the volume
    • Worked well enough when I was running k8s with some light workloads in it
    • Only seems to provide block storage, which I am much less interested in.

    OpenEBS

    • Never used it myself
    • Only seems to provide block storage, which I am much less interested in.