DefederateLemmyMl

  • Gen𝕏
  • Engineer ⚙
  • Techie 💻
  • Linux user 🐧
  • Ukraine supporter 🇺🇦
  • Pro science 💉
  • Dutch speaker
  • 2 Posts
  • 282 Comments
Joined 1 year ago
cake
Cake day: August 8th, 2023

help-circle







  • The thing is, you are absolutely free to use a /c,/d,/e mounting scheme, but you are not shackled to it like you are in Windows. Personally I like to organize my data in one big root (/) file system on my NVME drive and then /data for my bulk storage on HDD and /nas for my NAS shares. I never have any problems knowing where my data is.

    BTW, I notice all your complaints revolve around “OMG it’s different” and “OMG the user can choose to do things differently… so complicated”. That is kind of the point of Linux you know?

    At some point you just have to accept that it’s different and move on, or decide that it’s too complicated for you and use something else.

    BTW, I wonder why people never make this complaint about Apple devices? It also has a hierarchical file structure without drive letters, after all it is also a Unix variant.


  • I know the filesystem is simple to Linux users, but the semantic form of physical drives getting a letter always made more sense to me.

    That’s one of the things that semi-experienced Windows users need to wrap their head around, but I strongly disagree that drive letters are somehow inferior to a hierarchical file system structure. I mean, the A:, B:, C: … convention was originally just intended for the first IBM PC with 1 or 2 floppy drives. It was never intended to support complex storage configurations, whereas the hierarchical file system was designed for Unix systems that had to handle multiple magnetic drives from the start. It is a much more flexible system to organize your file storage.

    On Linux, as best I understand it, if I have three drives, two of them are at /dev/hdd0 and hdd1. But they’re not actually there.

    That’s because there is a difference between a block device and a mounted file system. Windows just obscures that difference from you with its archaic drive mapping system.

    All your block devices and partitions on your block devices will be in /dev with a meaningful name. You can list them with the lsblk command. If a partition contains a file system that Linux knows how to use, you can mount it anywhere you like.

    they’re accessed at /media/hdd0 after mounting them

    No that’s not “convention” at all. Some desktop environments may decide to mount undefined drives there, but there really is no convention, ultimately you mount it where you want it to be mounted.

    If you place an item in /home/documents/notporn, then who knows which drive it’s on because you don’t know what symlinks someone set up to make that folder.

    If your unsure, df /home/documents/notporn should tell you exactly what drive it’s on, but ultimately it’s up to you to know how you’ve organized your storage.

    BTW I’ve said this before, but Linux is probably harder for users who know Windows just well enough to be dangerous than it is for relative beginners, because there are so many concepts and things they take for granted that they have to unlearn.






  • DefederateLemmyMl@feddit.nltolinuxmemes@lemmy.worldNo Mercy
    link
    fedilink
    English
    arrow-up
    12
    ·
    edit-2
    5 months ago

    I know what it theoretically is for, I still think it’s a bad implementation.

    1. It often doesn’t tell you clearly what it is waiting for.
    2. It doesn’t allow you to checkout what’s going on with the process that isn’t responding, because logins are already disabled
    3. It doesn’t allow you to cancel the wait and terminate the process anyway. 9/10 when I get it, it has been because of something stupid like a stale NFS mount or a bug in a unit file.
    4. If it is actually something important, like your Redis example, it doesn’t allow you to cancel the shutdown, or to give it more time. Who’s to say that your Redis instance will be able to persist its state to disk within 90 seconds, or any arbitrary time?

    Finally, I think that well written applications should be resilient to being terminated unexpectedly. If, like in your Redis example, you put data in memory without it being backed by persistent storage, you should expect to lose it. After all, power outages and crashes do happen as well.


  • DefederateLemmyMl@feddit.nltolinuxmemes@lemmy.worldNo Mercy
    link
    fedilink
    English
    arrow-up
    11
    ·
    5 months ago

    That’s what systemd’s dreaded “a stop job is running” is

    The worst part of that is that you can’t quickly login to check what it is (so maybe you can prevent it in the future?), or kill it anyway because it’s likely to be something stupid and unimportant. And if it actually was important, well… it’s gonna be shot in the head in a minute anyway, and there’s nothing you can do to prevent it, so what’s the point of delaying?



  • You are misunderstanding.

    The file cache is never written out to the swapfile, because files are already on disk, like you say. The file cache is kept in memory and the kernel may decide it’s more advantageous to swap out unused anonymous memory pages to disk than flushing a file from the cache. You can use the vm.swappiness parameter to finetune this behavior to your liking btw. Lower values favor keeping more anonymous memory pages in memory, higher values favor file backed pages.

    To give an extreme example of where this is useful: I have a use case where I process a number of large video files (each 2GiB-10Gib in size). The job I’m doing involves doing several passes over the same file sequentially. You can bet your ass that caching them in memory speeds things up dramatically: the first pass, where it has to read the file on disk is at 200x speed (relative to the video’s duration), the second pass at 15000x speed.

    Even in less extreme circumstances it also helps by keeping frequently accessed files in your home directory in memory, for example your browser profile. Your browser and desktop environment would be much more sluggish if it had to reach out to disk every time for every file it touched.

    And you are free to disagree, but it’s not my opinion but the opinion of kernel developers of just about every operating system built the past 4 decades. So I’d say: take up the argument with them and see how far you get.


  • If he’s processing LLMs or really any non-trivial DB (read: any business DB)

    Actually… as a former DBA on large databases, you typically want to minimize swapping on a dedicated database system. Most database engines do a much better job at keeping useful data in memory than the Linux kernel’s file caching, which is agnostic about what your files contain. There are some exceptions, like elasticsearch which almost entirely relies on the Linux filesystem cache for buffering I/O.

    Anyway, database engines have query optimizers to determine the optimal path to resolve a query, but they rely on it that the buffers that they consider to be “in memory” are actually residing in physical memory, and not sitting in a swapfile somewhere.

    So typically, on a large database system the vendor recommendation will be to set vm.swappiness=0 to minimize memory pressure from filesystem caching, and to set the database buffers as high as the amount of memory you have in your system minus a small amount for the operating system.


  • I have 20GB in my current setup and it was never full. If anything gets swapped in this situation it means it needlessly slows me down.

    Not necessarily. Your memory also contains file backed pages (i.e. “file system cache”). These pages are typically not counted when determining “memory usage”, because they can always be discarded.

    It is often advantageous to keep frequently use files in cache in favor of unfrequently used memory pages.


  • I’ve never understood why GNU/Linux actually needs swap

    It doesn’t. It’s just good to have in most circumstances.

    Also, sidenote: “GNU” doesn’t apply here. Swapping is purely kernel business, no GNU involvement here.

    Okay, I created a 4G partition for it, having 32G of RAM. I never used all that RAM, but even so, stuff regularly ends up in swap. Why does the OS waste write cycles on my SSD if it doesn’t have to?

    Physical memory does not just contain program data, it also contains the filesystem cache, which is also important for performance and responsiveness. The idea is that some of the least recently used memory pages are sometimes evicted to swap in favor of more file caching.

    You can tweak this behavior by setting the vm.swappiness kernel parameter with sysctl. Basically higher values mean higher preference for keeping file backed pages in memory, lower values mean higher preference for keeping regular memory pages in memory.

    By default vm.swappiness = 60. If you have an abundance of memory, like a desktop system with 32G, it can be advantageous to lower the value of this parameter. If you set it to something low like 10 or 1, you will rarely see any of this paradoxical swap usage, but the system will still swap if absolutely necessary. I remember reading somewhere that it’s not a good idea to set it to 0, but I don’t remember the reason for that.

    Alternatively, there is no rule that says you can’t disable swap entirely. I’ve run a 32G desktop system without any swap for years. The downside is that if your 32G does run out, there will be no warning signs and the OOM killer will unceremoniously kill whatever is using the most memory.

    tl;dr just do this:

    sysctl vm.swappiness=10
    echo "vm.swappiness=10" > /etc/sysctl.d/99-swappiness.conf