Maybe my issue here is that the whole “buildPythonPackage” ecosystem of nix is fundamentally at odds with the python ecosystem of late, and the rest of the nix ecosystem is better.

Python has been moving toward a more “define your exact dependencies” model lately (see pipenv, poetry, piptools, etc.), while nixpkgs expects that python package dependencies can be replaced with whatever version is in the nixpkgs repo (essentially the opposite view). Nix also assumes that tests will catch any problem caused by a version mismatch, which means defensive dependency pinning won’t work. (A dependency that is pinned to be under some assumed api breaking change is trying to prevent code from breaking before it does. If the dependency changes behavior in a way that changes results but still works, it will break the code in ways that tests don’t necessarily catch)

This is wrong more often than it is right. Partly because of the pain of the arm64 transition on Mac, and the Mac vs linux issue, and the fact that data science based python packages frequently have dependencies on C libraries (which just multiplies the dependency space), I have yet to have a python package build correctly the first time from nix.

But this is partly beside the point. If I have to learn a new way to install packages for every language, then I kinda have to be knowledgable in nix and the language to install a throwaway piece of software in whatever ecosystem I want to try out. I can probably make the buildPythonPackage thing work for python, because I know python… but if I have to do that for a nodejs project? or a ruby project? When I just want to use the output of the package? That is a ton of work to use a nix system… is that really the only way?

Just an fyi, llama is float 16 under the hood I believe. Stable diffusion is float32. Basically no machine learning model I’ve ever heard of is float64 based… the only people using float64 on gpus is physicists/applied math people on doe supercomputers.

(Weirdly enough, commodity hardware is now on the opposite end: they used to struggle with float64 when people were trying to put physics models on commodity hardware, now they struggle with float16/float8 support when people are trying to put language models on commodity hardware.)