- cross-posted to:
- fediverse@lemmy.ml
- cross-posted to:
- fediverse@lemmy.ml
I found this link aggregator that someone made for a personal project and they had an exciting idea for a sorting algorithm whose basic principle is the following:
- Upvotes show you more links from other people who have upvoted that content
- Downvotes show you fewer links from other people who have upvoted that content
I thought the idea was interesting and wondered if something similar could be implemented in the fediverse.
They currently don’t have plans of open-sourcing their work which is fine but I think it shouldn’t be too hard to try and replicate something similar here right?
They have the option to try this out in guest mode where you don’t have to sign in, but it seems to be giving me relevant content after upvoting only 3 times.
There is more information on their website if you guys are interested.
Edit: Changed title to something more informative.
Based on these criteria:
That sounds a lot like you would need to keep track of the vote weights of every user for every other user.
With 100 users, that would require tracking and updating 10000 values. That seems quite manageable!
There are about 39863 active Lemmy users, according to Fediverse.observer. That means keeping track of (up to) 1,589,058,769 weights (assuming you use f16 representations, that’s over 3GB of data). For every single upvote and downvote.
Would this be possible to implement? Yeah, for sure! Would this be practical? Not really, no! This is an O(n²) complex algorithm in terms of data storage, and that simply won’t do.
Imagine if Lemmy were to gain popularity and all of its inactive accounts came back. We’re now up to 2 million users. For busy servers (.ml, world) that means tracking 4 trillion variables for every single interaction (that’d be 8TB of data).
Did I say 2 million? Threads.net has 140 million users. Foursquare has 55 million. The “classic” fediverse has a mere 14 million users (active + inactive). That’s 4.3681×10¹⁶ weights to update for each upvote.
Now, perhaps there are ways to do this more efficiently. For example, users’ devices could track these numbers, so exposing the upvote data would allow the end users’ device to “only” track a couple million data points in their browser and on their devices, and then locally calculate the score of batches of a couple hundred posts collected based on some kind of heuristic. Sorting would take a while and drain your phone’s battery, but it wouldn’t kill the server. With “just” a couple hundred megabytes of data transfer every time the browser gets refreshed and running the device’s GPU/AI accelerator chip full blow for a while, you could use this algorithm.
The idea is enticing, but it doesn’t scale well.
You’d also turn Lemmy into the strongest echo chamber you could possibly create. I’m not sure if that’s what we want to do. If that’s your goal, you should consider moving to Facebook or Threads, maybe?
Storing it as a sparse graph should reduce the storage requirements drastically, since most edges wouldn’t exist.
Probably, we’d need to analyze the statistics for a good overview. However, just upvoting one super popular post once would cause a huge spike in graph growth across the board. Malicious users could also mass upvote to take down other servers, by generating random usernames and upvoting every post from as many accounts as possible.
Even with reduced storage requirements, the processing power required to keep this graph up to date would be quite significant. I don’t think it’s feasible for servers larger than the “private server for me and my friends” type that usually have some leftover CPU resources.
Not an option
As for the rest yeah those do seem like genuine obstacles. Partially think the reason I liked the algorithm is because it reminded me of the Web of Trust things like Scuttlebutt use to get relevant information to users but with a lower barrier to entry.
Also as I’ve said elsewhere it doesn’t have to be this exact thing but since this is a new platform we have the chance to make algorithms that work for us and are transparent so I wanted to share examples that I thought were worthwhile.
Edit:
PS. I don’t think that’s true. Big tech companies that have more advanced algorithms would probably be much better at creating echo chambers.
I agree with that. I’m glad Lemmy added the “scaled” algorithm to give posts from smaller instances as chance, and I think the algorithms will be tweaked further in the future. I think this particular example has too many downsides, but there’s no doubt there are better sorting algorithms out there, especially for platforms with a general lack of content like Lemmy seems to have.
All your proposed algorithm does is increase the likelihood of seeing things you like and decreasing the likelihood of things you dislike. You can rely on big tech companies will at least try to introduce some variety so they can serve you more lucrative ads.
One of the way Facebook and Youtube trap you is by generating engagement. The best way to do that, is to make you mad. A sprinkle of dissenting ideas in your echo chamber will have you foaming at the mouth at the “bad types”. The algorithm’s goal itself may not be good or ethical, but at least it detracts from the echo chamber.
Through the complete lack of an algorithm, I find Mastodon to be a much stronger echo chamber than its corporate alternatives. You don’t get to see things you’re not interested in (which is a good concept) but you also end up creating an experience tailored to your world view. I think user control should be at the forefront of this type of software, but we should avoid reinforcing this mechanism where we can.
Instead of comparing every single individual users votes with every other one, you create clusters using data science techniques and bucket all users into those clusters, which are calculated on a nightly or weekly basis. By controlling the cluster size you can keep the number of comparisons managable, and still achieve OP’s vision.