I was working on a pretty well known game, porting it to consoles.

On PS4 we started getting OOM crashes after you’ve played a few levels, because PS4 doesn’t have that much memory. I was mostly new on the project and didn’t know it very well, so I started profiling.

It turned out that all the levels are saved in a pretty descriptive JSON files. And all of them are in Unity’s Scriptable Objects, so even if you are not playing that level, they all get loaded into memory, since once something references a SO, it gets loaded immediately. It was 1.7Gb of JSON strings loaded into memory once the game started, that stays there for the whole gameplay.

I wrote a build script that compresses the JSON strings using gzip, and then uncompresses it when loading the actual level.

It reduced the memory of all the levels to 46Mb down from 1.7Gb, while also reduced the game load by around 5 seconds.

I don’t know if this counts, but making sure not a single query in my app results in a full table scan in MySQL made a huge difference.

I patched someone else’s program which was known for being slower when it’s used for a longer time. It was iterating over items in its window just to reach the last element, the more items the slower, it became snappy when i taught it to keep a pointer to the last element.

Not mind blowing, I know, but it was a popular program and this made life better for many people.

I’ve got so many more stories about bad optimizations. I guess I’ll pick one of those.

There was an infamous (and critical) internal application somewhere I used to work. It took in a ton of data, putting it in the database, and then running a ton of updates to populate various fields and states. It was something like,

• Put all data in x table with batch y.
• Update rows in batch y with condition a, set as type a. (just using letters as placeholders for real states)
• Update rows in batch y that haven’t been updated and have condition b, set as type b.
• Update rows in batch y that haven’t been updated and have condition c, set as type c.
• Update rows in batch y that have condition b and c and condition d, set as type d.
• (Repeat many, many times)

It was an unreadable mess. Trying to debug it was awful. Business rules encoded as a chain of sql updates are incredibly hard to reason about. Like, how did this row end up with that data??

Me and a coworker eventually inherited the mess. Once we deciphered exactly what the rules were and realized they weren’t actually that complicated, we changed the architecture to:

• Pull data row by row (instead of immediately into a database)
• Hydrate the data into a model
• Set up and work with the model based on the business rules we painstakingly reverse engineered (i.e. this row is type b because conditions x,y,z)
• Insert models to database in batches

I don’t remember the exact performance impact, but it wasn’t markedly faster or slower than the previous “fast” SQL-based approach. We found and fixed numerous bugs, and when new issues came up, issues could be fixed in hours rather than days/weeks.

A few words of caution: Don’t assume that building things with a certain tech or architecture will absolutely be “too slow”. Always favor building things in a way that can be understood. Jumping to the wrong tool “because it’s fast” is a terrible idea.

Edit: fixed formatting on Sync

More than ten years ago, I was a team-lead in a game development company and during another crunch I was very overloaded with tasks, including a code review. And of course, I missed some things. We optimized the game, which in general did not do 30 fps on the target machine with minimum requirements, and the target was at least 40 fps. During the month, we jointly optimized everything so that we were able to achieve almost stable 40 frames, but this was not enough and periodically there were drawdowns up to 15-20 fps. Everyone is in panic, there are no ideas, we’re optimized everything.

No, I understand that there is no optimization limit apriori, but I mean an adequate opts without rewriting everything to assembler specifically for all supported architectures.

So, one night I looked into the event loop initialization and found that the initialization happens twice. Twice. Two parallel event loops. That is, two parallel cycles of logic and state updates. That night, by deleting one line, I optimized the performance of the game by more than 100%. 🤦🏻‍♂️

I investigated and found out that this line of secondary initialization was left by junior “for debugging” and forgotten in the crunch. And I missed it on the review.

I’ve keep this story a secret all these years. And now I’m not revealing names. Otherwise, it can have a dramatic impact on the careers of many.

A database optimization I made was changing a tables id generation from a manual generation scheme (some other table had an entry with the next usable id, it was updated with every entry written) to a uuid generation scheme. The table stores data from a daily import, on a fresh import all previous data is deleted. On some systems, there are more than 10 000 000 entries to be imported on a daily basis, which took 8 hours. Now, with batched inserts and the mentioned improvement in the db scheme, it’s at about 20 minutes.

TLDR: Reducing the amount of queries sent is good, because although network is usually fast (ms), db requests are still slow compared to the speed of an application (clock cycles).

And yeah, there is an option to only import changes daily, but sadly that isn’t supported in every environment.

Don’t have an objective definition of “Best” but here are few over the top of my head.

1. Profiled backend’s startup and reduced startup time by ~20-30 times. Issue was we were traversing node_modules folder to scan for API documentation.
2. Profiled python using cProfiler & snakeviz and optimized dozens of regex calls by simple pre-filtering. Improved performance by ~20 times.
3. Profiled python using pprofile & qcalgrind (IIRC) to pinpoint a single heavy regex compilation and optimized it by pre-filtering. Improved performance 8-10 times.
4. Use nodejs streams to do load data from s3, parse as CSV, transform into different structure and save into s3 in a continuous stream. Didn’t measure the performance gains but should be several times atleast. 5… many more, some coming from embedded world and even more drastic & unexpected

I just recently updated a database call to return the last 7 days of data instead of all data from all time. It went from taking 30+seconds to around 2-4 seconds. Still a long way to go to make it “good” but at least it’s not timing out now.

One of my most productive days was throwing away 1000 lines of code.

– Ken Thompson

I had a pretty standard linear-list scan initially. Each time the program started, I’d check the list for some values. The list of course grew each time the program started. I maximized the list size to like 2MB or something (I forget), but it was in the millions and therefore MBs range. I figured it was too small for me to care about optimization.

I was somewhat correct, even when I simulated a full-sized list, the program booted faster than I could react, so I didn’t care.

Later, I wrote some test code that exhaustively tested startup conditions. Instead of just running the startup once, I was running it millions of times. Suddenly I cared about startup speed, so I replaced it with a Hash Table so that my test-code would finish within 10 minutes (instead of taking a projected 3 days to exhaustively test all startup conditions).

Honestly, I’m more impressed at the opposite. This is perhaps one of the few times I’ve actually taken the linear-list and optimized it into a hash table. Almost all other linear-lists I’ve used in the last 10 years of my professional coding life remain just that: a linear scan, with no one caring about performance. I’ve got linear-lists doing some crazy things, even with MBs of data, that no one has ever came back to me and said it needs optimization.

Do not underestimate the power of std::vector. Its probably faster than you expect, even with O(n^2) algorithms all over the place. std::map and std::unordered_map certainly have their uses, but there’s a lot of situations where the std::vector is far, far, far easier to think about, so its my preferred solution rather than preoptimizing to std::map ahead of time.

I remember getting 9x improvement on a set of Shell scripts at work, piping the results instead of writing and reading from files at each step.

Was working on a game where we drew a fake audio graph to a black texture. The code originally cleared this texture every frame by drawing each pixel black before drawing the audio wave again. I cached the pixels we drew the audio wave to and only changed those back to black saving a huge amount of time.

Noticed that Hibernate session (DB ORM session) was leaking to Jackson (JSON marshalling), potentially causing infinite n+1 problem. Changing a few lines of code to lazy loading and fixing the session leak reduced our daily data transfer from DB from 5.6Gb to 170Mb.

Not sure if this was the biggest optimisation, but definitely the dumbest issue.

Memorization of pure functions can make a WORLD of difference. It’s just not as easy as it should be.

Meme answer: Changed a manual wait from 20 seconds to 18 seconds. Boom, 10% performance improvement. I can only do it so often though.