Rate limits with credits

Grow anything beyond a handful of users and you’ll want rate limits. Sounds complex, right? It turns out it’s not.

A back-story. localhost.run is written in python, specifically asyncio. My experience tells me that moving hundreds of Mb/s should be easily achievable even without optimisations. Sure, SSH isn’t like TLS where I can offload the heavy lifting of the encryption to the kernel, and I can’t do clever tricks like splice() syscalls, but I should certainly be seeing 3 figure speeds, right?

So when one of my users mentioned they were getting under 1Mb/s I was sad face and I set out to solve this grave injustice. When I tried it myself in my controlled test rig I got only slightly more than them, and I was running it locally over loopback! It was certainly my code’s logic putting the brakes on.

I traced it to speed limits recently introduced to free plans to help counter phishing abuse, I’d rushed this logic in so this wasn’t a big surprise. My code was trying to figure out the average speed of a connection and slow things down based on a sliding window of a minute. It was overly complex and badly written, and when I took it out performance was over 100Mb/s just like I’d expected. I had to fix it.

Enter credits.

I confess this is not my idea, it’s in my head because I’ve read about this shape of rate limit before, I think I’ve seen it called “buckets”. It’s so easy to reason about tho that I didn’t even go looking for other peeps implementations, I just got on with it.

Each tunnel has a number of credits c. It earns more credits every second c_s, up to a maximum max_c. When data passes over the tunnel credits are added since the last time the calculation was run c = min(max_c, c + (now-last_calc_time)*c_s) and the length of the data is subtracted from the credits c -= len(data). If the credits are below zero the socket reads are paused for the time it will take to get back above zero c/c_s, which puts TCP back pressure on the connection to slow down the endpoints.

The levers this shape offers you to pull are as simple as the logic. Set c_s to 1mb and that’s exactly how fast your connection can go. Set max_c to 10mb and it’ll burst 10mb when needed.

With this change I hugely reduced the amount of code and the logic involved and it’s now capable of around 100Mb/s in my test rig. In fact I see no significant speed difference between a server without the speed limit code and one with it. As an added bonus the same logic also now limits the connection rate (replacing another poorly performing piece of code), and could easily be applied to API limits if ever I need them too. And it’s much, much simpler to code and to reason about.