Server Issue Fixed

General
1

Hello guys

We had a two incidents in the last 24 hours and i would like to explain what happen. The first incident was nothing that we could control, some services that we use in AWS had some issues and this caused downtime for about 1 hour.

The second incident is more interesting and something that we have fixed. As you might know we have a 160 requests/min limit on the API for each device. After a device sends us more requests we serve a “Too Many Requests” 429 Error. Someone most likely made a mistake and made a script that sends us a lot of requests, like thousands a minute, which our server handled correctly by giving a 429 Error. However the issue on our end is that we are also logging in a database the 429 Errors for debugging, and this saturated our connection to the database refusing new device connections.

4 Likes
2

Welcome to cloud services, lol! You are at the mercy of infrastructure you don’t support. And yea, the DOS you experienced for being in “debug” is fairly common for high activity accounts. We only turn it on when we’re debugging an issue because of the resource constraint. Nice work sorting it out! AWS should give you a heads up if its something they are causing. Do you have a friendly page that warns API users when the service is down? I’m fine with going to down detector to look at AWS, but admittedly, mid-session isn’t the time when that’s at the top of my list, lol. Nothing like sorting a tech issue to turn the hardware into software, lol…

1 Like