Scaling issues: The games table is huge
We store the game data for every game that’s played on the site. We’ve never faced a scaling issue as such but today we hit one, a big one. We’ve now over 2M+ unique automated games played on the site that the db size is now 298 GB and we started off with 300 GB thinking it would be more than enough. The index for the table was lost. I’m guessing this is because MySQL deletes them when the disk space is low? We’ll have to investigate deeply and maybe move the text/blob data to S3 and fetch from there.
I sincerely apologize for the slowness & the downtime of the site. We’re working super hard to fix it. As a quick solution to get back the site we’re expanding the storage to 2048 GB.
Once this is done (in the next couple of hours), we’ll be pulling an all-nighter to optimize the game tables further.
We really love you (all our users!) and we’ll be back with a super slick fast version. In fact, we’re revamping the interface as well which is going live next week. Again, very sorry about what happened. We never expected this and we’re on to fix it now.
I’d love to speak to each of you about your experience with the site and would really appreciate your feedback: vivek [at] hackerrank [dot] com. I promise to look through and reply to each one of them personally.