backpack.tf / stats.tf

It seems that, at some point, Valve has clamped requests for the GetPlayerSummaries Web API. This has become a bit of a thorn.

Traditionally, backpack.tf would call this API on every profile refresh on a per-user basis, caching the results for a few minutes at a time. It was fine at first, but as demand grew, we now need to perform these calls several times a second. But alas, there's a lot of things we should have done differently ten years ago. We have been, for a while, exceeding the request limit on this endpoint, which seems to have been lowered.

Recently, we have begun batching these requests into fewer calls that run in the background. The caveat of this, is that profiles will not update immediately. Thus, completely new, unseen profiles were exempt from this behaviour -- this is so you can at least fetch a username for accounts we've never seen before.

However, we've still run into exhaustion problems. Exacerbating the problem are web spiders crawling pages that nobody cares about; typically, profiles of users who never use the website or play the game. We kind of want these pages to be crawled anyway, but the prior solution would likely add many extra GetPlayerSummaries calls we don't want. Today, I removed that exception so these, too, get pooled into the backend job.

And still, we're sitting here with a slowly incrementing backlog of profiles we'd like to parse.

Luckily, this information isn't too important -- the data we fetch from GetPlayerSummaries includes usernames, avatars and profile privacy statii. That's mostly it. However, we are aware some new users are having difficulty accessing the site. There's not much we can do about it, at the moment.

Hopefully, this problem will go away on its own. We'll update if there's any progress.

-- backpack.tf staff