bug: UpsertTemplateUsageStats creates lots of DB load

[spikecurtis]

In our dogfood deployment over 5 days in early June:

Avg Execution Time: 552ms
Num Times called: 1.8k
Avg Rows: 4

https://www.notion.so/coderhq/UpsertTemplateUsageStats-209d579be59280a68bbff80081a92b93?source=copy_link

[mafredri]

This is a complex query where 552ms seems like quite an acceptable execution time to me. We do run it once every 5 minutes, however, so we could try reducing that to perhaps once every 15 or 30 minutes? The effect of that is that the delay between showing updated template insights is delayed a bit.

Thoughts @spikecurtis?

[spikecurtis]

The total load is relatively small compared with other things --- the reason this is on the list is the long execution time. It likely causes spiky db load, which can affect tail latency elsewhere, and might cause us real problems at scale (e.g. timeouts).

Is there any way we can break down the work to be more incremental and spread out over the 5 minutes?