Rate limit issues

[bheemvennapureddy]

Hitting rate limits with Cron running every min

[bheemvennapureddy]

[svg153]

The CRON is not designed to be launched every minute. It is a reconciliation system that was put in place to consolidate the state in the safe-settings files and the real state of github, with the aim of not having a rate limit with so many requests to the API. Why do you have Chrome every minute? What are you trying to solve?
cc: @decyjphr

[bheemvennapureddy]

@svg153 i assume CRON is to gurantee that all the policies are correctly applied and some one trying to overwrite any of these policies like say adding labels manually should be overwritten by Safesettings no ? same goes to any status checks or branch protection rules cc: @decyjphr

i feel there is a bug in the system which is causing the rate limit issues - some stats below what i received from github

Top-url-CuroFinTechCorp.csv
Top_route-CuroFinTechCorp.csv

[martinm82]

What is the number of repositories within your organization? We have the same issue with rate limits but in our case the issue is that safe-settings just does not scale with large orgs (in our case >10k repos). We needed to change the cycle of the cron to be executed every 4-6 hours as otherwise we would hit immediately the rate limit.

Some changes, like branch protection rules are almost immediately changed back to the ones defined by the safe-settings configurations. This is achieved by webhook events (see index.js for the events that trigger an update of the repo settings).

To solve the rate limit issue, I started to change a bit the application and tried to split it into two, one for the cron part and the other for the webhook based events, so that it is always guaranteed that the webhook events will be properly executed by the safe-settings application by using its dedicated GitHub App with its own rate limit.

But there is a problem with consistency that makes it impossible without modifications in the code to have two safe-settings app running in parallel, as somehow data is cached and settings are overwritten either by the cron or webhook container.

[bheemvennapureddy]

We hardly have like 300 repos at most

[bheemvennapureddy]

@decyjphr any insights ?

[roryscarson]

Manual modifications should trigger a webhook event that will then trigger safe-settings to update settings of that type(ie Repository events will trigger repository setting sync, Branch protection will trigger branch protection, etc).

We have ~500 live, 1500 total repos hitting ratelimits trying to sync every 15 minutes. Changed the sync to ever 4 hours and it worked. FYI safe-settings still checks archived repos which could slow down your check a lot if you aren't deleting or moving old repos and are instead just archiving them.