Mirroring a large number of repositories in Forgejo triggered many 429 errors from the source preventing updates.
I solved my problem by editing the database to spread them out over time
Forgejo setup#
I setup a basic Forgejo instance to quickly mirror a large number of repositories for a project that’s closing down.
In doing so the source started blocking (rightly) further updates with HTTP error 429.
Database adventures#
The Forgejo setup used here makes use of an sqlite database and can be accessed via sqlite command line: sqlite3 gitea.db
Sqlite uses SQL just like a bigger SQL server but with a few small changes that aren’t relevant here.
All mirror update info is kept inside the mirror table and the field that needs to be updated is interval
This field stores an update frequency in nanoseconds and defaults to 28800000000000 ( about 8 hours )
To update them all I ran the following bash one liner:
for I in $(sqlite3 gitea.db "select id from mirror"); do sqlite3 gitea.db "update mirror set interval='$(echo "(60*60*8+(60*${I}))*1000000000"|bc)' where id='${I}'"; done
This takes the ID of the entry in the mirror database and runs a quick calculation where it works out the number of seconds are in 8 hours (60*60*8 = 28800 ) and then adds this number to the result of 1 minute times by the ID of the entry ( 60*${I} ) and then times it by 1000000000 to get the interval value in nano seconds.
The final step is to update the mirror database with the new interval value.
Now rather than update every mirror repo every 8 hours things will be spread out a little and not overwhelm the source.