Your Database Has Rollback - But Does It Have Rollback Netcode?
Bulk ingestion of data into a database can be a massively time-consuming operation. It can also be time-sensitive, especially if the data being ingested are coming from a real-time feed and being used for modelling. Delays in the data stream caused by changes in network latency or other disruptions can propagate onwards and cause other delays.
This is a known - and solved - problem in other fields. Fighting games, in particular, have to deal with the problem of receiving input from a remote user and applying it to the current local game state. While some merely apply a constant delay to local input in order to buy time to sync it up with remote input, this is rarely a satisfactory arrangement. A much more well-liked solution is “rollback netcode”.
In rollback netcode, the local system assumes that any delayed input will be the same as the previous input, and applies it as if that were the case. When the delayed input arrives, if that assumption was incorrect, the local system rewinds the game state, applies the newly-acquired remote inputs along with the corresponding local inputs, and displays the new game state. While this can lead to small disruptions as the game state jumps from one position to another, overall it provides a much smoother and more satisfying user experience.
In this paper, we propose applying this principle to bulk ingests. When input is delayed, the database simply assumes that the next record will be the same as the previous one and applies it, allowing whatever is processing the data to continue. If that assumption is later found incorrect, the transaction that added it can be rolled back, and it can be replaced with the correct data. Acknowledging the correction and re-processing it is left to the application using the data and is beyond the scope of this paper.