How to recover user data

General Discussions
1

Hi,

Requirement/scenario - Microservice receive around 1000 user data records each time from the queue and will process data from to store in the respective databases. Once all the records processed then insert these 1000 records data into the respective database tables.
Issue - How to recover/protect user processing data if Pod crashed suddenly in between processing (say 800 records processed out of 1000) due to some reason. I don’t want my newly created Pod to start processing records from the beginning again.
Please let me the solution to this problem.

Thanks,
Srinivas

2

If this is not a “this has happened” scenario, maybe plan to run your message queue in kafka, so the messages are still there when rebooted container returns for load that didn’t finish process. Kafka doesn’t delete messages upon delivery like some brokers, but stores them for a predetermined amount of time.