We've experienced a lot of issues with Alchemy/Infura providers because of block reorganisation.
Concept is described here
DB provides data consistency. When error occurred we reindex last 12 blocks according to Gavin Wood's words on this topic.
We've noticed network loss while using Digital Ocean. DB connection is not dropped when network is lost. In this case connected client waits for response from DB and will never receive it.
To prevent zombie connections and application stops when DB is not responding we added extreme timeouts (3 minutes) for all queries. We are sure there are no queries longer than 0.5 minutes so this timeout will only work when network was lost and client is 'stuck'.
Now logs are single lined JSONs and fully parsed and supported by ELK, which makes them accessible, sortable and usable from web interface.
We improved our custom Web3 solution to be less dependent and even more reusable.
Also improved other blocks of code in both Parser and Storage applications.
Not-optimised cache and high update rate caused network overuse and DB overload which led to extreme CPU/memory use indications. Now it's around 20 times faster and cheaper on resources.
Cache is block-oriented. Each query (with same frequency) get new data only if there is one. Once filled, cache won't reload data and will reuse and keep existing. Fork and block reorganisation protection also included so cache will be flushed when consistency check fails.