A short tale about troubleshooting improbable bug – part deux

Last month I hoped that we have valid workaround for the bug described in A short tale about troubleshooting improbable bug. Unfortunately, it appeared that it is not working for all cases. As the fix for the bug was still missing, I tried to find something more reliable.

A short tale about troubleshooting improbable bug

A few months ago, after adding yet another DB into ProxySQL, we started to experience instance crash. The case was interesting and challenging due to following reasons:

• Happening only with new DB type we had just added to ProxySQL for the 1st time – having already all the other DB types processing billions of queries per day through ProxySQL we never had such problems.
• Hitting only after 2-3 hours of run time, where there is nothing obvious pointing to any specific processing happening around that point in time.

“Improbable” from the post title may be a bit of exaggeration, but as you will see what is needed to reproduce the problem, it is the most unlikely to happen bug I was dealing with so far 😁. Hence the idea to write down the summary of how the investigation was conducted – hopefully helping you with future troubleshooting sessions, as the risk that you will hit the same bug in your environment is fairly small (it was confirmed to happen at least in 8.0.28 and still existing in 8.0.35).