So I've tried all the options.
In the end, I recreated a clean install - i.e. formatted the SD card and wrote a clean IMAGE to that SD.
On the existing Raspberry Pi 4, it all crashed again. However, when I replaced this original one with the other one I have at home and it is in its original state without updated drivers to boot from USB. So there this SD card booted without any problems and everything is functional from the SD card.
Obviously the problem is somewhere in the drivers to boot from USB when it boots it somehow messes each other up, unfortunately I can't get back on the original RPI.
I can't think of anything more to do, I've reached the maximum I'm able to do.
So please very much include creating an image for SSD boot in the development, after all your knowledge of this issue is much greater.