No, only one side should use this or signals will at best give nonsense. I already wonder that it is no problem with the serial/usb converter connected to same pins. But ok, no double use on same pins please.
As marlin works with it I'd assume power is not the real problem. On the other side firmware should work on usb if you remove the esp3d. What does this do? Is it a display with wlan or just a wifi to serial board? In the latter case you could configure firmware to use a second port pair for this, so you can connect usb and wifi the same time (also only one should print).
