Repetier Server crashing - can you help?
Some time last night, for the millionth time, a glitch caused all of my printers to freeze. Hoping for some insight. Here are my logs from server.log from around the time something could have happened until now:
2019-09-09 14:13:03: LUA job rendering.render#{"id":28,"image":"/var/lib/Repetier-Server/printer/TEN/models/00000028_2XSTOCKS-9-6_l.png","layer":"/var/lib/Repetier-Server/printer/TEN/models/00000028_2XSTOCKS-9-6.layer","list":"models","slug":"TEN"}
2019-09-09 14:15:05: Reading configuration file ...
Input file: /var/lib/Repetier-Server/printer/TEN/models/00000028_2XSTOCKS-9-6.layer
Rendering shadows ...17.0778 seconds.
Rendering gcode ...89.2857 seconds.
Number of end triangles: 217308
Number of normal triangles: 19457658
Rendering background ...
Time for raytracing: 13.0749 seconds.
images stored.
Computation time: 121.598 seconds
2019-09-10 02:59:46: Reset printer 09 - Printer
2019-09-10 03:34:21: Print of 2XSTOCKS-9-6 on printer 07 - Printer finished. Send 1211996 lines. Printing time: 13:32:08
2019-09-10 05:10:43: Closing websocket for missing ping
2019-09-10 05:10:51: Job created: /var/lib/Repetier-Server/printer/07__Printer1/jobs/00000027_2XSTOCKS-9-6.u
2019-09-10 05:10:51: finish job /var/lib/Repetier-Server/printer/07__Printer1/jobs/00000027_2XSTOCKS-9-6.u
2019-09-10 05:10:51: start printjob 2XSTOCKS-9-6 on printer 07 - Printer
2019-09-10 05:10:58: Job created: /var/lib/Repetier-Server/printer/NINE/jobs/00000009_2XSTOCKS-9-6.u
2019-09-10 05:11:02: start printjob 2XSTOCKS-9-6 on printer 09 - Printer
2019-09-10 05:11:05: finish job /var/lib/Repetier-Server/printer/NINE/jobs/00000009_2XSTOCKS-9-6.u
2019-09-10 05:11:05: Error creating new job: std::exception
2019-09-10 05:11:05: LUA job rendering.render#{"id":9,"image":"/var/lib/Repetier-Server/printer/NINE/jobs/00000009_2XSTOCKS-9-6_l.png","layer":"/var/lib/Repetier-Server/printer/NINE/jobs/00000009_2XSTOCKS-9-6.layer","list":"jobs","slug":"NINE"}
2019-09-10 05:11:05: Reading configuration file ...
Input file /var/lib/Repetier-Server/printer/NINE/jobs/00000009_2XSTOCKS-9-6.layer not found!
2019-09-10 05:11:12: Job created: /var/lib/Repetier-Server/printer/TEN/jobs/00000006_2XSTOCKS-9-6.u
2019-09-10 05:11:14: finish job /var/lib/Repetier-Server/printer/TEN/jobs/00000006_2XSTOCKS-9-6.u
2019-09-10 05:11:14: start printjob 2XSTOCKS-9-6 on printer 10 - Printer
2019-09-10 05:37:01: Print of 2XSTOCKS-9-6 on printer 06 - Printer finished. Send 1211996 lines. Printing time: 15:35:14
2019-09-10 07:38:50: Reset printer 09 - Printer
2019-09-10 08:57:27: Job created: /var/lib/Repetier-Server/printer/06__Printer/jobs/00000025_2XCLAMPS-9-6.u
2019-09-10 08:57:34: finish job /var/lib/Repetier-Server/printer/06__Printer/jobs/00000025_2XCLAMPS-9-6.u
2019-09-10 08:57:34: start printjob 2XCLAMPS-9-6 on printer 06 - Printer
2019-09-10 09:01:35: Closing websocket for missing ping
2019-09-10 09:02:42: Closing websocket for missing ping
2019-09-10 10:47:23: error:WifiManager::scanMode:System exception
2019-09-10 10:47:23: error:WifiManager::scanNetworks:System exception
2019-09-10 10:47:24: error:WifiManager::scanConnections:System exception
2019-09-10 10:48:27: error:WifiManager::scanMode:System exception
2019-09-10 10:48:28: error:WifiManager::scanNetworks:System exception
2019-09-10 10:48:29: error:WifiManager::scanConnections:System exception
2019-09-10 10:49:40: error:WifiManager::scanMode:System exception
2019-09-10 10:49:50: error:WifiManager::scanNetworks:System exception
2019-09-10 10:50:00: error:WifiManager::scanConnections:System exception
2019-09-10 10:51:28: error:WifiManager::scanMode:System exception
2019-09-10 10:51:38: error:WifiManager::scanNetworks:System exception
2019-09-10 10:51:47: error:WifiManager::scanConnections:System exception
2019-09-10 10:53:30: error:WifiManager::scanMode:System exception
2019-09-10 10:53:40: error:WifiManager::scanNetworks:System exception
2019-09-10 10:53:51: error:WifiManager::scanConnections:System exception
2019-09-10 10:55:44: Port closed for Printer082
2019-09-10 10:55:52: Port closed for NINE
2019-09-10 10:56:05: Connection closed: 08 - Printer
2019-09-10 10:56:08: Connection closed: 09 - Printer
2019-09-10 10:56:11: error:WifiManager::scanMode:System exception
2019-09-10 10:56:24: error:WifiManager::scanNetworks:System exception
2019-09-10 10:56:36: error:WifiManager::scanConnections:System exception
2019-09-10 10:58:36: error:WifiManager::scanMode:System exception
2019-09-10 10:58:46: error:WifiManager::scanNetworks:System exception
2019-09-10 10:58:58: error:WifiManager::scanConnections:System exception
2019-09-10 11:00:53: error:WifiManager::scanMode:System exception
2019-09-10 11:01:04: error:WifiManager::scanNetworks:System exception
2019-09-10 11:01:14: error:WifiManager::scanConnections:System exception
2019-09-10 11:03:09: error:WifiManager::scanMode:System exception
2019-09-10 11:03:25: error:WifiManager::scanNetworks:System exception
2019-09-10 11:03:36: error:WifiManager::scanConnections:System exception
2019-09-10 11:05:44: error:WifiManager::scanMode:System exception
2019-09-10 11:06:04: error:WifiManager::scanNetworks:System exception
2019-09-10 11:06:20: error:WifiManager::scanConnections:System exception
2019-09-10 11:08:37: error:WifiManager::scanMode:System exception
2019-09-10 11:08:51: error:WifiManager::scanNetworks:System exception
2019-09-10 11:08:59: error:WifiManager::scanConnections:System exception
2019-09-10 11:10:29: error:WifiManager::scanMode:System exception
2019-09-10 11:10:37: error:WifiManager::scanNetworks:System exception
2019-09-10 11:10:47: error:WifiManager::scanConnections:System exception
2019-09-10 11:12:31: error:WifiManager::scanMode:System exception
2019-09-10 11:12:40: error:WifiManager::scanNetworks:System exception
2019-09-10 11:12:50: error:WifiManager::scanConnections:System exception
2019-09-10 11:14:24: error:WifiManager::scanMode:System exception
2019-09-10 11:14:34: error:WifiManager::scanNetworks:System exception
2019-09-10 11:14:44: error:WifiManager::scanConnections:System exception
2019-09-10 11:16:43: error:WifiManager::scanMode:System exception
2019-09-10 11:16:59: error:WifiManager::scanNetworks:System exception
2019-09-10 11:17:12: error:WifiManager::scanConnections:System exception
2019-09-10 11:20:09: error:WifiManager::scanMode:System exception
2019-09-10 11:20:26: error:WifiManager::scanNetworks:System exception
2019-09-10 11:20:41: error:WifiManager::scanConnections:System exception
2019-09-10 11:22:28: error:WifiManager::scanMode:System exception
2019-09-10 11:22:38: error:WifiManager::scanNetworks:System exception
2019-09-10 11:22:50: error:WifiManager::scanConnections:System exception
2019-09-10 11:24:57: error:WifiManager::scanMode:System exception
2019-09-10 11:25:13: error:WifiManager::scanNetworks:System exception
2019-09-10 11:25:30: error:WifiManager::scanConnections:System exception
2019-09-10 11:27:46: error:WifiManager::scanMode:System exception
2019-09-10 11:27:58: error:WifiManager::scanNetworks:System exception
2019-09-10 11:28:13: error:WifiManager::scanConnections:System exception
2019-09-10 11:30:37: error:WifiManager::scanMode:System exception
2019-09-10 11:30:52: error:WifiManager::scanNetworks:System exception
2019-09-10 11:31:00: error:WifiManager::scanConnections:System exception
2019-09-10 11:33:22: error:WifiManager::scanMode:System exception
2019-09-10 11:33:39: error:WifiManager::scanNetworks:System exception
2019-09-10 11:33:55: error:WifiManager::scanConnections:System exception
2019-09-10 11:36:47: error:WifiManager::scanMode:System exception
2019-09-10 11:37:11: error:WifiManager::scanNetworks:System exception
2019-09-10 11:37:30: error:WifiManager::scanConnections:System exception
2019-09-10 11:42:10: error:WifiManager::scanMode:System exception
2019-09-10 11:42:23: error:WifiManager::scanNetworks:System exception
2019-09-10 11:42:41: error:WifiManager::scanConnections:System exception
2019-09-10 11:45:13: error:WifiManager::scanMode:System exception
2019-09-10 11:45:18: error:WifiManager::scanNetworks:System exception
2019-09-10 11:45:30: error:WifiManager::scanConnections:System exception
2019-09-10 11:47:00: error:WifiManager::scanMode:System exception
2019-09-10 11:47:15: error:WifiManager::scanNetworks:System exception
2019-09-10 11:47:32: error:WifiManager::scanConnections:System exception
2019-09-10 11:49:37: error:WifiManager::scanMode:System exception
2019-09-10 11:50:05: error:WifiManager::scanNetworks:System exception
2019-09-10 11:50:33: error:WifiManager::scanConnections:System exception
2019-09-10 11:50:48: Start logging...
2019-09-10 11:50:49: Imported external command Shutdown Server
2019-09-10 11:50:49: Imported external command Reboot Server
2019-09-10 11:50:49: Webdirectory: /usr/local/Repetier-Server/www/
2019-09-10 11:50:49: Storage directory: /var/lib/Repetier-Server/
2019-09-10 11:50:49: Configuration file: /usr/local/Repetier-Server/etc/RepetierServer.xml
2019-09-10 11:50:49: Directory for temporary files: /tmp/
2019-09-10 11:50:49: Reading firmware data ...
2019-09-10 11:50:49: Starting Network ...
2019-09-10 11:50:49: Active features:4095
2019-09-10 11:50:49: Reading printer configurations ...
2019-09-10 11:50:49: Reading printer config /var/lib/Repetier-Server/configs/NINE.xml
2019-09-10 11:50:49: Starting printjob manager thread for NINE
2019-09-10 11:50:49: Reading printer config /var/lib/Repetier-Server/configs/07__Printer1.xml
2019-09-10 11:50:49: Starting printjob manager thread for 07__Printer1
2019-09-10 11:50:49: Reading printer config /var/lib/Repetier-Server/configs/Printer082.xml
2019-09-10 11:50:49: Starting printjob manager thread for Printer082
2019-09-10 11:50:49: Reading printer config /var/lib/Repetier-Server/configs/06__Printer.xml
2019-09-10 11:50:49: Starting printjob manager thread for 06__Printer
2019-09-10 11:50:49: Reading printer config /var/lib/Repetier-Server/configs/TEN.xml
2019-09-10 11:50:49: Starting printjob manager thread for TEN
2019-09-10 11:50:49: Starting printer threads ...
2019-09-10 11:50:49: Starting printer thread for NINE
2019-09-10 11:50:49: Starting printer thread for 07__Printer1
2019-09-10 11:50:49: Starting printer thread for Printer082
2019-09-10 11:50:49: Starting printer thread for TEN
2019-09-10 11:50:49: Starting printer thread for 06__Printer
2019-09-10 11:50:49: Starting work dispatcher subsystem ...
2019-09-10 11:50:49: Starting user database ...
2019-09-10 11:50:49: Importing projects ...
2019-09-10 11:50:49: Initializing LUA ...
2019-09-10 11:50:49: Starting wifi watcher ...
2019-09-10 11:50:49: Register LUA cloud services
2019-09-10 11:50:49: add G-Code-Renderer
2019-09-10 11:50:49: LUA initalization finished.
2019-09-10 11:50:49: Work dispatcher thread started.
2019-09-10 11:50:49: Work dispatcher thread started.
2019-09-10 11:50:49: Starting web server ...
2019-09-10 11:50:49: Webserver started.
2019-09-10 11:50:50: Connection started: 10 - Printer
2019-09-10 11:50:50: Connection started: 09 - Printer
2019-09-10 11:50:50: Reset printer 09 - Printer
2019-09-10 11:50:50: Reset printer 10 - Printer
2019-09-10 11:50:50: Connection started: 06 - Printer
2019-09-10 11:50:50: Reset printer 06 - Printer
2019-09-10 11:50:50: Connection started: 08 - Printer
2019-09-10 11:50:50: Reset printer 08 - Printer
2019-09-10 11:50:50: Connection started: 07 - Printer
2019-09-10 11:50:50: Reset printer 07 - Printer
2019-09-10 11:50:55: New SSID Not Connected
Comments
Does the interface to server freeze or just print stop. If print stops, does connection get green again and you can send new commands?
Please check /var/lib/syslog around the freeze time for usb related messages.
dmesg
will show you if undervoltage happens from time to time. Not all messages cause a disconnect, but is a hint that it could be an issue here. If I see it correctly you are running 5 printers from one server so power, signals, usb bandwidth all could play a role.
I unfortunately can't tell you what happened, as I was not watching the server as it happened. It was on a long print, 3 of 5 machines running. All stopped in place and reset during print. Connection re-established and I could send commands after the fault.
could be interpreted like a server crash - at least it was not shut down cleanly and would explain all printers stopping. It is just impossible to say why. If this is a frequent issue you might need to debug it so I can say more.
https://www.repetier-server.com/knowledgebase/debugging-crashes-hangs-on-linux/
describes how to debug it. You need a constant putty connection with it running in gdb to catch it. The full back trace when it crashes is what I would need to find it.
What version are you using? We fix constantly parts with each release as we find some points not working. Maybe tomorrow we release the next update to 0.92.2, so please upgrade as soon as it is out so debug report matches our sources.