FireFox freezing Repetier Server sometimes - webcam issue?

vmario89

Hi, it seems that since Firefox 74.0 Repetier Server freezes sometimes when accessing some page where webcam is embedded. The page does not freeze if the webcam is unavailable. It also does not freeze if i use that MS Edge instead.

At the moment i have no idea how to fix that. The inspector console says nothing and firefox won't show a freeze popup (the yellow var that some script slows down).

But instead the page crashes with the result of an endless round spinner or the controls are just not able to use but page is visible as usual/mouse can move but nothing clickable.

Did anyone get the same experience?

Repetier

One problem with webcams is that they block a connection and the number of parallel connections is limited. The other problem is that switching between pages can keep such a connection open when browser does not detect that it is not used any more. We try to force browser to close them actively to prevent this. Now I have opened firefox 74.0 and switched a lot between the tabs without any problems so far. How long do you need to switch between webcam, control and console to get that hang? If you open console before and switch to network you see the open webcam connections - they have no status code as the connection is not closed yet. When you find a page that opens it and does not close it, that one would be causing the problem. But so far I do not see that happening on my firefox.

vmario89

Hi, its hard to say how long the switch takes. sometimes maybe 30 seconds, sometimes longer.

would it be a fix to have some control service on raspberry pi which looks for dangling port connections? with lsof there are sometimes some port states "CLOSE_WAIT" from mjpg_streamer

Repetier

Problem is that browser might not close connection browser->server, it is not a problem server-mjpg_streamer.

What do you do to produce the firefox hang? Does it stop working without switching a page or when you switch from page xy to anything else?

vmario89

Hey, the problem is that i cannot reproduce if i want. it happens sometimes, sometimes not. One idea: It could be a problem of connection quality because at our location our network sucks so hard that we have a DSL 100.000 which delivers 28-50 kb/s download from 16:00 ~ 24:00 o'clock (bad Vodafone ...). Maybe the wait for stream connection establishing itself is the problem.

Repetier

So you use it remotely while I test it locally. The big question is what firefox does when a connection breaks. If it detects this and closes mjpg download you should have no problems except that the webcam freezes.

If you can, please use it with open network in debug and when it freezes again have a look on the open connections without a status (sort status ascending and they are at top). If you have around 6 which is the limit I think nowadays check the urls - if they all belong to cammjpg it is not closing them when you switch pages. Do you use home page with webcams and how much do you have there? Maybe it gets a problem if we have to many cams on one page and then it doe snot manage to close them correctly.

vmario89

Hi!

it happened again and i tried to find out something more. It was hard to load the network debugging because this window itself freezed too in the tab. After a while it was loaded (after maybe 1 or 2 minutes). I made some screenshots. As you can see (maybe) only one cam url is without status. I only use one webcam in general for the server where it occures.

i have 2 other sources where the cam can be loaded from. I put them in Instar Android Application (with some other cams in parallel) and in some grafana dashboard. But at the moment i am the only person who can access them. And i did not access them at the same time while loading Repetier Server.

Do you have some idea? regards, Mario

Repetier

Only one open webcam visible here, so no limit reached. Interesting is the raced message for the other files. It means that loading the file from server is faster then finding it in your firefox cache.
I see you hat 3 websocket connections, so you lost 2 times connection, but you said internet was not that stable so no problem. I get it quite often when I recompile server and no problems with that.

So now that we know it is not the webcam that blocks firefox there must be something else being wrong. If you still have it open can you open the latest connection to /socket and check messages. The last 20 or so are of interest. Something must block the browser - also I would expect the not responding message then. Anyhow it might be a unusual message also since it does not happen on chrome/edge it must be something that is handled differently in firefox and not directly wrong. And the fact that it depends on a webcam is strange - webcam does not require different messages. They are opened outside the websocket. And no error in console is also not indicating anything. Pausing javascript might be an idea, but it is compressed so it is hard to read what it does.

vmario89

Hi,

occured again. I had a look into htop and Repetier Server "gets hot" when webcam is freezing in browser window.

#

vmario89

had a another test in chrome and also figured out some strange wobbling of the image. i made some desktop video of this. its really psychedelic to look at.

https://cloud.leyghis.de/f/e47633dff1a44a8a8f00/?dl=1

Repetier

What is your sorting rule? Not cpu usage as it seems. Wonder what processes use in total over 300% - server added what I see is 88% plus maybe 30% for your python processes at the top. Already a lot but still around 200% cpu load missing. Maybe chromium on local display?
I also see the long time load is low compared to short time load so something has happened increasing the load. So much is sure.
When you hit F2->Display options->Show custom thread names you will see wat type of thread in server is taking the high load. Activate also "update thread name on every refresh " if you do not see the name. cammjpg is the name for active webcam stream. Each open webcam stream creates one such thread. My cause 4% load but I have low resolution 640x480 so might differ. Assuming these are your webcam streams - how high is the load there normally and what resolution/framerate do you use? Question is mainly is 35-50% normal load or particular high if it is webcam stream.

Does reloading firefox reduce load significantly?

vmario89

Hey,

it happened again. I believe it has to do something with accessing the host locally in the network (like http://192.168.1.100:3344). I believe that the cam never freezes when accessing Repetier Server by reverse proxy by hosting under regular public subdomain. It seems that it only freezes when opening Repetier Server in localnet.

Sorting the htop processes did not really help a lot because it switches too frequently.

The cam i am using is a C920 from Logitech. It's configured to deliver 1920x1080p at 30fps. So it's a high resolution. This works well usually. I think it's not a problem with mpjg_streamer because i can stream the cam from browser, or from embedded Grafana <img src> and also from Duet Web Control. None of these three software programs ever freezed when trying to display the camera.

In htop you can see apache2 getting high load. This happens not when accessing from external domain over reverse proxy instead. I inserted yellow dots for all the camera relevant stuff in the screenshot.

regards, Mario

Repetier

It is the first time I realise you are using apache. Our images always use nginx as reverse proxy. What device are you using?

I see server has 8.5% load with mjpg forwarding. Not much compared with apache having 35% at the same moment. One problem with proxy is when they start buffering data.  I had big delays in webcam streams in the past when bandwidth was too low. So now we dynamically drop images from the stream when that happens. Works great unless a part in between like apache would start buffering the data. From memory usage it seems not to do that - at least not in ram.
But you think it only happens without proxy. That would be just one stage less. Apache queries on port 3344 just as you do when you use it directly. That would mean if the http client is apache there is no problem and if the client is firefox a problem can happen.

Why is the apache load so high when you have the error accessing server locally? Is it normally also that high? On my pi 4 nginx has 6-7% cpu load when viewing a stream 640x480.

I also still do not make the connection to the website freezing - website freezes if the websocket does not send data and that is unrelated to webcam. Only if this causes a deadlock but then reloading webpage would not work.

Really hard to find a cause if it happens only to others so you can not debug it on your own. And the fact that it only happens in new firefox versions make me believe that they handle something different causing the problem. So you would need to debug the firefox side why it gets unresponsive. open networks, javascript and what it tries to do... not easy with production code that is optimized and minimized.

vmario89

Hi,

i am using Raspberry Pi 3 B with Raspbian Buster Lite (no Desktop) running apache2, Repetier Server, python3.7, most recent mjpg_streamer

The load of apache usually is much lower. It just gets so high when calling Repetier Server locally in FireFox. Still no idea but i have an eye on it.

by the way i also did reverse proxy with apache for the local port 3344 because i "ported" it to SSL-only using mkcert to another port. Then i installed the mkcert file into browser which allows me to have encrypted connection in localnet. But the problem with the camera already occured before this - so directly on port 3344. That means that normally apache should have no influence on this - but it does. It's a really strange issue.

I just tested Repetier Server monitor with remote address and local address. Both work well - no cam issue here; like in Google Chrome. Guess you use chromium core for Windows Desktop software so that's basically the same good-working behaviour.

Repetier

Monitor uses Electron which uses in deed chromium. The fact that it only happens to Firefox also shows that the client fetching data is relevant here. Over apache as proxy it is apache fetching it and not firefox so that then makes it not happen. So main question is what does firefox do here and why only so random. I think there must be some special condition when it happens that in combination with uncommon behaviour then results in the problem.

vmario89

Hey,
refactoring this issue after a series of new tests to the following short facts
- using: Repetier Server 0.92.1, mjpg_streamer, Raspbian Buster, latest apache2
- Google Chrome Repetier Server is running fine - no freeze (tested with localhost on 3344 and also remote forwarding with apache proxy)
- Repetier Server Monitor is running fine - no freeze (tested with localhost on 3344 and also remote forwarding with apache proxy)
- Firefox is running fine remotely (always), but freezes on localhost port 3344 (always)
- turning off apache does not change anything

- accessing the webcam by other tools like Grafana or mjpg_streamer on port 8080 there is no freeze

I think the issue has to be a problem mix of Repetier Server and/or Firefox.

I found a thread of a guy who have had the same problem but with Octoprint and Firefox: https://community.octoprint.org/t/webcam-causes-browser-to-freeze/1477

... so far this is the behaviour when running the cam with 1920x1080 pixels at 100% jpg quality and 30 fps.

Now: as soon as i pull down the FPS to 15 fps Repetier Server starts working in Firefox without freezing, but lagging. As soon as i set fps to 10 it the user interface stops laggings.

Its not a good fix to throttle fps because it makes less quality timelapse videos but its a helping information how to work with it until somebody found out the real solution. At the moment i cannot use the full power of the camera

Maybe somebody has some more ideas now. Maybe there's some setting in Firefox about threading or decoding.

maybe some of these settings?

Repetier

Ok these are good hints. Using https://www.cctvcalculator.net/en/calculations/bandwidth-calculator/ I got a bandwidth of 94.9 mBit/s so that would be clearly too much for wifi. With ethernet it should work if you have gigabit network to client. What makes me wonder is 15fps lag - how can then 30fps work. Or does it work until it freezes? How long does it take to freeze? Assuming you can handle the bandwidth because chrome has no problems the guess is that firefox would get problems with the data amount - but that does not explain why it can handle same amount over apache. Does apache maybe strip parts? What I also see on chrome are warnings that the jpg was not complete. I even had on a pi zero the problem that it constantly striped bottom lines for higher resolutions. Guess low power of pi is here a problem. 

Anyhow knowing resolution/fps is an issue I tested again. Now I get my firefox to freeze on first view if a webstream is contained that is so big. Nginx proxy did also not work. Using mjpg_streamer web surface directly also did not work. So in my case it does not work in any case. All I see is a FirefoxCP Web Content process taking around 100% CPU and nothing works. So looks like firefox is too slow to compress the stream and gets stuck somehow. Will test a bit further to see what happens if load is < 100%. My guess is that this is the requirement to not freeze.

Since you mentioned timelapse - the FPS has no influence on it at all. For timelapse we fetch the image independently from any mjpg streams.

Repetier

Ok after some more more test with pi webcam 1920x1080, quality 85%
- 10 FPS: Works good on firefox, cpu load 75-90%
- 15 FPS: Lagging stream, not usable cpu load 125%
- 30 FPS: Total freeze, cpu > 100%

So the truth is that high res mjpg is just causing too much cpu load on firefox breaking user experience. Chrome has 70% load for the 15FPS stream. And I guess they just skip frames if they can not decompress fast enough and make it healthy that way.

So I see only 2 solution:
1. Use chrome
2. Reduce FPS or resolution