Display often hangs

[RobertE]

Ah. Interesting one this morning. After about 40 mins I thought it was restarting the viewer service, but in fact now has a black screen (not the usual hang, as that usually shows one of the assets). Green light is not full on, just flicking occasionally. It’s now 10:53 and no assets have been checked since 10:20 . . . The memory used is down at 231, which usually shows a recent restart, but the Viewer log says it has been up for 1 Hour 17 mins. I guess the viewer service restart did not complete properly?
Sorry to keep adding problems!

Option Value
Load Average 0.18
Free Space 26G
Memory Total: 811 Used: 231 Free: 485 Shared: 12 Buff: 9 Available: 515
Uptime 0 days and 1.29 hours
Monitor Info state 0xa [HDMI CEA (4) RGB lim 16:9], 1280x720 @ 60.00Hz, progressive
Display Power On
Raspberry Model Model 3B+ Revision: 1.3 Ram: 1 GB Sony UK
Screenly Version master@f2cbd9f
MAC Address b8:27:eb:ca:55:4d

Latest Viewer Logs

● screenly-viewer.service - Screenly Viewer
Loaded: loaded (/etc/systemd/system/screenly-viewer.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-03-05 09:34:32 GMT; 1h 17min ago
Process: 614 ExecStartPost=/bin/rm -f /tmp/screenly_html/* (code=exited, status=0/SUCCESS)
Process: 611 ExecStartPost=/bin/rm -f /tmp/uzbl_* (code=exited, status=0/SUCCESS)
Process: 605 ExecStartPre=/usr/bin/xset s noblank (code=exited, status=0/SUCCESS)
Process: 601 ExecStartPre=/usr/bin/xset -dpms (code=exited, status=0/SUCCESS)
Process: 600 ExecStartPre=/usr/bin/xset s off (code=exited, status=0/SUCCESS)
Main PID: 610 (python)
Tasks: 7 (limit: 4915)
Memory: 26.3M (limit: 486.6M)
CGroup: /system.slice/screenly-viewer.service
└─610 /usr/bin/python /home/pi/screenly/viewer.py

Mar 05 10:16:31 raspberrypi python[610]: current url is https://docs.google.com/presentation/d/e/2PACX-1vQFoZ2DUxdeCjNl5d5UFi-9MGVCefYlUKBJg1tQjfTK5apw0y14yHsDyfdLvq5KCnukC810eFkXL-GJ/pub?start=true&loop=true&delayms=10000
Mar 05 10:16:31 raspberrypi python[610]: Sleeping for 60
Mar 05 10:17:32 raspberrypi python[610]: Generating asset-list…
Mar 05 10:17:32 raspberrypi python[610]: Showing asset BBC News (webpage)
Mar 05 10:17:37 raspberrypi python[610]: current url is https://www.bbc.co.uk/news
Mar 05 10:17:37 raspberrypi python[610]: Sleeping for 20
Mar 05 10:17:58 raspberrypi python[610]: Generating asset-list…
Mar 05 10:17:58 raspberrypi python[610]: Showing asset Screenly Clock Widget 2 (webpage)
Mar 05 10:18:08 raspberrypi python[610]: current url is https://clock.srly.io
Mar 05 10:18:08 raspberrypi python[610]: Sleeping for 10
Mar 05 10:18:19 raspberrypi python[610]: Generating asset-list…
Mar 05 10:18:20 raspberrypi python[610]: Showing asset Google Slides 2 (webpage)
Mar 05 10:18:30 raspberrypi python[610]: current url is https://docs.google.com/presentation/d/e/2PACX-1vQFoZ2DUxdeCjNl5d5UFi-9MGVCefYlUKBJg1tQjfTK5apw0y14yHsDyfdLvq5KCnukC810eFkXL-GJ/pub?start=true&loop=true&delayms=10000
Mar 05 10:18:30 raspberrypi python[610]: Sleeping for 60
Mar 05 10:19:30 raspberrypi python[610]: Generating asset-list…
Mar 05 10:19:31 raspberrypi python[610]: Showing asset BBC Weather (webpage)
Mar 05 10:19:41 raspberrypi python[610]: current url is https://www.bbc.co.uk/weather/2636935
Mar 05 10:19:41 raspberrypi python[610]: Sleeping for 20
Mar 05 10:20:01 raspberrypi python[610]: Generating asset-list…
Mar 05 10:20:02 raspberrypi python[610]: Showing asset Screenly Clock Widget (webpage)

[ealmonte32]

@RobertE

Thanks. If I’m right that the green light full on indicates processor overload, then I don’t think tweaking will help. Most of the time the green light is just flicking on every few seconds (indicating it is not stressed at all?) and then something happens to make it full on. My guess is a runaway process has gone ‘wrong’. In windows I would look in task manager to see which process was using 100 CPU, but not familiar enough with Pi and Linux to try to do the same. I’m beginning to think that my best bet is to repower the Pi every 40 minutes. I have it on an electronic clock anyway (with 10 possible on/off time per day), so it would be easy to do. Do you not get yours hanging with green light on?? Have you tried bbc.co.uk/news and bbc.co.uk/weather?

Yes the green light is processor activity, this is normal though when the viewer is active and running, it spawns different processes, but regardless we have to see why yours doesn’t restart and mine does (has been during my tests (development branch).
I have tested by putting more assets then you including bbc news (not the weather one) , but if you see this page https://github.com/Screenly/screenly-ose/pull/1304#issuecomment-589461549
You can see my screenshots of assets and all the ones I tested with, and the Pi does not freeze anymore, it just restarts the process when memory is reached, and if it does at some point just stay black, it still is for now much better than before without the memory limits because it would hang and freeze because the oom-killer was for some reason not fully killing enough processes to allow me to SSH into the pi and troubleshoot it, and yes my green light would be lit in a forever loop because the process that made it that way was not killed…

Yes you could for now make a sudo crontab -e job and tell it to restart the viewer every 40 mins, I had that as the workaround before I made the memory limits.

open the job scheduler as sudo:
sudo crontab -e

make the job by adding this line all the way at the bottom:
40 * * * * sudo systemctl restart screenly-viewer.service

save the file: ctrl+x , y, enter

---

Just to make sure we are both running the same Pi and screenly versions, you have Pi 3 and screenly development/master branch? That way, I can test and see what is the difference between the installation i had that would restart the viewer along with all processes vs your setup…

[ealmonte32]

@RobertE
P.S. forgot to say, since it’s always good to have multiple people test something like this, when you have some time, can you change your KillMode from control-group to mixed, as stated here just to see what effect that kill method has when it deals with multiple child processes while python being the main one:
https://www.freedesktop.org/software/systemd/man/systemd.kill.html+&cd=2&hl=en&ct=clnk&gl=us

KillMode=mixed

and add this underneath that too:

SendSIGHUP=yes

Thanks… let me know how it goes (remember try this before creating the cron job just to see if this solves it) I want to probably leave the SendSIGHUP=yes and then go back to KillMode=control-group (i’ll test with control-group while you test with mixed mode), but definitely adding the SendSIGHUP=yes is something I want to do since it is the closest thing we have as a restart process signal.

[ealmonte32]

I forgot to set CPU limits on the service… I wrote this self reminder on a github issue after noticing that the uzbl-core process would consume 100% of cpu for extended periods of time without being killed because the memorylimit would not reach the threshold, thus at least the self healing of the Pi (not freezing) is working with memory limits, but now we need to add the CPU limits as well.
Working on the right setting and hopefully I can figure out what the proper limits should be for this to work.

Ok, this is what my current screenly-viewer.service looks like, if you could make your settings look like mine and test that would help since it seems to work for me…

updated screenly viewer service that was more stable (with celery service memory limits disabled)

[RobertE]

Thanks again ealmonte32. This is my Pi and version.
Raspberry Model|Model 3B+ Revision: 1.3 Ram: 1 GB Sony UK
Screenly Version|master@f2cbd9f
It’s a 3B+ if that makes any difference. I’ve made the CPU quota changes, and the SendSIGHUP=yes. I see you have allocated CPUQuota=100%??? Won’t this allow Viewer Service to consume ALL the CPU? Anyway, you’re the expert here, not me. Not made other chages yet (one thing at a time) so not yet changed from KillMode=control-group. I’ll let you know whether the CPU changes help.

[ealmonte32]

Yes that version is fine, I have multiple, 3 and 3+ so I just wanted to confirm, they are both capable of same raspbian lite version and have the same amount of RAM so not really a factor here.
I have tested with the same 3+ as you, with those values on the screenshot I attached, and I restarted of course after making those changes…

This part is a bit complicated and confusing (maybe for some?) and I am still researching the proper setting, but basically as far as I know (if anyone reading this wants to chime in and correct me) the raspberry pi 3+ and others have 4 CPUs, but some programs arent compiled to use multi-threading, like some python programs, so setting CPUQuota 100% is actually 25% of the “CPU” because it has 4 cores, so the quota is being set to the entire viewer service, and if you were looking at the processes as your Pi is running (use top or htop command) you would see that when the browser or any loading begins, the processor is usually ramped up to over 100%, so what I wanted to accomplish is to not let it sit at 100% for longer than the default, which can also cause the system to not function properly, and while the CPU could be at 100% or higher, the memory could be below the limit because the correlation between memory usage and CPU usage as you know is not necessarily the same, you could load a website that eats up 500MB of memory, but once everything is done loading, you have the CPU back at normal since it has nothing else to process in that site, while the memory is still sitting at 500MB… this is why we want both limits, the CPU and memory because we want a sort of self-healing viewer that does not cause the Pi to become unusable/hang/freeze/etc…
To have a real fix, I am going to do some more research into the CPU limits because the Pi can also freeze by not reaching memory limits but by having a process stuck at 100%+ and so for example right now I have my screenly properly restarting with all those assets, but it sometimes “freezes” on a page because the CPU stays in R state (running), so I need to find the proper CPU quota to prevent this from staying running by limiting the time the process can stay in this “R” state.

So for most of screenly-ose users, my current limits should suffice for the number of assets and types of websites they use, especially considering that if these limits weren’t in place, you would need to physically power cycle the Pi unless you had some cron job that would restart it daily but would still need to wait for that to happen, so this “stopgap” limit method is my current way of at least keeping the pi running screenly-ose from completely becoming unusable.

---

Self-notes:
Ongoing tests… trying to find the right mix of limits between screenly-celery.service and screenly-viewer.service, and how to handle the processes when they affect the system resources… increasing memory limit to 65% did not help, made Pi freeze due to memory exhaustion while mixed killmode did not execute pkill for viewer, set back to 60%… removed limit on celery since logs showed continuous oom-killer for celery but not for uzbl except once (for documentation purposes, screenshot attached)

[ealmonte32]

Ok, so with the current setup of removing limits from celery service and the viewer settings below, I am able to put all those assets shown on screenshot and the service is restarted as soon as the pi reaches the memory limit… and as you have found out, making the asset time more than 1 minute allows the pi more time to load the website, so i suggest you make your assets at least 1 minute or 2 each…

I even added these demo dashboards for good measure since they consume plenty of cpu and memory, and allowing it 2 minutes to load, and no black screen yet (its been running for 2 hours and at every memory limit reached it did its job)…

Sample dashboard ( https://public.datapine.com/#board/BgneeI1yWQMcSXIy4Se53P )

[RobertE]

Hi, I’ve got a bit sidetracked, as I’ve hit a more serious problem . . . My Google Presentation which has been fine for months now isn’t. See new topic Google Presentation problems . . . don’t know if you might have ideas about that? Thanks

[ealmonte32]

@RobertE
I’ve actually got good news for both:

1. For this issue with screenly-ose after a certain amount of time being stuck in one page, this causes the uzbl-core to get stuck in a Running state, after so much trial and error and searching for best way to alleviate this, the only thing I’ve been able to do to resolve it once and for all is to limit the time that each process spends in the CPU with LimitCPU.

This is my final working viewer service as a workaround while on the master/dev branch which uses uzbl which is most likely the primary cause of this…

---

2. For the google slide issue is simple: the problem is the way the viewer parses the URL to the browser, you simply need to change the URL from the long google slides one into a bitly one, so that the browser parses normal like this https://bit.ly/336A2Bi - instead of the long google one with special encoding characters, thus your loop and time delay will be properly executed in the browser… this is your google slides presentation URL in bitly form: https://bit.ly/336A2Bi , give it a try, you can do that for all URLs that have certain characters in the address…

[RobertE]

Wonderful. My display has now been up for over 3 hours. Better than it’s done for months, and looks like 3+ hours = ‘for ever’. Thanks so much for all your hard work. I can now take my Pi back to the Hall where the display is (I’ve had it at home for all this testing).
Might it be worth trying again with my Pi Zero W? I have several of these, and only one 3B+. Or might that be tempting providence? (I’m sure I read when I started that Screenly OSE should run on Pi Zero?
I shan’t complain if it doesn’t work, though - I have a working solution.
Do you think Screenly OSE will continue to use uzbl-core? It does seem to cause problems?

[RobertE]

Hi, Just for interest I plugged in my old Pi Zero, with Production version production@6c2c2fd and it copes with the Google Slides URL without problem?? I plan to change to Master/Development and the changes you’ve done, and see what happens.

[ealmonte32]

@RobertE

Might it be worth trying again with my Pi Zero W? I have several of these, and only one 3B+. Or might that be tempting providence? (I’m sure I read when I started that Screenly OSE should run on Pi Zero?
I shan’t complain if it doesn’t work, though - I have a working solution.
Do you think Screenly OSE will continue to use uzbl-core? It does seem to cause problems?

Yes screenly-ose works on the Pi Zero W as I have one and tested, but yes you need to definitely tweak it with things like setting gpu_mem to 192 (as you did with your Pi 3B).
The programmer who works on screenly-ose has started the move to a different browser because yes, uzbl is one of the main problems, but that is the experimental branch, long way to go before getting an experimental to production use. After all, this is all open source so we need volunteers who can help with the code.

---

@RobertE

Hi, Just for interest I plugged in my old Pi Zero, with Production version production@6c2c2fd and it copes with the Google Slides URL without problem?? I plan to change to Master/Development and the changes you’ve done, and see what happens.

Production and Master branch use the same browser yes, but the encoding of URL and parsing it (passing the URL onto the browser in a way it understands the characters and symbols) is done at the viewer.py (the main python program running the show) and imported modules and utilities all combined, and so rusko the programmer who works on that might have changed some things around in some part that takes the URL and parses it to the browser in order to accommodate something where maybe the production version did not do properly but the master version did, and maybe hasn’t had time to go into that part of the code to fix, or maybe it is a imported module issue, the reason why might not be something he is going to spend time with is because if the experimental browser will fix a lot of issues, he might as well not waste time with uzbl version and instead move onto working on the experimental branch instead… get it?

[RobertE]

Agree totally. I’m happy to use OpenSource software, and wish I could help with coding . . . but my days of working with Assembler Code in the 1970s (yes really!) are long behind me. I’ve had one or two dips into Python, but . . . I love problem-solving, and I hope helping with problem solvers! Good luck to you and Rusko in developing a more stable product.
I’ve just done all the tweaks on my Pi Zero, and set it off overnight (you probably guess I’m in UK?) to see how it gets on.

[luisscreenly]

I tried this and afterwards, the version didn’t change, still had production@6c2c2fd and there were complaints about version of python .

After, I tried /usr/local/sbin/upgrade_screenly.sh and that gave bunch error near the end during database migration…

My end goal is to get the changes in as suggested by ealmonte32 to get a stable system. I made the changes to the production OSE as ealmonte32 suggested but still hangs , so now I’m trying to install the latest development version.

Any idea what I could try next?

TASK [screenly : Run database migration] *******************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: [“python”, “/home/pi/screenly/bin/migrate.py”], “delta”: “0:00:00.148256”, “end”: “2020-04-08 10:09:21.854662”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2020-04-08 10:09:21.706406”, “stderr”: “Traceback (most recent call last):\n File “/home/pi/screenly/bin/migrate.py”, line 163, in \n migrate_drop_filename()\n File “/home/pi/screenly/bin/migrate.py”, line 153, in migrate_drop_filename\n with open_db_get_cursor() as (cursor, _):\n File “/usr/lib/python2.7/contextlib.py”, line 17, in enter\n return self.gen.next()\n File “/home/pi/screenly/bin/migrate.py”, line 53, in open_db_get_cursor\n with sqlite3.connect(database, detect_types=sqlite3.PARSE_DECLTYPES) as conn:\nsqlite3.OperationalError: unable to open database file”, “stderr_lines”: [“Traceback (most recent call last):”, " File “/home/pi/screenly/bin/migrate.py”, line 163, in “, " migrate_drop_filename()”, " File “/home/pi/screenly/bin/migrate.py”, line 153, in migrate_drop_filename", " with open_db_get_cursor() as (cursor, _):", " File “/usr/lib/python2.7/contextlib.py”, line 17, in enter", " return self.gen.next()", " File “/home/pi/screenly/bin/migrate.py”, line 53, in open_db_get_cursor", " with sqlite3.connect(database, detect_types=sqlite3.PARSE_DECLTYPES) as conn:", “sqlite3.OperationalError: unable to open database file”], “stdout”: “”, “stdout_lines”: []}

PLAY RECAP *************************************************************************************************************************************************************************
localhost : ok=36 changed=14 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

[ealmonte32]

just curious, instead of running the upgrade, can you simply run the install script again?

the upgrade script is the same as the master install.sh

[luisscreenly]

I deleted a couple of posts because of errors during the upgrade caused by running the upgrade as root. After upgrading as pi user upgrade completed Ok, so I’ll monitor now to see if it hangs on certain assets.

side note: system info … Unable to run tvservice. Unable to determine display power.

manually it works:
pi@media-x:~ $ tvservice -l
1 attached device(s), display ID’s are :
Display Number 2, type HDMI 0

[luisscreenly]

I think moving away from uzbl browser has really helped, so far no hanging on an asset that used to hang.
Thank You.

[ealmonte32]

@luisscreenly

If you happen to need to use the development/master version/branch but the only problem you run into is the uzbl browser causing too much memory consumption (which is the biggest issue) and making the pi unstable/unusable/hang, I can suggest you simply edit the python viewer file to limit the resources thus two lines added to the code will make it that any time the process viewer.py + uzbl-core consumes more than what you set, the process gets closed down and the browser starts again with fresh memory after restarting the browser.

Until qt browser becomes the default browser in the master branch, this is one of the easiest stop-gap methods you can use. Other methods i worked on like MemoryLimit in the service are too much compared to this simple way.

---

(see the attached screenshot for how i have mine running)

if you can SSH or ctrl+alt+f1 into the console:

edit the viewer.py file with nano:
nano ./screenly/viewer.py

1. on the top, just add:
   import resource

2. on the area before the author, add:
   resource.setrlimit(resource.RLIMIT_AS, (600000000,700000000))

these numbers are in bytes (soft limit, hard limit), for example it may seem like 600 MB soft limit and 700MB hard limit are a lot, but you have to remember in linux theres virtual and resident, and if you were to use the command top while testing this, you would see that your uzbl-core process terminates when the VIRT memory reaches that you have set, and the VIRT size will almost always be larger than RES size.

Give it a try, let me know how it goes, you can play around with the memory but since my pi 3 has 1GB and I use 192MB for GPU, it only gives like 800 of usable, and other services running take up memory as well, so you should not limit that high, usually the soft limit gets hit first and that kills the browser for 10 seconds or so, and this at least keeps your pi alive and reachable.
I also run different tests with the Pi and Screenly so for example right now this pi is running DietPi which is a slimmed down version of Raspbian Lite, so if you are running the regular raspbian lite or screenly-ose image from github, your memory usage might differ from mine, but it’s all closely similar and you should be able to play around with the 600 MB and 700 MB in order to find your sweet spot.

[RobertE]

Hi guys, great to hear that things are still moving. During the virus crisis, I’m afraid my system is ‘on hold’ so I can’t help with testing at the moment. But I’ll keep watching what goes on here, and hope to join you more actively - in a while.

[ealmonte32]

Hey @RobertE
I tested on multiple Pi’s and can confirm that this simpler method works. The person that adds those 2 lines will not have to worry about their Pi freezing/hanging/unresponsive due to memory consumption of viewer service.
If you add many many memory consuming web assets, you will just reach the limit sooner and the browser/service will restart itself, but no longer have to worry about coming to a display that has crashed =]