Display often hangs

[ealmonte32]

@RobertE
P.S. forgot to say, since it’s always good to have multiple people test something like this, when you have some time, can you change your KillMode from control-group to mixed, as stated here just to see what effect that kill method has when it deals with multiple child processes while python being the main one:
https://www.freedesktop.org/software/systemd/man/systemd.kill.html+&cd=2&hl=en&ct=clnk&gl=us

KillMode=mixed

and add this underneath that too:

SendSIGHUP=yes

Thanks… let me know how it goes (remember try this before creating the cron job just to see if this solves it) I want to probably leave the SendSIGHUP=yes and then go back to KillMode=control-group (i’ll test with control-group while you test with mixed mode), but definitely adding the SendSIGHUP=yes is something I want to do since it is the closest thing we have as a restart process signal.

[ealmonte32]

I forgot to set CPU limits on the service… I wrote this self reminder on a github issue after noticing that the uzbl-core process would consume 100% of cpu for extended periods of time without being killed because the memorylimit would not reach the threshold, thus at least the self healing of the Pi (not freezing) is working with memory limits, but now we need to add the CPU limits as well.
Working on the right setting and hopefully I can figure out what the proper limits should be for this to work.

Ok, this is what my current screenly-viewer.service looks like, if you could make your settings look like mine and test that would help since it seems to work for me…

updated screenly viewer service that was more stable (with celery service memory limits disabled)

[RobertE]

Thanks again ealmonte32. This is my Pi and version.
Raspberry Model|Model 3B+ Revision: 1.3 Ram: 1 GB Sony UK
Screenly Version|master@f2cbd9f
It’s a 3B+ if that makes any difference. I’ve made the CPU quota changes, and the SendSIGHUP=yes. I see you have allocated CPUQuota=100%??? Won’t this allow Viewer Service to consume ALL the CPU? Anyway, you’re the expert here, not me. Not made other chages yet (one thing at a time) so not yet changed from KillMode=control-group. I’ll let you know whether the CPU changes help.

[ealmonte32]

Yes that version is fine, I have multiple, 3 and 3+ so I just wanted to confirm, they are both capable of same raspbian lite version and have the same amount of RAM so not really a factor here.
I have tested with the same 3+ as you, with those values on the screenshot I attached, and I restarted of course after making those changes…

This part is a bit complicated and confusing (maybe for some?) and I am still researching the proper setting, but basically as far as I know (if anyone reading this wants to chime in and correct me) the raspberry pi 3+ and others have 4 CPUs, but some programs arent compiled to use multi-threading, like some python programs, so setting CPUQuota 100% is actually 25% of the “CPU” because it has 4 cores, so the quota is being set to the entire viewer service, and if you were looking at the processes as your Pi is running (use top or htop command) you would see that when the browser or any loading begins, the processor is usually ramped up to over 100%, so what I wanted to accomplish is to not let it sit at 100% for longer than the default, which can also cause the system to not function properly, and while the CPU could be at 100% or higher, the memory could be below the limit because the correlation between memory usage and CPU usage as you know is not necessarily the same, you could load a website that eats up 500MB of memory, but once everything is done loading, you have the CPU back at normal since it has nothing else to process in that site, while the memory is still sitting at 500MB… this is why we want both limits, the CPU and memory because we want a sort of self-healing viewer that does not cause the Pi to become unusable/hang/freeze/etc…
To have a real fix, I am going to do some more research into the CPU limits because the Pi can also freeze by not reaching memory limits but by having a process stuck at 100%+ and so for example right now I have my screenly properly restarting with all those assets, but it sometimes “freezes” on a page because the CPU stays in R state (running), so I need to find the proper CPU quota to prevent this from staying running by limiting the time the process can stay in this “R” state.

So for most of screenly-ose users, my current limits should suffice for the number of assets and types of websites they use, especially considering that if these limits weren’t in place, you would need to physically power cycle the Pi unless you had some cron job that would restart it daily but would still need to wait for that to happen, so this “stopgap” limit method is my current way of at least keeping the pi running screenly-ose from completely becoming unusable.

---

Self-notes:
Ongoing tests… trying to find the right mix of limits between screenly-celery.service and screenly-viewer.service, and how to handle the processes when they affect the system resources… increasing memory limit to 65% did not help, made Pi freeze due to memory exhaustion while mixed killmode did not execute pkill for viewer, set back to 60%… removed limit on celery since logs showed continuous oom-killer for celery but not for uzbl except once (for documentation purposes, screenshot attached)

[ealmonte32]

Ok, so with the current setup of removing limits from celery service and the viewer settings below, I am able to put all those assets shown on screenshot and the service is restarted as soon as the pi reaches the memory limit… and as you have found out, making the asset time more than 1 minute allows the pi more time to load the website, so i suggest you make your assets at least 1 minute or 2 each…

I even added these demo dashboards for good measure since they consume plenty of cpu and memory, and allowing it 2 minutes to load, and no black screen yet (its been running for 2 hours and at every memory limit reached it did its job)…

Sample dashboard ( https://public.datapine.com/#board/BgneeI1yWQMcSXIy4Se53P )

[RobertE]

Hi, I’ve got a bit sidetracked, as I’ve hit a more serious problem . . . My Google Presentation which has been fine for months now isn’t. See new topic Google Presentation problems . . . don’t know if you might have ideas about that? Thanks

[ealmonte32]

@RobertE
I’ve actually got good news for both:

1. For this issue with screenly-ose after a certain amount of time being stuck in one page, this causes the uzbl-core to get stuck in a Running state, after so much trial and error and searching for best way to alleviate this, the only thing I’ve been able to do to resolve it once and for all is to limit the time that each process spends in the CPU with LimitCPU.

This is my final working viewer service as a workaround while on the master/dev branch which uses uzbl which is most likely the primary cause of this…

---

2. For the google slide issue is simple: the problem is the way the viewer parses the URL to the browser, you simply need to change the URL from the long google slides one into a bitly one, so that the browser parses normal like this https://bit.ly/336A2Bi - instead of the long google one with special encoding characters, thus your loop and time delay will be properly executed in the browser… this is your google slides presentation URL in bitly form: https://bit.ly/336A2Bi , give it a try, you can do that for all URLs that have certain characters in the address…

[RobertE]

Wonderful. My display has now been up for over 3 hours. Better than it’s done for months, and looks like 3+ hours = ‘for ever’. Thanks so much for all your hard work. I can now take my Pi back to the Hall where the display is (I’ve had it at home for all this testing).
Might it be worth trying again with my Pi Zero W? I have several of these, and only one 3B+. Or might that be tempting providence? (I’m sure I read when I started that Screenly OSE should run on Pi Zero?
I shan’t complain if it doesn’t work, though - I have a working solution.
Do you think Screenly OSE will continue to use uzbl-core? It does seem to cause problems?

[RobertE]

Hi, Just for interest I plugged in my old Pi Zero, with Production version production@6c2c2fd and it copes with the Google Slides URL without problem?? I plan to change to Master/Development and the changes you’ve done, and see what happens.

[ealmonte32]

@RobertE

Might it be worth trying again with my Pi Zero W? I have several of these, and only one 3B+. Or might that be tempting providence? (I’m sure I read when I started that Screenly OSE should run on Pi Zero?
I shan’t complain if it doesn’t work, though - I have a working solution.
Do you think Screenly OSE will continue to use uzbl-core? It does seem to cause problems?

Yes screenly-ose works on the Pi Zero W as I have one and tested, but yes you need to definitely tweak it with things like setting gpu_mem to 192 (as you did with your Pi 3B).
The programmer who works on screenly-ose has started the move to a different browser because yes, uzbl is one of the main problems, but that is the experimental branch, long way to go before getting an experimental to production use. After all, this is all open source so we need volunteers who can help with the code.

---

@RobertE

Hi, Just for interest I plugged in my old Pi Zero, with Production version production@6c2c2fd and it copes with the Google Slides URL without problem?? I plan to change to Master/Development and the changes you’ve done, and see what happens.

Production and Master branch use the same browser yes, but the encoding of URL and parsing it (passing the URL onto the browser in a way it understands the characters and symbols) is done at the viewer.py (the main python program running the show) and imported modules and utilities all combined, and so rusko the programmer who works on that might have changed some things around in some part that takes the URL and parses it to the browser in order to accommodate something where maybe the production version did not do properly but the master version did, and maybe hasn’t had time to go into that part of the code to fix, or maybe it is a imported module issue, the reason why might not be something he is going to spend time with is because if the experimental browser will fix a lot of issues, he might as well not waste time with uzbl version and instead move onto working on the experimental branch instead… get it?

[RobertE]

Agree totally. I’m happy to use OpenSource software, and wish I could help with coding . . . but my days of working with Assembler Code in the 1970s (yes really!) are long behind me. I’ve had one or two dips into Python, but . . . I love problem-solving, and I hope helping with problem solvers! Good luck to you and Rusko in developing a more stable product.
I’ve just done all the tweaks on my Pi Zero, and set it off overnight (you probably guess I’m in UK?) to see how it gets on.

[luisscreenly]

I tried this and afterwards, the version didn’t change, still had production@6c2c2fd and there were complaints about version of python .

After, I tried /usr/local/sbin/upgrade_screenly.sh and that gave bunch error near the end during database migration…

My end goal is to get the changes in as suggested by ealmonte32 to get a stable system. I made the changes to the production OSE as ealmonte32 suggested but still hangs , so now I’m trying to install the latest development version.

Any idea what I could try next?

TASK [screenly : Run database migration] *******************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: [“python”, “/home/pi/screenly/bin/migrate.py”], “delta”: “0:00:00.148256”, “end”: “2020-04-08 10:09:21.854662”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2020-04-08 10:09:21.706406”, “stderr”: “Traceback (most recent call last):\n File “/home/pi/screenly/bin/migrate.py”, line 163, in \n migrate_drop_filename()\n File “/home/pi/screenly/bin/migrate.py”, line 153, in migrate_drop_filename\n with open_db_get_cursor() as (cursor, _):\n File “/usr/lib/python2.7/contextlib.py”, line 17, in enter\n return self.gen.next()\n File “/home/pi/screenly/bin/migrate.py”, line 53, in open_db_get_cursor\n with sqlite3.connect(database, detect_types=sqlite3.PARSE_DECLTYPES) as conn:\nsqlite3.OperationalError: unable to open database file”, “stderr_lines”: [“Traceback (most recent call last):”, " File “/home/pi/screenly/bin/migrate.py”, line 163, in “, " migrate_drop_filename()”, " File “/home/pi/screenly/bin/migrate.py”, line 153, in migrate_drop_filename", " with open_db_get_cursor() as (cursor, _):", " File “/usr/lib/python2.7/contextlib.py”, line 17, in enter", " return self.gen.next()", " File “/home/pi/screenly/bin/migrate.py”, line 53, in open_db_get_cursor", " with sqlite3.connect(database, detect_types=sqlite3.PARSE_DECLTYPES) as conn:", “sqlite3.OperationalError: unable to open database file”], “stdout”: “”, “stdout_lines”: []}

PLAY RECAP *************************************************************************************************************************************************************************
localhost : ok=36 changed=14 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

[ealmonte32]

just curious, instead of running the upgrade, can you simply run the install script again?

the upgrade script is the same as the master install.sh

[luisscreenly]

I deleted a couple of posts because of errors during the upgrade caused by running the upgrade as root. After upgrading as pi user upgrade completed Ok, so I’ll monitor now to see if it hangs on certain assets.

side note: system info … Unable to run tvservice. Unable to determine display power.

manually it works:
pi@media-x:~ $ tvservice -l
1 attached device(s), display ID’s are :
Display Number 2, type HDMI 0

[luisscreenly]

I think moving away from uzbl browser has really helped, so far no hanging on an asset that used to hang.
Thank You.

[ealmonte32]

@luisscreenly

If you happen to need to use the development/master version/branch but the only problem you run into is the uzbl browser causing too much memory consumption (which is the biggest issue) and making the pi unstable/unusable/hang, I can suggest you simply edit the python viewer file to limit the resources thus two lines added to the code will make it that any time the process viewer.py + uzbl-core consumes more than what you set, the process gets closed down and the browser starts again with fresh memory after restarting the browser.

Until qt browser becomes the default browser in the master branch, this is one of the easiest stop-gap methods you can use. Other methods i worked on like MemoryLimit in the service are too much compared to this simple way.

---

(see the attached screenshot for how i have mine running)

if you can SSH or ctrl+alt+f1 into the console:

edit the viewer.py file with nano:
nano ./screenly/viewer.py

1. on the top, just add:
   import resource

2. on the area before the author, add:
   resource.setrlimit(resource.RLIMIT_AS, (600000000,700000000))

these numbers are in bytes (soft limit, hard limit), for example it may seem like 600 MB soft limit and 700MB hard limit are a lot, but you have to remember in linux theres virtual and resident, and if you were to use the command top while testing this, you would see that your uzbl-core process terminates when the VIRT memory reaches that you have set, and the VIRT size will almost always be larger than RES size.

Give it a try, let me know how it goes, you can play around with the memory but since my pi 3 has 1GB and I use 192MB for GPU, it only gives like 800 of usable, and other services running take up memory as well, so you should not limit that high, usually the soft limit gets hit first and that kills the browser for 10 seconds or so, and this at least keeps your pi alive and reachable.
I also run different tests with the Pi and Screenly so for example right now this pi is running DietPi which is a slimmed down version of Raspbian Lite, so if you are running the regular raspbian lite or screenly-ose image from github, your memory usage might differ from mine, but it’s all closely similar and you should be able to play around with the 600 MB and 700 MB in order to find your sweet spot.

[RobertE]

Hi guys, great to hear that things are still moving. During the virus crisis, I’m afraid my system is ‘on hold’ so I can’t help with testing at the moment. But I’ll keep watching what goes on here, and hope to join you more actively - in a while.

[ealmonte32]

Hey @RobertE
I tested on multiple Pi’s and can confirm that this simpler method works. The person that adds those 2 lines will not have to worry about their Pi freezing/hanging/unresponsive due to memory consumption of viewer service.
If you add many many memory consuming web assets, you will just reach the limit sooner and the browser/service will restart itself, but no longer have to worry about coming to a display that has crashed =]

[RobertE]

Hi, I’ve just noticed that a new version was released only a week or so ago, so I downloaded that. Will that include these tweaks, or the new browser, or should I go back to installing as above, and making the tweak to the viewer.py file? Also I’m testing again with a Pi Zero W so only have 512 M RAM I think?

[ealmonte32]

That version was not really a new version, it was a rebuild according to vpetersson.
The tweak I made is a pending PR which means that until one of them looks at it and tests it themselves, the tweak does not go into the master branch.
Until that happens, you can simply look at the current pending PRs here:
Screenly-OSE Pull Requests

These are the current lines of code that need to be set in viewer.py in order to set limits based on percentage of RAM and not a hardcoded number:

import resource
from psutil import virtual_memory

# setting memory resource limits to prevent viewer from rendering the pi unresponsive due to memory consumption
mem = virtual_memory()
mem_soft = (mem.total * .75)
mem_hard = (mem.total * .90)
num_format = "{:.0f}".format

setlimit = 'resource.setrlimit(resource.RLIMIT_AS, (' + num_format(mem_soft) + ',' + num_format(mem_hard) + '))'
exec(setlimit)

logging.info("The current memory resource limit is set as:")
logging.info(setlimit)
#

As you can probably guess, the mem.total * .75 just means set the soft memory limit to 75% of the total current memory of the Pi, the soft vs hard is explained this way:

The soft limit is a value upon which the operating system will notify the process via a signal or typically restrict it.
An upper bound on the values is defined by the hard limit.

If you want, you can customize yours for the Pi Zero since it has less memory to be 90% on both soft and hard, that way, you at least get the most of the available memory before the process restarts itself, the good news is that you wont have to worry about the Pi hanging and freezing and becoming unreachable anymore as far as all my tests have shown.