502 Bad Gateway

I have few of these…
Raspberry Model|Model 3B+ Revision: 1.3 Ram: 1 GB Sony UK |
Screenly Version| production@d12df55c3cfec0cb8e31ec3fc811b1e783a0b636

and we get the following error messages often; most times a reboot will resolve but sometimes I have to run the upgrade script to fix.

Blockquote
2020/09/23 18:34:40 [error] 6339#6339: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.1, server: , request: “GET / HTTP/1.1”, upstream: “http://127.0.0.1:8080/”, host: “192.168.201.7”
2020/09/23 18:34:40 [error] 6339#6339: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.1, server: , request: “GET /favicon.ico HTTP/1.1”, upstream: “http://127.0.0.1:8080/favicon.ico”, host: “192.168.201.7”> Blockquote

If you need more info, please let me know I will provide.

Thank you.

who’s IP’s are these?
probably 192.168.101.1 is your gateway?
Who is 192.168.201.7 ? and do you have the proper route to get back to your gateway? because it’s a different subnet.

101.1 is my PC and 201.7 is the raspberry pi. I do have access to the web admin and ssh from from the 101 network when it is working.

The 502 gateway message shows up occasionally and usually often after several days of running. This happens to several of my Pi’s; not just the 201.7.

@jtlatgf9
are you sure that these subnets (192.168.201.x and 192.168.101.x) are routable? Doesn’t make sense to be anything else except a routing issue. Also, if 101.1 is your PC seems a bit weird since gateways are usually either “.1” or “.254” , also can you explain a bit more of this error, is this an asset you are adding? where is the error showing? on the actual screen or only logs? pls be more detailed, maybe show some screenshots.

Yes, those subnets are good. 101.x is my VPN to the office. I was able to get 201.7 working again by re-installing screenly-ose using /home/pi/screenly/bin/run_upgrade.sh but now have another at 205.7 that is acting up.

C:\Windows\System32>ssh pi@192.168.205.7
pi@192.168.205.7's password:
Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Sep 30 22:40:40 2020 from 192.168.101.1

permitted by applicable law.
Last login: Wed Sep 30 22:40:40 2020 from 192.168.101.1
pi@raspberrypi:~ $ w
 20:01:16 up 4 days,  3:39,  1 user,  load average: 0.77, 0.81, 0.77
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
pi       pts/2    192.168.101.8    20:01    3.00s  0.22s  0.02s w
pi@raspberrypi:~ $ exit
logout
Connection to 192.168.205.7 closed.

C:\Windows\System32>

here is an image while trying to access http://192.168.205.7 to log in to get to the assets… It will not go any further than this.
bad_gateway

This even happens with a computer on the same subnet.

pi@raspberrypi:/var/log/nginx $ cat error.log
2020/10/07 19:59:33 [error] 436#436: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.8, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "192.168.205.7"
2020/10/07 20:12:01 [error] 436#436: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.8, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "192.168.205.7"
2020/10/07 20:12:01 [error] 436#436: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.8, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "192.168.205.7"
2020/10/07 20:12:04 [error] 436#436: *6 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.8, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "192.168.205.7"
2020/10/07 20:14:24 [error] 436#436: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.101.8, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "192.168.205.7"
2020/10/07 20:15:28 [error] 436#436: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.205.42, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "192.168.205.7"
2020/10/07 20:15:28 [error] 436#436: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.205.42, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "192.168.205.7"

Maybe this is something … from /var/log/daemon.log as this repeats itself every 3 seconds…

Oct  7 20:21:19 raspberrypi systemd[1]: screenly-web.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:19 raspberrypi systemd[1]: Stopped Screenly Web UI.
Oct  7 20:21:19 raspberrypi systemd[1]: Starting Screenly Web UI...
Oct  7 20:21:19 raspberrypi systemd[1]: Started Screenly Web UI.
Oct  7 20:21:19 raspberrypi python[12700]: Traceback (most recent call last):
Oct  7 20:21:19 raspberrypi python[12700]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:19 raspberrypi python[12700]:     import psutil
Oct  7 20:21:19 raspberrypi python[12700]: ImportError: No module named psutil
Oct  7 20:21:19 raspberrypi systemd[1]: screenly-web.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:19 raspberrypi systemd[1]: screenly-web.service: Unit entered failed state.
Oct  7 20:21:19 raspberrypi systemd[1]: screenly-web.service: Failed with result 'exit-code'.
Oct  7 20:21:21 raspberrypi systemd[1]: screenly-celery.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:21 raspberrypi systemd[1]: Stopped Screenly celery worker.
Oct  7 20:21:21 raspberrypi systemd[1]: Started Screenly celery worker.
Oct  7 20:21:23 raspberrypi celery[12702]: Traceback (most recent call last):
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/bin/celery", line 10, in <module>
Oct  7 20:21:23 raspberrypi celery[12702]:     sys.exit(main())
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
Oct  7 20:21:23 raspberrypi celery[12702]:     _main()
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
Oct  7 20:21:23 raspberrypi celery[12702]:     cmd.execute_from_commandline(argv)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
Oct  7 20:21:23 raspberrypi celery[12702]:     super(CeleryCommand, self).execute_from_commandline(argv)))
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 273, in execute_from_commandline
Oct  7 20:21:23 raspberrypi celery[12702]:     argv = self.setup_app_from_commandline(argv)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
Oct  7 20:21:23 raspberrypi celery[12702]:     self.app = self.find_app(app)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 501, in find_app
Oct  7 20:21:23 raspberrypi celery[12702]:     return find_app(app, symbol_by_name=self.symbol_by_name)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 359, in find_app
Oct  7 20:21:23 raspberrypi celery[12702]:     sym = symbol_by_name(app, imp=imp)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 504, in symbol_by_name
Oct  7 20:21:23 raspberrypi celery[12702]:     return imports.symbol_by_name(name, imp=imp)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py", line 57, in symbol_by_name
Oct  7 20:21:23 raspberrypi celery[12702]:     module = imp(module_name, package=package, **kwargs)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 104, in import_from_cwd
Oct  7 20:21:23 raspberrypi celery[12702]:     return imp(module, package=package)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
Oct  7 20:21:23 raspberrypi celery[12702]:     __import__(name)
Oct  7 20:21:23 raspberrypi celery[12702]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:23 raspberrypi celery[12702]:     import psutil
Oct  7 20:21:23 raspberrypi celery[12702]: ImportError: No module named psutil
Oct  7 20:21:23 raspberrypi systemd[1]: screenly-celery.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:23 raspberrypi systemd[1]: screenly-celery.service: Unit entered failed state.
Oct  7 20:21:23 raspberrypi systemd[1]: screenly-celery.service: Failed with result 'exit-code'.
Oct  7 20:21:25 raspberrypi systemd[1]: screenly-web.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:25 raspberrypi systemd[1]: Stopped Screenly Web UI.
Oct  7 20:21:25 raspberrypi systemd[1]: Starting Screenly Web UI...
Oct  7 20:21:25 raspberrypi systemd[1]: Started Screenly Web UI.
Oct  7 20:21:25 raspberrypi python[12718]: Traceback (most recent call last):
Oct  7 20:21:25 raspberrypi python[12718]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:25 raspberrypi python[12718]:     import psutil
Oct  7 20:21:25 raspberrypi python[12718]: ImportError: No module named psutil
Oct  7 20:21:25 raspberrypi systemd[1]: screenly-web.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:25 raspberrypi systemd[1]: screenly-web.service: Unit entered failed state.
Oct  7 20:21:25 raspberrypi systemd[1]: screenly-web.service: Failed with result 'exit-code'.
Oct  7 20:21:28 raspberrypi systemd[1]: screenly-celery.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:28 raspberrypi systemd[1]: Stopped Screenly celery worker.
Oct  7 20:21:28 raspberrypi systemd[1]: Started Screenly celery worker.
Oct  7 20:21:30 raspberrypi celery[12721]: Traceback (most recent call last):
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/bin/celery", line 10, in <module>
Oct  7 20:21:30 raspberrypi celery[12721]:     sys.exit(main())
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
Oct  7 20:21:30 raspberrypi celery[12721]:     _main()
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
Oct  7 20:21:30 raspberrypi celery[12721]:     cmd.execute_from_commandline(argv)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
Oct  7 20:21:30 raspberrypi celery[12721]:     super(CeleryCommand, self).execute_from_commandline(argv)))
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 273, in execute_from_commandline
Oct  7 20:21:30 raspberrypi celery[12721]:     argv = self.setup_app_from_commandline(argv)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
Oct  7 20:21:30 raspberrypi celery[12721]:     self.app = self.find_app(app)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 501, in find_app
Oct  7 20:21:30 raspberrypi celery[12721]:     return find_app(app, symbol_by_name=self.symbol_by_name)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 359, in find_app
Oct  7 20:21:30 raspberrypi celery[12721]:     sym = symbol_by_name(app, imp=imp)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 504, in symbol_by_name
Oct  7 20:21:30 raspberrypi celery[12721]:     return imports.symbol_by_name(name, imp=imp)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py", line 57, in symbol_by_name
Oct  7 20:21:30 raspberrypi celery[12721]:     module = imp(module_name, package=package, **kwargs)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 104, in import_from_cwd
Oct  7 20:21:30 raspberrypi celery[12721]:     return imp(module, package=package)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
Oct  7 20:21:30 raspberrypi celery[12721]:     __import__(name)
Oct  7 20:21:30 raspberrypi celery[12721]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:30 raspberrypi celery[12721]:     import psutil
Oct  7 20:21:30 raspberrypi celery[12721]: ImportError: No module named psutil
Oct  7 20:21:30 raspberrypi systemd[1]: screenly-celery.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:30 raspberrypi systemd[1]: screenly-celery.service: Unit entered failed state.
Oct  7 20:21:30 raspberrypi systemd[1]: screenly-celery.service: Failed with result 'exit-code'.
Oct  7 20:21:30 raspberrypi systemd[1]: screenly-web.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:30 raspberrypi systemd[1]: Stopped Screenly Web UI.
Oct  7 20:21:30 raspberrypi systemd[1]: Starting Screenly Web UI...
Oct  7 20:21:31 raspberrypi systemd[1]: Started Screenly Web UI.
Oct  7 20:21:31 raspberrypi python[629]: omxplayer timed out
Oct  7 20:21:31 raspberrypi python[629]: Generating asset-list...
Oct  7 20:21:31 raspberrypi python[629]: Showing asset SilentSalesman_BestSellerWall.jpg (image)
Oct  7 20:21:31 raspberrypi python[629]: Sleeping for 20
Oct  7 20:21:31 raspberrypi python[12737]: Traceback (most recent call last):
Oct  7 20:21:31 raspberrypi python[12737]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:31 raspberrypi python[12737]:     import psutil
Oct  7 20:21:31 raspberrypi python[12737]: ImportError: No module named psutil
Oct  7 20:21:31 raspberrypi systemd[1]: screenly-web.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:31 raspberrypi systemd[1]: screenly-web.service: Unit entered failed state.
Oct  7 20:21:31 raspberrypi systemd[1]: screenly-web.service: Failed with result 'exit-code'.
Oct  7 20:21:35 raspberrypi systemd[1]: screenly-celery.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:35 raspberrypi systemd[1]: Stopped Screenly celery worker.
Oct  7 20:21:35 raspberrypi systemd[1]: Started Screenly celery worker.
Oct  7 20:21:36 raspberrypi systemd[1]: screenly-web.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:36 raspberrypi systemd[1]: Stopped Screenly Web UI.
Oct  7 20:21:36 raspberrypi systemd[1]: Starting Screenly Web UI...
Oct  7 20:21:36 raspberrypi systemd[1]: Started Screenly Web UI.
Oct  7 20:21:37 raspberrypi python[12762]: Traceback (most recent call last):
Oct  7 20:21:37 raspberrypi python[12762]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:37 raspberrypi python[12762]:     import psutil
Oct  7 20:21:37 raspberrypi python[12762]: ImportError: No module named psutil
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-web.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-web.service: Unit entered failed state.
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-web.service: Failed with result 'exit-code'.
Oct  7 20:21:37 raspberrypi celery[12747]: Traceback (most recent call last):
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/bin/celery", line 10, in <module>
Oct  7 20:21:37 raspberrypi celery[12747]:     sys.exit(main())
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
Oct  7 20:21:37 raspberrypi celery[12747]:     _main()
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
Oct  7 20:21:37 raspberrypi celery[12747]:     cmd.execute_from_commandline(argv)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
Oct  7 20:21:37 raspberrypi celery[12747]:     super(CeleryCommand, self).execute_from_commandline(argv)))
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 273, in execute_from_commandline
Oct  7 20:21:37 raspberrypi celery[12747]:     argv = self.setup_app_from_commandline(argv)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 479, in setup_app_from_commandline
Oct  7 20:21:37 raspberrypi celery[12747]:     self.app = self.find_app(app)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 501, in find_app
Oct  7 20:21:37 raspberrypi celery[12747]:     return find_app(app, symbol_by_name=self.symbol_by_name)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 359, in find_app
Oct  7 20:21:37 raspberrypi celery[12747]:     sym = symbol_by_name(app, imp=imp)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 504, in symbol_by_name
Oct  7 20:21:37 raspberrypi celery[12747]:     return imports.symbol_by_name(name, imp=imp)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py", line 57, in symbol_by_name
Oct  7 20:21:37 raspberrypi celery[12747]:     module = imp(module_name, package=package, **kwargs)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 104, in import_from_cwd
Oct  7 20:21:37 raspberrypi celery[12747]:     return imp(module, package=package)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
Oct  7 20:21:37 raspberrypi celery[12747]:     __import__(name)
Oct  7 20:21:37 raspberrypi celery[12747]:   File "/home/pi/screenly/server.py", line 10, in <module>
Oct  7 20:21:37 raspberrypi celery[12747]:     import psutil
Oct  7 20:21:37 raspberrypi celery[12747]: ImportError: No module named psutil
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-celery.service: Main process exited, code=exited, status=1/FAILURE
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-celery.service: Unit entered failed state.
Oct  7 20:21:37 raspberrypi systemd[1]: screenly-celery.service: Failed with result 'exit-code'.
Oct  7 20:21:42 raspberrypi systemd[1]: screenly-web.service: Service hold-off time over, scheduling restart.
Oct  7 20:21:42 raspberrypi systemd[1]: Stopped Screenly Web UI.
Oct  7 20:21:42 raspberrypi systemd[1]: Starting Screenly Web UI...
Oct  7 20:21:42 raspberrypi systemd[1]: Started Screenly Web UI.

Thank you for looking into this issue and let me know if there is anything else I can post to assist.
Jeff

There you go, seems like the module psutil is missing…
Please type this in:
sudo pip install -U psutil==5.6.7
then reboot.

Actually, if you want to be 100% sure you have all requirements, use the file from my fork because the PR for updating this is pending:

wget https://raw.githubusercontent.com/ealmonte32/screenly-ose/master/requirements/requirements.txt -O /tmp/requirements.txt

sudo pip install -U --no-cache-dir -r /tmp/requirements.txt

reboot, then it should work…

This is why your having the issue:

pi@rpi3blk:~ $ cat ./screenly/server.py | more
#!/usr/bin/env python
# -*- coding: utf-8 -*-

__author__ = "Screenly, Inc"
__copyright__ = "Copyright 2012-2020, Screenly, Inc"
__license__ = "Dual License: GPLv2 and Commercial License"

import json
import pydbus
import psutil
import re
import sh

As you can see, the server.py requires psutil amongst others, if one module fails, the program doesn’t run.

That did the trick … wonder why the psutil module didn’t get installed on some of my devices but did on others.

Thank you for the resolution!!

1 Like

@jtlatgf9 , can you tell us how this device that didnt get it was installed? pls be very specific so that we can try and figure out if this is an issue with sprint 10.1 or something else, thanks.

I am 99% certain this was installed using Noobs but have done a couple of clones as well so there is that 1% chance it was a cloned copy of a Noobs install. We have several of these installed by a cloned image and directly from Noobs that have had this issue in the present and the past; not just this one.

I have run the run_upgrade.sh script a few times since then on this box and all the others; some have been fixed and others have not so, someone might want to check that as well.

Thanks.