Sorry, I do not have access to my 2 screenly remote. I can not make the change for now.
Ok I am ready with a screenly development branch ready to test.
Which website do you want me to test besides screenly weather and boardclock one?
ok I would like you to test this link. thank you
https://www.mywellness.com/kcchambery/classtimetable
And this
Ok, test started.
Quick question related to this issue, do you know if the get requests being done by viewer.py are set to verify=False by default because of the logs displaying the InsecureRequestWarning? also because it is the exact response when tested like this:
>>> requests.get('https://weather.srly.io/?lat=40.930240285321055&lng=-74.06504906356537&lang=en&24h=0&wind_speed=0')
<Response [200]>
>>> requests.get('https://weather.srly.io/?lat=40.930240285321055&lng=-74.06504906356537&lang=en&24h=0&wind_speed=0', verify=False)
/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
<Response [200]>
>>>
If the request is being made with False verification which is not the default, where in the viewer.py is this set, I cant find the (uri, verify=False) flag as part of the get request… I just want to confirm this before I keep looking around other areas for possible troubleshooting.
While working on some of our kubernetes clusters this weekend, I noticed an incorrectly configured DNS record that could be related to this. I’m curious if you are able to still reproduce this?
@vpetersson
You dont even know how many things I’ve tried, being a network admin / sys admin for 12 years I hate to not be able to solve network related things like this, so I put up different test scenarios…
I know 100% it is not screenly related, but more along the lines of python/urllib3/requests issue, but you know not many people know that and think it is something with screenly.
Which DNS settings? resolv.conf generates the correct nameservers from the network daemons.
Three of my test scenarios are running screenly-ose non-stop for about a month now without any errors, browser restarts, browser hangs, etc, and that is because I am adding the Assets from my local web server internally, and using iframes for the sites that have given the errors (screenly weather one, boardclock still do), and some are running from Google sites, which seems to prove the error is when a website does not respond to the request within the timeout period…
One of my latest test scenario includes running pip install requests[security] and updating all dependencies, I changed your lib/utils.py line and set ssl_verify to True, and changed timeout from 10 to 20:
That should help with users complaining about InsecureRequestWarning since that makes all requests default to verify=True, and should give the connection timeout more time to complete.
Tests included: Assets with only HTTP sites, only HTTPS sites, using webpagetest.org to see all the requests being made from each site, and I’ve compiled so many notes, uzbl config-screenly making no difference when the request module is used in terms of the TLS errors… and ssl verification changed to 0…
I’ve been running these specific assets for 3 days without the TLS error:
No, this was a DNS issue on our end (i.e. in the back-end). Hence it was totally out of your control unfortunately. It was a tricky one i discovered on accident. That said - i’m not 100% sure that was the root cause but it stood out to me as a potential one and would have been resolved this past Saturday if so. Keep me posted if it actually did resolve it. If so, I do apologize for all the wasted time.
Sorry but TLS error still happens when using weather.srly.io and boardclock.com - it’s like guaranteed that if you have those assets as URLs you will run into the issue (whether on wired or wifi) on prod and dev branches (AFAIK).
I think I’m getting closer to narrowing down the exact issue by researching the specific details of certificates used on sites that give error vs the ones that dont.
This is what I believe has to do with issue.
The differences I’ve seen (using SSL tools to check) between assets that give the TLS error and others that dont is that the certificate of the sites that give TLS error have their certificate as ECC and the ones that dont are RSA.
Which is weird because as I read, ECC are smaller size, faster handshake, etc, vs RSA… with same level of security… but, this is the only difference I can find by comparing certificates.
I also know that python and ECC were at one point not compatible (in openssl or something like that).
I just installed ecc package sudo pip install ecc to see if helps, but I dont think it will since it is not being used/imported to viewer.py?
Thoughts?
@vpetersson
Lol, this makes no sense at all, can you believe this might be the issue?
Since it is a link reference no wonder it didn’t show up here on the trace of requests:
http://webpagetest.org/result/190329_MC_7829cac8d4a72db2c5c66a1916dbb03a/1/details/
But maybe if I add all the icon links one by one to Assets and see which one fails…
I am going to remove the weather.srly.io URL and keep the boardclock one so that when that gives the error as well I can go into and inspect the element and see which full URL it shows as the one causing error.
This one is from clock.srly.io
I can’t get boardclock URL to give me the same error, it shows error performing handshake but the inspect does not show an error… ughh.
If you have some time, can you check if you have something in the back-end of weather.srly.io that references the favicon like this forum topic discusses… just trying to isolate as much as possible since that error only happens in very few sites…
I don’t think this has to do with EC vs RSA. You’re absolutely right that EC keys are smaller than RSA, but if that was the case, we should (in theory) be able to easily reproduce the same issue with a simple requests.get('https://[...]').
Assuming that the certificate chain used in UZBL is somehow different than the one used by Python. If so, that would potentially explain the problem.
If you want to dive even deeper, I suggest using openssl. I wrote a blog-post sometime back about how to troubleshoot similar issues.
(As a side-note, there is no need to install any additional package to use EC with requests. The system supports both EC and RSA out of the box.)
@vpetersson
Thanks for pointing me in the right direction.
I read your blog post plus other posts regarding openssl and curl.
Lots of testing done using curl and openssl… comparing certs and responses, etc…
The pattern I see is that when a request is made and system uses openssl backend, depending on the certificate type like ones that host multiple domains, and how the chain is set up, you get the tls error unless you specify the -servername flag.
I cant find anything with regards to uzbl and certificate chain used, but I have edited the config file and had set ssl verify there to 0 and many other mix of config settings to no avail.
What can we do to specify that on the requests being made that the -servername flag be added? I changed a few lines in openssl.cnf but I dont see anything with regards to default connection parameters.
All screenshots:
For the first screenshot, you must use curl -I https://X --resolve X:443. In the example above, you’re using http://, which renders the --resolve for 443 moot.
Wow… how did I miss that, ugh, must of been way too late for such mistake…
So curl was not really providing me much info with regards to handshake error, so I started focusing more on openssl and I saw the handshake errors when those sites were not “connected” with the servername flag… Which I have been trying to look into at the library level and see if I can find some parameters that would do this on the backend as default.
What doesn’t make sense is the fact that this can’t be reproduced on the spot, the browser cycles through all the URLs and at a random time it decides that it will give the tls error… 
Yeah it’s a very annoying error indeed. We shouldn’t rule out that this can be a bug in UZBL, which is why i wanted to do this exercise. If we are able to reproduce it either directly in Python or using curl/openssl, that would imply that it’s a more lower level issue.
As far as abstraction goes, using openssl will give you much lower level access than curl (which uses openssl behind the scenes).
With regards to your question, I’m not well versed enough with the servername flag in openssl, but it might be worth checking out the docs to see what that flag actually does, but IIRC, it has something to do with SNI, which would explain the mismatch if you do not have it set.
@vpetersson
Correct, it adds the ability for openssl to use SNI during the ClientHello.
This article quickly explains it in two paragraphs…
And the openssl s_client --help menu shows this:
-servername val Set TLS extension servername in ClientHello
But as you point out, this is simply the inability of UZBL to not “crash” or stop going to next Asset/URL when it encounters the tls handshake error during negotiation on the sites that have multiple certificates and require SNI.
I still can’t comprehend why sometimes and sometimes it doesn’t encounter the handshake error if without the servername flag these websites always give the fatal error as the openssl tests show. I guess only the UZBL developers could answer that (I emailed them hoping to get some answers).
So what I have done at the config level is tell UZBL to simply “exit” when it encounters any load error event, this doesn’t really exit the browser as I’ve experienced, but I am hoping it doesn’t stop the browser the way it does and so it allows it to continue onto next asset. I have not seen the error page yet (testing is still ongoing).
@androidofon
If you are still trying to find a workaround for this, you could try using the one line I added to the config file or wait until I can give feedback that after multiple days the error has not appeared on screen.
It is better if we both have the same configuration so that multiple people test at the same time, maybe the error happens to you maybe it doesn’t, but this helps with testing.
sudo nano ./.config/uzbl/config-screenly
add the line below:
@on_event LOAD_ERROR exit
Let me know what you experience…
@androidofon
I have not seen the error “Unable to load page” for one week now with my changes to config-screenly file, I think you should give it a try.
it looks like this:
set show_status = 0
set on_event = request ON_EVENT
set cache_control = document_viewer
set fifo_dir = /tmp
set socket_dir = /tmp
@on_event LOAD_START js alert=function(e){return true};confirm=function(e){return true};prompt=function(e){return true};
@on_event LOAD_COMMIT js alert=function(e){return true};confirm=function(e){return true};prompt=function(e){return true};
@on_event LOAD_ERROR exit
Note:
Just to have here as reference as well so users understand a bit more of what seems to be happening:
On this scenario (@vpetersson correct me if I’m wrong or a bit off) the requests module/library uses the pyopenssl thin wrapper to call functions from the openssl library and can use the certifi package to verify SSL certificates. It seems UZBL (the current browser screenly production uses to show URLs/Assets) sometimes runs into issues when trying to connect to and display websites that have multiple certificates set up for the same IP.
This looks like a sensible workaround for the problem, @ealmonte32. Would you mind doing a PR for this?
Thanks @ealmonte32 , I’ve implemented your work around however now when the SSL issue arrises I just get a loop of a black screen then a white screen
I’ve checked config-screenly file and it’s definitely exactly as you suggested. Maybe I’ve missed a step?
I want to, but I need to make sure it works for others as the workaround since that guy on github said he did do this and 10mins later he got the error… but I am not so sure he did the exact steps properly…
I need to make sure someone who is having the TLS errors constantly, does this, and then gives feedback since I was also messing with one of the cacert.pem files (replacing it with mozilla root cert) and certifi library.
I see now that @morktron gets black screen which I got once or twice but not a lot.
I think it might have to do with setting those directories to /tmp, instead of leaving the browser to handle it however it wants.
Mark, I have commented out some extra few lines of my config file so let me see what happens to see if that helps.
I am on vacation this coming week so I wont posting for a bit.
This is how I am leaving the config file running:
set show_status = 1
set on_event = request ON_EVENT
@on_event LOAD_START js alert=function(e){return true};confirm=function(e){return true};prompt=function(e){return true};
@on_event LOAD_COMMIT js alert=function(e){return true};confirm=function(e){return true};prompt=function(e){return true};
@on_event LOAD_ERROR exit
2 Likes