Discussion:
lua haproxy-auth-request - round 2
Computerisms Corporation
2018-09-24 23:08:54 UTC
Permalink
Hi Tim and other Gurus,

I am using the auth-request lua script from here (thanks Tim, you rock
for publishing this):

https://github.com/TimWolla/haproxy-auth-request

Haproxy is sitting in front of two nodejs apps; etherpad and ethercalc.
it is also sitting in front of apache which is serving standard websites
such as static html, Joomla, Wordpress, Drupal, etc. I am using the lua
auth script to employ apache's authnz external authentication program,
which is authenticating against an IMAP server, and I have got my
haproxy configured such that it only fires the lua script when accessing
the nodejs apps.

Things are mostly working, but there is one problem I am having with
etherpad that can be fixed by disabling the lua script, so even though I
am not 100% sure the problem is not on the etherpad side, I am starting
from here. To date I am not able to replicate this issue on ethercalc,
the lua auth script seems to work fine every where else.

specifically; I can load the etherpad landing page reliably. to load a
pad one changes the path of the url, for example one could have two pads
with the following urls: http://pad.domain.tld/p/nameofpad and
http://pad.domain.tld/p/otherpadname. These pads eventually time out
and produce an error regarding a null variable from etherpad if the lua
auth script is enabled in the haproxy config. etherpad has other
modules that can be accessed at other URL paths, such as /metrics and
/stats, these work fine, the error seems to be limited to loading the
pads only.

The error I am seeing in etherpad is described quite accurately here:

https://github.com/ether/etherpad-lite/issues/3047

The short version of that is that a variable or variables are empty and
should not be. According to the guy who got things working there, he
"redid" his proxy setup and it solved the problem. unfortunately,
etherpad doesn't have any documentation for using haproxy as of yet, so
trying to puzzle out what is different. this is my one/only indication
that this problem might need to be solved on the etherpad side.

I figured there must be some difference with a cookie or header or
something, so I stripped all SSL configs and did a tcpdump to capture
the full text of the traffic. I expected and found some differences,
but every thing looks pretty much the same to me. I have spent quite a
few hours searching those dumps, if the answer is there it is too
slippery for my eye to land on.

Log files are equally unhelpful; the etherpad log shows the null
variable, but I am able to find no clues in logs for haproxy, etherpad,
or the apache instance that is doing the authnz auth.

In my investigation, I found that sometimes apache would return a 304 or
408 from the authentication instance. I altered the lua script to
return an auth_response_successful on these codes, but it didn't fix the
problem.

through my tcpdump, I discovered that etherpad is using websockets. I
found this page:

https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/

which indicates that haproxy will automatically change to tunnel mode
and support websockets. I can find no reason why the lua script might
interfere with this, and ethercalc also uses websockets, but so far it
is the only thing I can find that *might be causing data to not be
passed for the variable(s) to be populated.

if you are still with me, thank you so much for reading this far. I
would truly appreciate any thoughts you might have on how to diagnose
what is causing this issue...
--
Bob Miller
Cell: 867-334-7117
Office: 867-633-3760
www.computerisms.ca
Tim Düsterhus
2018-09-26 22:38:43 UTC
Permalink
Bob,
if you are still with me, thank you so much for reading this far.  I
would truly appreciate any thoughts you might have on how to diagnose
what is causing this issue...
While I don't have any idea from the top of my head your configuration
surely would be helpful.

You might also want to check whether the webbrowser is able to:
a) Set-Up the Websocket with auth-request in between (you should see a
101 Switching Protocols in it's network console).
b) Send credentials for basic authentication for Websockets.

Best regards
Tim Düsterhus
Computerisms Corporation
2018-10-05 20:10:14 UTC
Permalink
Hi Tim,

Thanks for the response, and apologies for the delay. Popularity is
advertised as a good thing, but I have my doubts. Regardless, I am back
on track with this project for the moment.
Post by Tim Düsterhus
While I don't have any idea from the top of my head your configuration
surely would be helpful.
right, should have included that the first time, will put it at the
bottom of this mail.
Post by Tim Düsterhus
a) Set-Up the Websocket with auth-request in between (you should see a
101 Switching Protocols in it's network console).
Okay, this is something I hadn't looked at. Not 100% sure I am
interpreting correctly, but assuming I am, then the browser is reporting
that the connection is successfully upgrading to websockets both with
and without the Lua script enabled. As far as I can tell, the only
thing that changes is the cookies and keys. There is a significant
difference in the "waiting" response, though; ~350ms with the lua
script, but less than 5ms without it.
Post by Tim Düsterhus
b) Send credentials for basic authentication for Websockets.
hm, so I don't seem to be able to connect to etherpad directly using a
ws:// schematic in chrome or firefox, and I think that is what you mean.
not sure if that is just me, though, will work some more on that and
see if I can figure out if I am doing something wrong there...

My haproxy.cfg:

Note in the front end I have two lines commented; with these lines
commented, everything works, can reload hundreds of times with no error.
With the lines uncommented, the auth works, the main landing page
works, but accessing the actual pad does not work. the tables entries
seem to work fine either way.

I have uncommented these lines to induce failure, and pasted a copy of
the haproxy logs of the event at
http://www.computerisms.ca/haproxy.txt

At the bottom of the log file, it appears to me that I get a 200 after
the websocket upgrade, which I interpret to mean it was successful, but
at that point the page spins and some 150-300 seconds later I get the
error page displayed on the screen with no more log entries in haproxy.


global
debug
log /dev/log local1 debug
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-options no-sslv3
lua-load /Computerisms/config/etc/haproxy.auth.lua

defaults
log global
mode http
option httplog
option dontlognull

frontend httpfront
bind ${ADDRESS}:80 v4v6
bind ${ADDRESS}:443 v4v6 ssl crt /Computerisms/config/certificates/
redirect scheme https code 301 if !{ ssl_fc }
mode http
option httplog
log global
# http-request lua.auth-request auth_request /index.html
## ACLs
acl tables.computerisms.ca ssl_fc_sni -i tables.computerisms.ca
acl pad.computerisms.ca ssl_fc_sni -i pad.computerisms.ca
## AUTHREQ
use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } tables.computerisms.ca
# use_backend auth_request if ! {
var(txn.auth_response_successful) -m bool } pad.computerisms.ca
## AUTHBACKEND
use_backend tables.computerisms.ca if tables.computerisms.ca
use_backend pad.computerisms.ca if pad.computerisms.ca
default_backend mooglehttps

backend auth_request
mode http
server auth-request 127.0.0.1:8044 check
# option httpclose
# option forwardfor

backend mooglehttps
balance leastconn
mode http
option httpclose
option forwardfor
option log-health-checks
option httpchk
server sand1lian 192.168.25.52:48443 check send-proxy-v2 ssl
verify none
server sand2lian 192.168.25.53:48443 check send-proxy-v2 ssl
verify none

## BEGIN pad.computerisms.ca
backend pad.computerisms.ca
balance leastconn
mode http
cookie sessionID insert nocache indirect
# option httpclose
option forwardfor
server sand1lian 192.168.25.52:19008 cookie sand1pad
server sand2lian 192.168.25.53:19008 cookie sand2pad
## END pad.computerisms.ca
## BEGIN tables.computerisms.ca
backend tables.computerisms.ca
balance leastconn
mode http
cookie sessionID insert nocache indirect
option httpclose
option forwardfor
server sand1lian 192.168.25.52:29000 check cookie sand1tables
server sand2lian 192.168.25.53:29000 check cookie sand2tables
## END tables.computerisms.ca

Thanks again for taking a look, truly appreciated...
Post by Tim Düsterhus
Best regards
Tim Düsterhus
Tim Düsterhus
2018-10-09 17:40:48 UTC
Permalink
Bob,
Okay, this is something I hadn't looked at.  Not 100% sure I am
interpreting correctly, but assuming I am, then the browser is reporting
that the connection is successfully upgrading to websockets both with
and without the Lua script enabled.  As far as I can tell, the only
thing that changes is the cookies and keys.  There is a significant
difference in the "waiting" response, though; ~350ms with the lua
script, but less than 5ms without it.
Use your browser's dev tools (F12 for Firefox, Chrome). Search for the
websocket request. It's just called '/websocket' for Etherpad. It should
return a 101 Switching Protocols. In Google Chrome you can even look at
the WebSocket Frames.
Post by Tim Düsterhus
b) Send credentials for basic authentication for Websockets.
hm, so I don't seem to be able to connect to etherpad directly using a
ws:// schematic in chrome or firefox, and I think that is what you mean.
 not sure if that is just me, though, will work some more on that and
see if I can figure out if I am doing something wrong there...
No, use the dev tools and look at the request headers, whether there is
an Authorization header. Of course oauth_proxy needs to be enabled.

I attached a screenshot of Chrome's dev tools.

If you send me credentials in private I can take a look myself, if you want.

Best regards
Tim DÃŒsterhus
Computerisms Corporation
2018-10-10 20:39:22 UTC
Permalink
Hi Tim,
Post by Tim Düsterhus
Bob,
Okay, this is something I hadn't looked at.  Not 100% sure I am
interpreting correctly, but assuming I am, then the browser is reporting
that the connection is successfully upgrading to websockets both with
and without the Lua script enabled.  As far as I can tell, the only
thing that changes is the cookies and keys.  There is a significant
difference in the "waiting" response, though; ~350ms with the lua
script, but less than 5ms without it.
Use your browser's dev tools (F12 for Firefox, Chrome). Search for the
websocket request. It's just called '/websocket' for Etherpad. It should
return a 101 Switching Protocols. In Google Chrome you can even look at
the WebSocket Frames.
okay, that is what I was looking at, and yes, it does show the websocket
upgrade request, see below. it looks like the websockets upgrade is
/socket.io followed by some arguments denoted with a /?.

Can't paste, but it starts with a call to this file with code 200;
getting en-gb.json with code 304;
another call to /socket.io with code 200;
then the code 101 with an upgrade;
10 seconds later it shows a POST to the /socket.io with code 200.
Post by Tim Düsterhus
Post by Tim Düsterhus
b) Send credentials for basic authentication for Websockets.
hm, so I don't seem to be able to connect to etherpad directly using a
ws:// schematic in chrome or firefox, and I think that is what you mean.
 not sure if that is just me, though, will work some more on that and
see if I can figure out if I am doing something wrong there...
No, use the dev tools and look at the request headers, whether there is
an Authorization header. Of course oauth_proxy needs to be enabled.
I think by oauth_proxy you mean the auth method I am using with the lua
script, in my case authnz_external from apache. if you literally mean
oauth_proxy, maybe this is the link I am missing. I have not configured
oauth_proxy any where, as I am/(was?) not planning to use it.

But by the request headers, I understand what you mean now. I am using
firefox primarily, but appears I get the same basic thing in chrome.
Firefox and Chrome both report 101 switching protocols, the websocket
upgrade header, and the authorization header:

Response headers (175 B)
Connection Upgrade
Sec-WebSocket-Accept gU996yXfDiutgquFxiRxGuffglc=
Sec-WebSocket-Extensions permessage-deflate
Upgrade websocket
Request headers (936 B)
Accept text/html,application/xhtml+xm…plication/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate, br
Accept-Language en-CA,en-US;q=0.7,en;q=0.3
Authorization Basic Ym9iQGNvbXB1dGVyaXNtcy5j…HMHJnZTB1cyhXZWUpRWdnbGl0cw==
Cache-Control no-cache
Connection keep-alive, Upgrade
Cookie _ga=GA1.2.1611432529.153149834…fFXUaWksCAAAJ; language=en-gb
Host pad.computerisms.ca
Origin https://pad.computerisms.ca
Pragma no-cache
Sec-WebSocket-Extensions permessage-deflate
Sec-WebSocket-Key DhxDcq4PkH+/TF2kaSW8JQ==
Sec-WebSocket-Version 13
Upgrade websocket
User-Agent Mozilla/5.0 (X11; Ubuntu; Linu…) Gecko/20100101 Firefox/62.0
Post by Tim Düsterhus
I attached a screenshot of Chrome's dev tools.
Thanks, that confirms I am using the correct thing.
Post by Tim Düsterhus
If you send me credentials in private I can take a look myself, if you want.
will follow up off list...
Post by Tim Düsterhus
Best regards
Tim Düsterhus
Loading...