Yes, I should have done last in my last email post. Sorry about that.
Built with multi-threading support.
Built with network namespace support.
Total: 3 (3 usable), will use epoll.
A. the application that we are trying to use: jupyterhub --
B. We have shibboleth IDP that all our internal sites do authentication
from.
C. This application Jupyterhub spawns its own ( internally built ) proxy
section in apache config above.
D. Jupyterhub can directly authenticate with shibboleth ( some additional
needed. therefore, I decided to wrap jupytrehub within apache as apache
shibboleth RPM ).
get the SP working correctly.
-- from the backend, shibboleth responds to me with a login screen.
-- Shibboleth hands my username as an attribute called "UserID". to
Apache, and I forward that "UserID" to jupyterhub. Apache pulls the
jupyterhub instance, presents to the user.
machine. I think this is due to SSO being done by shibboleth. Since
IP. In this process, HAProxy loses all accounting data to this session,
except the first connection attempt.
This setup could be good or bad, I am not sure. My initial thoughts were,
all traffic passes through HAProxy. In retrospect, HAProxy is not being
backend node over a public IP ( of the backend node). But, what I don't
Hi Imam.
haproxy -vv
apache httpd version
shibboleth version
Client -> haproxy -> apache httpd -> shibboleth ?
Post by Imam ToufiqueHi Igor,
Thanks very much for offering to help! I will do this in sections,
hopefully, I
Post by Imam Toufiquecan keep this from being too cluttered.
--------------------------------------------------------------------------------------
Post by Imam Toufiqueglobal
#log /dev/log local0 debug
#log /dev/log local1 debug
log 127.0.0.1 local2
chroot /var/lib/haproxy
stats timeout 30s
user haproxy
group haproxy
tune.ssl.default-dh-param 2048
daemon
defaults
log global
mode http
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
timeout tunnel 9h
option tcp-check
frontend http_front
bind :80
bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
stats uri /haproxy?stats
default_backend web1_cluster
option httplog
log global
#option dontlognull
log /dev/log local0 debug
The 2 log entries are redundant, imho.
Is set at default block. Please take a look at
https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
Post by Imam Toufiqueoption forwardfor # forward IP
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
I personally would use here also set-header instead of add.
Post by Imam Toufiqueredirect scheme https if !{ ssl_fc }
acl host_web2 hdr(host) -i crsplab2.oit.uci.edu/webdav
This should be only the host name.
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-hdr
and
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.6-req.hdr
I would add a addition acl.
acl host_web2 hdr(host) -i crsplab2.oit.uci.edu
acl path_web2 path_beg -i /webdav
Post by Imam Toufiqueuse_backend webdav_cluster if host_web2
an add it to the use_backend line
use_backend webdav_cluster if host_web2 path_web2
Post by Imam Toufiqueacl host_web3 path_beg /jhub
use_backend web3_cluster if host_web3
backend webdav_cluster
balance roundrobin
I would add here the following line.
cookie SRV1 insert indirect nocache
Post by Imam Toufiqueserver web1 10.1.100.156:8080 check inter 2000 cookie w1
server web2 10.1.100.160:8080 check inter 2000 cookie w2
backend web3_cluster
I would add here the following line.
cookie SRV2 insert indirect nocache
Post by Imam Toufiqueserver publicIP:443 check ssl verify none inter 2000 cookie w1
-----------------------------------------------------------------------------------------------------
Post by Imam ToufiqueNote: I have a single backend node, as it was easy to test with just one
node,
Post by Imam Toufiqueinstead of making changes to 2 nodes at a time.
in httpd.conf, only change I have made is ( the rest is a stock centos
7.5
Post by Imam Toufique-------------------------------------
ServerName 10.1.100.160:80 ( Internal IP of the backend node)
Redirect permanent /jhub https://crsplabweb1.domain.com/jhub
-------------------------------------
in my ssl.conf, where I access the jupyterhub instance running in
127.0.0.1:8000
Post by Imam Toufique<http://127.0.0.1:8000> . Also, note that the backend is running
shibboleth
Post by Imam ToufiqueSP. One of the issues I encountered is, If I did not have SSL , i was
getting a
Post by Imam Toufiquebrowser warning for not having SSL.
Can you set up shibboleth in that manner that he answers with
proxy.domain.com?
As we don't know which version is in place I send you just as a hint here
some
links.
https://wiki.shibboleth.net/confluence/display/SHIB2/SPReverseProxy
http://shibboleth.1660669.n2.nabble.com/shibboleth-sp-behind-an-HAproxy-td5960149.html
https://duckduckgo.com/?q=shibboleth+behind+haproxy
--------------------------------------------------------------------------
Post by Imam ToufiqueListen 443 https
SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog
SSLSessionCache shmcb:/run/httpd/sslcache(512000)
SSLSessionCacheTimeout 300
SSLRandomSeed startup file:/dev/urandom 256
SSLRandomSeed connect builtin
SSLCryptoDevice builtin
<VirtualHost _default_:443>
UseCanonicalName on
ServerName crsplabweb1.domain.com:443
Maybe you can change this to proxy.domain.com and add a ServerAlias
ServerName proxy.domain.com
ServerAlias crsplabweb1 crsplabweb1.domain.com
https://httpd.apache.org/docs/2.4/mod/core.html#serveralias
Post by Imam ToufiqueErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite HIGH:3DES:!aNULL:!MD5:!SEED:!IDEA
SSLCertificateFile /etc/pki/tls/certs/crsplabweb1.domain.com_cert.cer
SSLCertificateKeyFile /etc/pki/tls/certs/crsplabweb2.key
SSLCertificateChainFile
/etc/pki/tls/certs/crsplabweb1.domain.com_interm_reverse.c
Post by Imam Toufique<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
<Directory "/var/www/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
<Location /jhub>
ProxyPass http://127.0.0.1:8000/jhub
ProxyPassReverse http://127.0.0.1:8000/jhub
RequestHeader unset Accept-Encoding
ProxyPreserveHost on
AuthType shibboleth
ShibRequestSetting requireSession 1
Require shibboleth
ShibUseHeaders On
ShibBasicHijack On
RewriteEngine On
RequestHeader set X-Remote-User %{REMOTE_USER}s
</Location>
<LocationMatch
"/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
Post by Imam ToufiqueProxyPassMatch ws://127.0.0.1:8000/jhub/$1/$2$3
ProxyPassReverse ws://127.0.0.1:8000/jhub/$1/$2$3
</LocationMatch>
BrowserMatch "MSIE [2-5]" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>
----------------------------------------------------------------------------------
Regards
Aleks
Post by Imam ToufiqueOn Fri, Oct 26, 2018 at 8:34 PM Igor Cicimov <
Hi Imam,
Hi,
I came up with the following config, things seem to be working
now, for
Post by Imam Toufiquethe most part.
frontend http_front
bind :80
bind 0.0.0.0:443 ssl crt /etc/haproxy/crsplab2_1.pem
stats uri /haproxy?stats
default_backend web1_cluster
option httplog
log global
#option dontlognull
log /dev/log local0 debug
mode http
option forwardfor # forward IP
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
redirect scheme https if !{ ssl_fc }
acl host_web3 path_beg /jhub
use_backend web3_cluster if host_web3
web3_cluster
backend web3_cluster
mode http
balance source
server crsplabweb1.domain.com publicIP:443 check ssl verify
none inter 2000 cookie w1
Post by Imam ToufiqueThe above config gets me to the backend node -- where I have a
jupyterhub instance running + . Shibboleth SP running for
authentication. As I could not get shibboleth SP to work by
staying in
Post by Imam Toufiquemy private network, I had to set up a public IP for the backend
node,
Post by Imam Toufiqueget SSL certs - so shibboleth authentication could be done. I
am sure
Post by Imam Toufiquethere is a better approach to this, but I don't know what it
is. I will
Post by Imam Toufiquebe trying out SNAT to see if that will allow me to keep using my
private
Post by Imam ToufiqueIP for the backend nodes. If any of you know how to do SNAT,
please
Post by Imam Toufiquechime in, it would be worth the time/effort to try it out.
Now, the interesting thing I have noticed with the above setup
-- when I
Post by Imam Toufiqueconnect to HAProxy, let's say with https://proxy.domain.com , I
authenticate with shibboleth, and then the URL in the browser
points to
Post by Imam Toufiquethe backend node.
my proxy address: https://proxy.domain.com/jhub
after I connect to the backend, the URL turns into -
https://crsplabweb1.domain.com/jhub/tree?
...and everything works thereafter.
I tried the rewrite method that Igor has suggested before, that
did not
Post by Imam Toufiquemake any difference. But what I noticed is, after I connect, no
traffic
Post by Imam Toufiquego through the proxy anymore, my client ( i.e. laptop) connects
directly
Post by Imam Toufiqueto the backend server. Not sure if this good or bad though (?) ,
but, I
Post by Imam Toufiqueam not sure how to configure this so that I will go through a
proxy but
Post by Imam Toufiquestill be connected in the backend via a private IP and I can (
still )
Post by Imam Toufiqueauthenticate via shibboleth.
server crsplabweb1 privateIP:80 inter 2000 cookie w1
and, I set backend apache to accept connection on port 80, then
I break
Post by Imam Toufiqueshibboleth authentication.
Any inputs here?
thanks, guys!
I think it is time for you to provide the full HAP and Apache
configs so we
Post by Imam Toufiquecan see what is going on (please obfuscate any sensitive data). Also
the use
Post by Imam Toufiqueof the "cookie w1" is not clear since you are not setting it in HAP
and is
Post by Imam Toufiquekinda redundant for single backend setup.
On Thu, Oct 25, 2018 at 1:21 AM Igor Cicimov
On Thu, Oct 25, 2018 at 6:31 PM Igor Cicimov
On Thu, 25 Oct 2018 6:13 pm Imam Toufique <
so I almost got this to work, based on the situation
I am
Post by Imam Toufiquein. To elaborate just a bit, my setup involves a
shibboleth
Post by Imam ToufiqueSP that I need to authenticate my application.
Since I
Post by Imam Toufiquecan't set up the HA proxy node with shibboleth SP -
I had to
Post by Imam Toufiquewrap my application in the backend with apache so I
can pass
Post by Imam ToufiqueREMOTE_USER to the application. the application I
have is -
Post by Imam Toufiquejupyterhub and it start with its own proxy. Long
story
Post by Imam Toufiquefrontend
bind :80
bind :443 ssl crt /etc/haproxy/crsplab2_1.pem
stats uri /haproxy?stats
default_backend web1_cluster
option httplog
log global
#option dontlognull
log /dev/log local0 debug
mode http
option forwardfor # forward IP
http-request set-header X-Forwarded-Port
%[dst_port]
Post by Imam Toufiquehttp-request add-header X-Forwarded-Proto https
if { ssl_fc }
Post by Imam Toufiqueredirect scheme https if !{ ssl_fc }
acl host_web3 path_beg /jhub
use_backend web3_cluster if host_web3
backend
server web1.oit.uci.edu <http://web1.oit.uci.edu>
128.110.80.5:80 <http://128.110.80.5:80> check
this works for the most part. But I am confused
with a
Post by Imam Toufiqueproblem. when I get to my application, my backend IP
address
Post by Imam Toufiqueshows up in the browser URL.
http://128.110.80.5/jhub/user/itoufiqu/tree?
whereas, I was expecting that it would show the
original
Post by Imam Toufiquehttp://crsplab2.domain.com/jhub/user/itoufiqu/tree
<http://crsplab2.domain.com/jhub/user/itoufiqu/tree>?
(
Post by Imam Toufiquewhere crsplab2.domain.com <
http://crsplab2.domain.com> is
Post by Imam Toufiquethe URL to get HAproxy )
You need to tell your backend app that it runs behind
reverse
Post by Imam Toufiqueproxy with ssl termination and that it's domain/url
is https://crsplab2.domain.com
<http://crsplab2.domain.com/jhub/user/itoufiqu/tree>.
How you do
Post by Imam Toufiquethat depends on the backend app you are using but most
of them
Post by Imam Toufiquelike apache2, tomcat etc. have specific configs that you
can
Post by Imam Toufiquefind in their documentation. For example if your backend
is
Post by Imam Toufiqueapache2 I bet you don't have the DomainName set in the
config in
Post by Imam Toufiquewhich case it defaults to the host ip address.
rspirep ^Location:\ http://(.*):80(.*) Location:\
https://crsplab2.domain.com
<http://crsplab2.domain.com/jhub/user/itoufiqu/tree>:443\2
if {
Post by Imam Toufiquessl_fc }
to fix the URL but note that this will not save you from
hard coded
Post by Imam Toufiqueurl's in the returned html pages the way apache does.
While I am no expert in HA proxy world, I think this
might
Post by Imam Toufiquedue to the fact that my backend does not have SSL and
HAproxy frontend does have SSL. At this point, I
would
Post by Imam Toufiqueavoid that IP address showing up in the browser.
what is
Post by Imam Toufiquethe best way to accomplish this?
thanks for your continues help!
On Tue, Oct 23, 2018 at 8:35 AM Aleksandar Lazic
Hi.
Post by Imam ToufiqueI am looking for some help on how to write the
following apache proxypass rules
Post by Imam Toufiquein HAproxy. Not to mention I am at a bit of
loss with
Post by Imam Toufiquemy first try :-) . Here
Post by Imam ToufiqueProxyPass http://10.1.100.156:8000/jhub
ProxyPassReverse http://10.1.100.156:8000/jhub
Well ProxyPass and ProxyPassReverse do a lot of
thinks
Post by Imam Toufiquenot just rewrites, as
mentioned in the doc
https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass
https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypassreverse
"/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)">
10.1.100.156:8000/jhub/$1/$2$3
Post by Imam Toufique<http://10.1.100.156:8000/jhub/$1/$2$3>
ws://10.1.100.156:8000/jhub/$1/$2$3
<http://10.1.100.156:8000/jhub/$1/$2$3>
Post by Imam Toufique</LocationMatch>
As I am not well versed in the massive HAproxy
configuration guide, if any of
Post by Imam Toufiqueyou can give me a hand with this, I would
very much
Post by Imam Toufiqueappreciate it.
I'm also not "that" expert but I would try the
following, untested.
###
defaults
mode http
log global
#... maybe some other settings
timeout tunnel 10h
frontend https_001
#... maybe some other settings
acl websocket path_beg /jhub
#... maybe some other acls
use_backend websocket_001 if websocket
backend websocket_001
reqrep "^([^\ :]*)
/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
Post by Imam Toufique"/jhub/\1/\2\3"
# You will need to replace the first column
with the
Post by Imam Toufiqueresponse from the
# backend response
/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
Post by Imam Toufique/jhub/\1/\2\3"
# OR
# http-response replace-header Location
"/jhub/(user/[^/]*)/(api/kernels/[^/]+/channels/websocket)(.*)"
Post by Imam Toufique"/jhub/\1/\2\3"
# add some checks
server ws_01 10.1.100.156:8000
<http://10.1.100.156:8000> check
###
Here are some links which may help you also.
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-reqirep
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-rspirep
Post by Imam ToufiqueI would run haproxy in Debug mode and see how the
request pass haproxy and adopt
the config.
It would be nice when you show us the working
conf ;-)
Post by Imam ToufiqueIt would be nice to have a
http-request replace-uri <match-regex>
<replace-fmt>
Post by Imam Toufiqueto replace the reqrep.