HAProxy and Client Certificates for Admin site access

[Matthew Sanders]

NOTE: *I am not subscribed to the mailing list, would you be able to CC me
in the responses? Thanks!*

I have started integrating HAProxy as the entry point to all of my server
infrastructure projects.

I wanted to get the community's thoughts on how best to handle Client
Certificates for secure portions of a site (preferably on a single
machine). I wanted to make this as optimal as possible, but it seems like
in order to do this with HAProxy currently I need to make a few sacrifices.

I may be wrong, but I would imagine the primary use case for Client
Certificates would be to secure a site (or in many, if not most cases, a
section of a site like admin.example.comor example.com/admin, etc.). The
full shotgun approach works well as it is... but even with 'verify
optional' some users may have a degraded experience even if they are not
using the secure portion of the site. They may be prompted for certificate
by the browser for example.

I ran into a few work arounds to the problem, but I fear there is a few
performance considerations with these approaches and felt there must be a
more native way HAProxy could help with this situation.

In this blog post:
https://www.loadbalancer.org/blog/client-certificate-authentication-with-haproxy/

The author references an answer from another community member who suggests
the first approach here:
https://discourse.haproxy.org/t/how-to-set-ssl-verify-client-for-specific-domain-name/1489/3

The second (but pretty much the same) approach I found was on Stack
Exchange here: https://serverfault.com/a/663923

I prefer the second approach as it doesn't use the abstract namespace which
as the documentation points out are not 'rebindable' with multi-process
soft-restart.

My concerns are:

1. This relies on SNI which will add some delay up front 'tcp-request
inspect-delay 5s'. This happens even if I plan to terminate the TLS
connection at HAProxy.
2. Then we route to a backend internally to then route to another loopback
causing some more minor delay.
3. The internal routing would then increase the unix open file count per
request

Is there any other solutions that may avoid these issues, or am I
needlessly worried about it in the first place? Can HAProxy be improved to
more natively support such a workflow?

I know I could create a second machine to reserve for the secure routes. I
also could utilize a stage environment (still another machine) and allow
the admin site to allow the user to select the environments to work against
(example operations would be migrating user/project data, making custom
queries for management, etc.)

Thank you so much for your time,
Matt

[Lukas Tribus]

Hi Matt,

I ran into a few work arounds to the problem, but I fear there is a few performance considerations with these
approaches and felt there must be a more native way HAProxy could help with this situation.
https://www.loadbalancer.org/blog/client-certificate-authentication-with-haproxy/
The author references an answer from another community member who suggests the
first approach here: https://discourse.haproxy.org/t/how-to-set-ssl-verify-client-for-specific-domain-name/1489/3
https://serverfault.com/a/663923
I prefer the second approach as it doesn't use the abstract namespace which
as the documentation points out are not 'rebindable' with multi-process soft-restart.

Right, whatever works best for you. IP loopback through 127/8, unix
domain sockets or abstract namespace have each it's own advantages and
disadvantages. I'd assume that unix domain sockets would be ideal for
your use-case (just use the correct user/group settings if you drop
privileges).

1. This relies on SNI which will add some delay up front 'tcp-request inspect-delay 5s'. This happens even if I plan to terminate the TLS connection at HAProxy.

No, that's not the case. 5 seconds is the time when haproxy gives up
waiting for a TLS client_hello on this connection, but it doesn't mean
you suffer from this delay for regular connections. The moment the
hello is there, the hello is analyzed and forwarded.

2. Then we route to a backend internally to then route to another loopback causing some more minor delay.
3. The internal routing would then increase the unix open file count per request
Is there any other solutions that may avoid these issues, or am I needlessly worried about it in the first place?

Whether this is something that critically affects your product or not
is something you'll have to benchmark. If you are netflix and trying
to squeeze 100gbit/s out of a single edge-cluster, those things will
matter. If you are forwarding a few hundred mbit/s of traffic and a
few thousand concurrent connections with any half-decent HW, it
probably doesn't.

Can HAProxy be improved to more natively support such a workflow?

Actually I believe we already have a way. You can set TLS options per
certificate in Haproxy 1.8, including the verify option. You can do
that with crt-list:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-crt-list

So instead of referring to a certificate you just refer to a crt-list,
which then contains a list of certificates with specific
configurations:

frontend httpsin
bind :443 ssl crt-list /etc/haproxy/crtlist

$ cat /etc/haproxy/crtlist
www.example.org [verify none]
admin.example.org [verify required]


Of course that means using dedicated certificates (and also, they
should not overlap - so no wildcard certificate on the main site and a
more specific one for admin area - otherwise you the browser may use
the wildcard cert connection for both).


Regards,
Lukas

[Matthew Sanders]

Awesome Lukas THANK YOU!! You were obviously the member I was referring
to in the first solution. :)

I ran into a discussion on another mailing list about crt-list, but I
wasn't sure how it would solve the problem as I didn't see a mention of
applying verify rules separately.

I think that sounds like another option for my use case.

Basically I run several cloud instances for different clients and some want
to recycle as many resources as possible until they are ready to scale
out. This would allow me to make a more bare bone solution stretch as far
as I can.

Much appreciated!
-Matt

Post by Lukas Tribus
Hi Matt,

Post by Matthew Sanders
I ran into a few work arounds to the problem, but I fear there is a few

performance considerations with these

Post by Matthew Sanders
approaches and felt there must be a more native way HAProxy could help

with this situation.
https://www.loadbalancer.org/blog/client-certificate-authentication-with-haproxy/

Post by Matthew Sanders
The author references an answer from another community member who

suggests the
https://discourse.haproxy.org/t/how-to-set-ssl-verify-client-for-specific-domain-name/1489/3

Post by Matthew Sanders
The second (but pretty much the same) approach I found was on Stack
https://serverfault.com/a/663923
I prefer the second approach as it doesn't use the abstract namespace

which

Post by Matthew Sanders
as the documentation points out are not 'rebindable' with multi-process

soft-restart.
Right, whatever works best for you. IP loopback through 127/8, unix
domain sockets or abstract namespace have each it's own advantages and
disadvantages. I'd assume that unix domain sockets would be ideal for
your use-case (just use the correct user/group settings if you drop
privileges).

Post by Matthew Sanders
1. This relies on SNI which will add some delay up front 'tcp-request

inspect-delay 5s'. This happens even if I plan to terminate the TLS
connection at HAProxy.
No, that's not the case. 5 seconds is the time when haproxy gives up
waiting for a TLS client_hello on this connection, but it doesn't mean
you suffer from this delay for regular connections. The moment the
hello is there, the hello is analyzed and forwarded.

Post by Matthew Sanders
2. Then we route to a backend internally to then route to another

loopback causing some more minor delay.

Post by Matthew Sanders
3. The internal routing would then increase the unix open file count per

request

Post by Matthew Sanders
Is there any other solutions that may avoid these issues, or am I

needlessly worried about it in the first place?
Whether this is something that critically affects your product or not
is something you'll have to benchmark. If you are netflix and trying
to squeeze 100gbit/s out of a single edge-cluster, those things will
matter. If you are forwarding a few hundred mbit/s of traffic and a
few thousand concurrent connections with any half-decent HW, it
probably doesn't.

Post by Matthew Sanders
Can HAProxy be improved to more natively support such a workflow?

Actually I believe we already have a way. You can set TLS options per
certificate in Haproxy 1.8, including the verify option. You can do
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.1-crt-list
So instead of referring to a certificate you just refer to a crt-list,
which then contains a list of certificates with specific
frontend httpsin
bind :443 ssl crt-list /etc/haproxy/crtlist
$ cat /etc/haproxy/crtlist
www.example.org [verify none]
admin.example.org [verify required]
Of course that means using dedicated certificates (and also, they
should not overlap - so no wildcard certificate on the main site and a
more specific one for admin area - otherwise you the browser may use
the wildcard cert connection for both).
Regards,
Lukas