Server IP resolution using DNS in HAProxy

[Baptiste]

hi all,

As you may have noticed already, HAProxy 1.6-dev2 version has
integrated a new feature: server IP address resolution using DNS.
Main purpose of this dev is to make HAProxy aware of a server IP
change when using environment such as AWS or docker.

Here is the current status of HAProxy and server name resolution:
- when parsing the configuration, HAProxy uses libc functions and
resolvers provided by the operating system => if the server can't be
resolved at this step, then HAProxy can't start
- in order to make DNS resolution operational at run time, health
checks must be enabled on the server. Actually, the health check
triggers name resolution
- HAProxy uses its own resolvers using the new section called "resolvers".
- HAProxy queries ALL resolvers and take the first non-error response
- a resolution is considered in error when ALL resolvers failed
(whatever the failure was)
- When a resolution is successful, HAProxy keep it for "hold valid"
period. Once "hold valid" has expired, next health check will trigger
a new DNS resolution

Documentation about it:
- http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#resolvers%20%28Server%20and%20default-server%20options%29
- http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#5.3


Now current status is briefly explained, we have a few WIP tasks we
want to discuss with the community.
We want to here feedback about additional features we have in mind.



First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.
IE, a new server directive could be "init-addr" (for inital IP
address) and would take a list of directive from 'libc', 'dns',
'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to
come...)
Currently, HAProxy works like this: "init-addr libc,dns"
A new value could be "init-addr dns"
Or "init-addr 1.2.3.4,dns"


Second, we want to log server IP changes.
For now, there are 2 ways to change a server IP address: DNS
resolution or using the stats socket command: "set server addr"
2 options:
- we setup a parameter to enable logging server IP changes, whatever
has updated the server IP
- we allow HAProxy to log server IP changes from a specific source
only. IE, log only when DNS change a server's IP


Third, we have to handle DNS response errors.
We thought about the 4 following cases:
- NX domain : all DNS servers can't resolve this host name
- response timeout : no response was received
- query refused : the DNS servers refused our query
- other : all other cases
=> For each error, we can maintain the latest good IP for a period
decided by the user.
IE, if you want to keep a server up for 5 minutes while your servers
return NX, then setup "hold nx 5m" in your resolvers section


Fourth, we need a new server state when a DNS resolution is in error.
Currently, we have 2 types of state: operational or administrative
- administrative states: ready, maint, drain
- operational states: down, failed, stopped
We have to create a new state (should be operational) which reports
that HAProxy is not able to perform a proper DNS resolution for this
server. Once in that state, the server won't be able to get new
traffic, health checks will be stopped too.
HAProxy will turn the server in this state after the hold period
described in step #3.


That's all for now.
Looking forward to read your feedback!

Baptiste

[Baptiste]

Post by Baptiste
hi all,
As you may have noticed already, HAProxy 1.6-dev2 version has
integrated a new feature: server IP address resolution using DNS.
Main purpose of this dev is to make HAProxy aware of a server IP
change when using environment such as AWS or docker.
- when parsing the configuration, HAProxy uses libc functions and
resolvers provided by the operating system => if the server can't be
resolved at this step, then HAProxy can't start
- in order to make DNS resolution operational at run time, health
checks must be enabled on the server. Actually, the health check
triggers name resolution
- HAProxy uses its own resolvers using the new section called "resolvers".
- HAProxy queries ALL resolvers and take the first non-error response
- a resolution is considered in error when ALL resolvers failed
(whatever the failure was)
- When a resolution is successful, HAProxy keep it for "hold valid"
period. Once "hold valid" has expired, next health check will trigger
a new DNS resolution
- http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#resolvers%20%28Server%20and%20default-server%20options%29
- http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#5.3
Now current status is briefly explained, we have a few WIP tasks we
want to discuss with the community.
We want to here feedback about additional features we have in mind.
First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.
IE, a new server directive could be "init-addr" (for inital IP
address) and would take a list of directive from 'libc', 'dns',
'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to
come...)
Currently, HAProxy works like this: "init-addr libc,dns"
A new value could be "init-addr dns"
Or "init-addr 1.2.3.4,dns"
Second, we want to log server IP changes.
For now, there are 2 ways to change a server IP address: DNS
resolution or using the stats socket command: "set server addr"
- we setup a parameter to enable logging server IP changes, whatever
has updated the server IP
- we allow HAProxy to log server IP changes from a specific source
only. IE, log only when DNS change a server's IP
Third, we have to handle DNS response errors.
- NX domain : all DNS servers can't resolve this host name
- response timeout : no response was received
- query refused : the DNS servers refused our query
- other : all other cases
=> For each error, we can maintain the latest good IP for a period
decided by the user.
IE, if you want to keep a server up for 5 minutes while your servers
return NX, then setup "hold nx 5m" in your resolvers section
Fourth, we need a new server state when a DNS resolution is in error.
Currently, we have 2 types of state: operational or administrative
- administrative states: ready, maint, drain
- operational states: down, failed, stopped
We have to create a new state (should be operational) which reports
that HAProxy is not able to perform a proper DNS resolution for this
server. Once in that state, the server won't be able to get new
traffic, health checks will be stopped too.
HAProxy will turn the server in this state after the hold period
described in step #3.
That's all for now.
Looking forward to read your feedback!
Baptiste

Hey everyone!

I know the message above is very long, but we really need your feedback!

An other point I want to add: do you think it would make sense to
allow updating the server hostname?
It could be useful in environment where people want to pre-configure a
farm for scalability, but server host names are not predictable
(amazon ??).

Baptiste

[Marco Corte]

Post by Baptiste
- when parsing the configuration, HAProxy uses libc functions and

Post by Baptiste
resolvers provided by the operating system => if the server can't be
resolved at this step, then HAProxy can't start

[...]

Post by Baptiste
First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.

Hi, Baptiste.

Since I am used to IP address I cannot figure out all possible
implication of the server name DNS resolution :-)

IMHO HAproxy should start in any case if the configuration is valid;
only the unresolvable items should be marked as disabled or failing or
down or whatever.
A wrong DNS entry could stop a otherwise perfectly working configuration.

Why not providing an option to start haproxy even if not all servers can
be resolved?

Your proposal of the "init-addr" could be useful for a trick: I can set
a surely unreacheable address to let haproxy start and then force/wait
for the name resolution to have a working server.

A "NX" server state would be very nice.

.marcoc

[Robin Geuze]

Hey,

I don't understand the necessity of the hold valid config option. DNS
has something that takes care of this for you called the TTL. Besides if
hold valid is shorter then the TTL it would be kind of pointless since
the resolvers you are querying won't re-resolve until the TTL expires.

Tbh I don't really see the point of configuring the resolvers in haproxy
when the OS has perfectly fine working facilities for this? What is the
benefit besides possibly causing lookups to happen twice, once from the
OS resolving stack and once from haproxies? If you really want exactly
the same behavior as described you could always configure a local
resolver that queries multiple other resolvers instead of recursing itself.

-Robin-

Post by Marco Corte

Post by Baptiste
- when parsing the configuration, HAProxy uses libc functions and

Post by Baptiste
resolvers provided by the operating system => if the server can't be
resolved at this step, then HAProxy can't start

[...]

Post by Baptiste
First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.

Hi, Baptiste.
Since I am used to IP address I cannot figure out all possible
implication of the server name DNS resolution :-)
IMHO HAproxy should start in any case if the configuration is valid;
only the unresolvable items should be marked as disabled or failing or
down or whatever.
A wrong DNS entry could stop a otherwise perfectly working configuration.
Why not providing an option to start haproxy even if not all servers
can be resolved?
Your proposal of the "init-addr" could be useful for a trick: I can
set a surely unreacheable address to let haproxy start and then
force/wait for the name resolution to have a working server.
A "NX" server state would be very nice.
.marcoc

[Nenad Merdanovic]

Hello Robin,

Post by Robin Geuze
Tbh I don't really see the point of configuring the resolvers in haproxy
when the OS has perfectly fine working facilities for this? What is the
benefit besides possibly causing lookups to happen twice, once from the
OS resolving stack and once from haproxies? If you really want exactly
the same behavior as described you could always configure a local
resolver that queries multiple other resolvers instead of recursing itself.

Because this would perfectly integrate with things like Consul
(https://www.consul.io/docs/agent/dns.html), which are currently very
widely used to provide service discovery.

Post by Robin Geuze
-Robin-

Regards,

[Robin Geuze]

Hey Nenad,

Actually a local resolver can take care of that for you as well since
every resolver I know allows configuring a different destination on
domain basis. Also as described in the first email, the server has to be
resolvable via the OS resolving stack as well otherwise haproxy won't
start. This means you cannot use custom domains without configuring some
sort of custom resolver anyway.

-Robin-

Post by Nenad Merdanovic
Hello Robin,

Post by Robin Geuze
Tbh I don't really see the point of configuring the resolvers in haproxy
when the OS has perfectly fine working facilities for this? What is the
benefit besides possibly causing lookups to happen twice, once from the
OS resolving stack and once from haproxies? If you really want exactly
the same behavior as described you could always configure a local
resolver that queries multiple other resolvers instead of recursing itself.

Because this would perfectly integrate with things like Consul
(https://www.consul.io/docs/agent/dns.html), which are currently very
widely used to provide service discovery.

Post by Robin Geuze
-Robin-

Regards,

[Baptiste]

Actually a local resolver can take care of that for you as well since every
resolver I know allows configuring a different destination on domain basis.
Also as described in the first email, the server has to be resolvable via
the OS resolving stack as well otherwise haproxy won't start.

That's the purpose of this thread.
We want / need to get rid of this limitation and that's why we ask our
community if the way we wanted to fix it makes sense.

This means you
cannot use custom domains without configuring some sort of custom resolver
anyway.

HAProxy's internal resolver can be made flexible enough for this
purpose without being intrusive in the underlying operating system.

Baptiste

-Robin-

Post by Nenad Merdanovic
Hello Robin,

Post by Robin Geuze
Tbh I don't really see the point of configuring the resolvers in haproxy
when the OS has perfectly fine working facilities for this? What is the
benefit besides possibly causing lookups to happen twice, once from the
OS resolving stack and once from haproxies? If you really want exactly
the same behavior as described you could always configure a local
resolver that queries multiple other resolvers instead of recursing itself.

Because this would perfectly integrate with things like Consul
(https://www.consul.io/docs/agent/dns.html), which are currently very
widely used to provide service discovery.

Post by Robin Geuze
-Robin-

Regards,

[Baptiste]

Hi Robin,

I don't understand the necessity of the hold valid config option. DNS has
something that takes care of this for you called the TTL. Besides if hold
valid is shorter then the TTL it would be kind of pointless since the
resolvers you are querying won't re-resolve until the TTL expires.

Your server won't wait until end of TTL to fail ;)
So you don't want to followup TTLs and prefer force HAProxy to resolve
more often.
In some cases, you don't choose the TTL (amazon), so 'hold valid'
allows you to choose your own TTL.

Tbh I don't really see the point of configuring the resolvers in haproxy
when the OS has perfectly fine working facilities for this?

Imagine a big company. Imagine the ops team managing HAProxy and the
IT team managing the DNS servers.
(It's a real case)
When the ops team start up a new server, DNS propagation can be long
(several minutes) before the DNS servers managed by the IT team are
aware of the update (we speak about worldwide deployment).
In order to start up the new service asap, then the ops team want to
use the regular DNS servers and their own DNS server...

There are many cases like this one, where the ops team doesn't have
the hand over the DNS server.
Same if you use a service discovery, then HAProxy can point its DNS
request to it instead of regular DNS servers.

What is the
benefit besides possibly causing lookups to happen twice, once from the OS
resolving stack and once from haproxies? If you really want exactly the same
behavior as described you could always configure a local resolver that
queries multiple other resolvers instead of recursing itself.

you say this because you have the hand over your OS.
We have many customers and community users where it's not the case.

Once again, HAProxy, is a load-balancer, it needs the most accurate
information and as fast as possible.
You don't want to tune your local bind or powerdns just for HAProxy
and prevent any other service to operate as usual.

Baptiste

[Baptiste]

Post by Baptiste
- when parsing the configuration, HAProxy uses libc functions and

Post by Baptiste
resolvers provided by the operating system => if the server can't be
resolved at this step, then HAProxy can't start

[...]

Post by Baptiste
First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.

Hi Marco,

Why not providing an option to start haproxy even if not all servers can be
resolved?

That's the purpose of my mail. I need this feature, but I want to make
it in a way which would satisfy the community.

Your proposal of the "init-addr" could be useful for a trick: I can set a
surely unreacheable address to let haproxy start and then force/wait for the
name resolution to have a working server.

That's what we want.
The hidden feature, is that you can start large farms and simply turn
on DNS when spawning new application servers.
Scale up without reloading HAProxy ;)

A "NX" server state would be very nice.

noted, thx

Baptiste

[Conrad Hoffmann]

Hello,

<snip>
I know the message above is very long, but we really need your feedback!
</snip>

First, many thanks for tackling this! It surely makes many peoples live
much easier. Reading this makes me want two things, one of them being a
little "not-haproxy-like" maybe.
First would be resolution of SRV records and actually using the port
supplied by the SRV record as the port for the server. I looked at the code
and it doesn't seem like too much work, most of it would probably be
changing the config stuff accordingly.
The other one is... well you asked for it ;) so here it goes: it would be
great to express in the config something like "resolve this name and use up
to X servers from the reply". The backend then allocates X servers.
Assuming that the initial lookup returns Y results, the (sorted) records
get assinged to the first Y servers, the other X-Y servers get marked as
down. Upon a new lookup, same procedure for a potentially changing value of Y.
I realize this a pretty bold feature request for several reasons, but I
have actually spent some thought on it think it might be doable without
really violating any of HAProxy's design paradigms. I would also be willing
to invest some time (code) into this myself.
If you think this might be at least worth a discussion, I'd be happy to
share some more detailed thoughts and it would be great to hear your
thoughts on that, too.

Cheers,
Conrad

[Baptiste]

Post by Conrad Hoffmann
First would be resolution of SRV records and actually using the port
supplied by the SRV record as the port for the server. I looked at the code
and it doesn't seem like too much work, most of it would probably be
changing the config stuff accordingly.

You're right, this could be an interesting option.
Actually, we though about the SRV records to populate a full backend
server list with IP, name, port, weight using a single DNS query.

Post by Conrad Hoffmann
The other one is... well you asked for it ;) so here it goes: it would be
great to express in the config something like "resolve this name and use up
to X servers from the reply". The backend then allocates X servers.
Assuming that the initial lookup returns Y results, the (sorted) records
get assinged to the first Y servers, the other X-Y servers get marked as
down. Upon a new lookup, same procedure for a potentially changing value of Y.
I realize this a pretty bold feature request for several reasons, but I
have actually spent some thought on it think it might be doable without
really violating any of HAProxy's design paradigms. I would also be willing
to invest some time (code) into this myself.
If you think this might be at least worth a discussion, I'd be happy to
share some more detailed thoughts and it would be great to hear your
thoughts on that, too.

First, there are some limitations about creating servers on the fly in
a backend.
So instead, we propose you to pre-allocate servers by configuration
and then wait for the DNS to populate it.
I don't speak about a request per server, I speak here about one
request for the whole farm :)

You go one step further than the design we have about SRV records to
populate the backend.
We thought using priority to decide whether a server is active or backup.
The advantage is that you don't need to reload HAProxy to change your X value ;)

I would welcome a contribution about SRV record type.
That said, before this, I have to rewrite part of the response parser
to store the response in a real DNS packet structure instead of
keeping data in a buffer.

Baptiste

[Conrad Hoffmann]

Post by Baptiste

Post by Conrad Hoffmann
First would be resolution of SRV records and actually using the port
supplied by the SRV record as the port for the server. I looked at the code
and it doesn't seem like too much work, most of it would probably be
changing the config stuff accordingly.

You're right, this could be an interesting option.
Actually, we though about the SRV records to populate a full backend
server list with IP, name, port, weight using a single DNS query.

Awesome! That's exactly what I want :)
It's what I tried to express in the second part of my request, probably in
a slightly weird way.

Post by Baptiste

Post by Conrad Hoffmann
The other one is... well you asked for it ;) so here it goes: it would be
great to express in the config something like "resolve this name and use up
to X servers from the reply". The backend then allocates X servers.
Assuming that the initial lookup returns Y results, the (sorted) records
get assinged to the first Y servers, the other X-Y servers get marked as
down. Upon a new lookup, same procedure for a potentially changing value of Y.
I realize this a pretty bold feature request for several reasons, but I
have actually spent some thought on it think it might be doable without
really violating any of HAProxy's design paradigms. I would also be willing
to invest some time (code) into this myself.
If you think this might be at least worth a discussion, I'd be happy to
share some more detailed thoughts and it would be great to hear your
thoughts on that, too.

First, there are some limitations about creating servers on the fly in
a backend.
So instead, we propose you to pre-allocate servers by configuration
and then wait for the DNS to populate it.

It's kind of what I meant, although I had something in mind where you say
once "preallocate 20 servers for this backend" instead of listing 20
servers with the same hostname. But that's syntactic sugar and I would
happily use the solution listing a server for each desired allocation.

Post by Baptiste
I don't speak about a request per server, I speak here about one
request for the whole farm :)

Again, awesome! That's what I'm looking for!

Post by Baptiste
You go one step further than the design we have about SRV records to
populate the backend.
We thought using priority to decide whether a server is active or backup.
The advantage is that you don't need to reload HAProxy to change your X value ;)

Yes, perfect. And, what I tried to express above, HAProxy should mark some
servers down if the number records received is less than the number of
servers configured.

Post by Baptiste
I would welcome a contribution about SRV record type.
That said, before this, I have to rewrite part of the response parser
to store the response in a real DNS packet structure instead of
keeping data in a buffer.

Sounds good. I am not sure what you have in mind there or what parts of the
code would be touched by that, so if you let me know when it's done I'll
happily send some patches as a base for further discussion!

Thanks, Baptiste, this get's me even more excited about HAProxy than I am
already :)

Cheers,
Conrad

[Cyril Bonté]

Hi Baptiste,

late feedback but it's better than none ;-)

Post by Baptiste
First, we want to fix the error when HAProxy fails starting up because
the resolvers pointed by the system can't resolve a server's IP
address (but HAProxy resolvers could).

It is pretty awaited from some teams at work. I'll add that even if
HAProxy resolvers can't resolve the entries or are not configured, we
should allow to make it non fatal and leave such server entries
definitely DOWN with healthchecks disabled (except if it is updated from
the CLI).

I see that Marco Corte has provided the same feedback ;-)

Post by Baptiste
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.
IE, a new server directive could be "init-addr" (for inital IP
address) and would take a list of directive from 'libc', 'dns',
'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to
come...)
Currently, HAProxy works like this: "init-addr libc,dns"
A new value could be "init-addr dns"
Or "init-addr 1.2.3.4,dns"

To be clear, does it mean that we could meet such configuration ?
server server1 host1.somewhere:80 init-addr 127.0.0.1,libc,dns

Which would mean :
- Use the init IP first
- At start up, try to resolve whith the libc resolver
- At runtime, try to resolve with HAProxy resolvers.

Then, maybe we'll have to prevent such (un)funny things :
server server1 192.0.2.1:80 init-addr 127.0.0.1,libc,dns

Post by Baptiste
Second, we want to log server IP changes.
For now, there are 2 ways to change a server IP address: DNS
resolution or using the stats socket command: "set server addr"
- we setup a parameter to enable logging server IP changes, whatever
has updated the server IP
- we allow HAProxy to log server IP changes from a specific source
only. IE, log only when DNS change a server's IP

Why not something similar to init-addr ? like "log-resolve-update
libc,dns" or "log-resolve-update dns" to prevent logging dns resolution
during the init step (which will keep the current behaviour, to prevent
flooding the logs when haproxy starts with a lot of servers declared).

About the stats socket, I think it's important to always log server
changes even it nothing is set in the configuration.

Post by Baptiste
Third, we have to handle DNS response errors.
- NX domain : all DNS servers can't resolve this host name
- response timeout : no response was received
- query refused : the DNS servers refused our query
- other : all other cases
=> For each error, we can maintain the latest good IP for a period
decided by the user.
IE, if you want to keep a server up for 5 minutes while your servers
return NX, then setup "hold nx 5m" in your resolvers section
Fourth, we need a new server state when a DNS resolution is in error.
Currently, we have 2 types of state: operational or administrative
- administrative states: ready, maint, drain
- operational states: down, failed, stopped
We have to create a new state (should be operational) which reports
that HAProxy is not able to perform a proper DNS resolution for this
server. Once in that state, the server won't be able to get new
traffic, health checks will be stopped too.
HAProxy will turn the server in this state after the hold period
described in step #3.

When servers fall to this state, maybe we'll have to define how haproxy
will react with some configurations. For example, what's the effect of
"force-persist" on a server in this state ? I'd say it's not like a DOWN
server and it shouldn't persist, but we'll have to be clear on such use
cases.

[Willy Tarreau]

Hi Cyril, Baptiste,

Just a quick response on this part to clear up some confusion.

Post by Cyril Bonté

Post by Baptiste
The idea here would to create a new flag on the server to tell HAProxy
which IP to use. The server would be enabled when the IP has been
provided by the expected tool.
IE, a new server directive could be "init-addr" (for inital IP
address) and would take a list of directive from 'libc', 'dns',
'a.b.c.d' (arbitrary IP address), etc... (non exhaustive live, more to
come...)
Currently, HAProxy works like this: "init-addr libc,dns"
A new value could be "init-addr dns"
Or "init-addr 1.2.3.4,dns"

In fact not, right now it's only "init-addr libc" since we cannot do DNS
during init time.

Post by Cyril Bonté
To be clear, does it mean that we could meet such configuration ?
server server1 host1.somewhere:80 init-addr 127.0.0.1,libc,dns
- Use the init IP first
- At start up, try to resolve whith the libc resolver
- At runtime, try to resolve with HAProxy resolvers.

The purpose of init-addr was not this in fact but to declare how to
resolve the initial address. Ultimately if Baptiste's DNS code can
work out of the poll loop, we could consider using it for the very
first resolution. In your case, the idea would be to have then :

init-addr dns,libc,127.0.0.1

or :

init-addr dns,libc,none

This means :
- try to resolve the name using DNS ;
- if it fails, try again using libc ;
- if it fails, use 127.0.0.1 or don't set any address

This last option (hard-coded value or "none") allows the server to
start eventhough the resolution failed. We wanted to support an extra
keyword such as "last" when haproxy is restarted and a dump file is
present to provide the last IP address. It will solve the different
needs of different people who want to set some preferences on config
or on last dump, or on dns, etc...

Post by Cyril Bonté
server server1 192.0.2.1:80 init-addr 127.0.0.1,libc,dns

It's not absurd because this address is also the one used by the checks
by default and some people might want it to continue to work this way.

Regards,
Willy