Haproxy failover: DNS RR vs Virtual IP (heartbeat, keepalived)

[David]

Hi,

Let's say I have an architecture where a couple of servers are put
behind a haproxy instance for load balancing, to serve content at
www.example.com. For reliability/availability purpose, I want to have
more than one haproxy instance to ensure continuous servicing when one
haproxy fails:

LB1 LB2
| |
------------------------------------
| | |
Server1 Server2 Server3

The issue is how to "distribute" the load across both load balancers. I
am aware of at least two solutions:

- DNS Round Robin: www.example.com is resolved to both LB1 and LB2's
IP. If e.g. LB1 crashes, clients will then look at the next entry, LB2
in this case
- High Availability IP (heartbeat, keepalive) between both load
balancers. Only one load balancer is proxying all the requests at a time
(I assume one load balancer has enough power to serve all our traffic).

I have been asked to implement the DNS RR method, but from what I have
read, method 2 is the one most commonly used. What are the pros/cons of
each ?

cheers,

David

[Willy Tarreau]

Hi David,

Post by David
Hi,
Let's say I have an architecture where a couple of servers are put
behind a haproxy instance for load balancing, to serve content at
www.example.com. For reliability/availability purpose, I want to have
more than one haproxy instance to ensure continuous servicing when one
LB1 LB2
| |
------------------------------------
| | |
Server1 Server2 Server3
The issue is how to "distribute" the load across both load balancers. I
- DNS Round Robin: www.example.com is resolved to both LB1 and LB2's
IP. If e.g. LB1 crashes, clients will then look at the next entry, LB2
in this case
- High Availability IP (heartbeat, keepalive) between both load
balancers. Only one load balancer is proxying all the requests at a time
(I assume one load balancer has enough power to serve all our traffic).
I have been asked to implement the DNS RR method, but from what I have
read, method 2 is the one most commonly used. What are the pros/cons of
each ?

The first one is just pure theory. You may want to test it by yourself
to conclude that it simply does not work at all. Most clients will see
a connection error or timeout, and few of them will be able to perform
a retry on the other address but after some delay which will cause some
unpleasant experience. Also, most often the browser does not perform a
new lookup if the first one has already worked. That means that until
the browser is closed, the visitor will remain bound to the same IP.

Then you might think that it's enough to update the DNS entries upon
failure, but that does not propagate quickly, as there are caches
everywhere. To give you an idea, the haproxy ML and site were moved to
a new server one month ago, and we're still receiving a few requests a
minute on the old server. In general you can count on 1-5% of the visitors
to cache an entry more than one week. This is not a problem for a disaster
recovery, but it certainly is for a server failover because that means you
cannot put it offline at all.

High availability has the big advantage of always exposing a working
service for the same IP address, so it's a *lot* more reliable and
transparent to users. There are two common ways to provide HA under
Linux :
- heartbeat
- keepalived

The first one is more suited to data servers, as it ensures that no more
than one node is up at a time. This is critical when you share file systems.
The second one is more suited to stateless servers such as proxies and load
balancers, as it ensures that no less than one node is up at a time. Sadly
people generally confuse them and sometimes use keepalived for NFS servers
or use heartbeat with haproxy...

High availability presents a small inconvenient though : the backup node
is never used so you don't really know if it works well, and there is a big
temptation not to update it as often as the master node. This is also an
advantage in that it allows you to validate your new configs on it before
loading them on the master node. If you want to use both LBs at the same
time, the solution is to have two crossed VIPs on your LBs and use DNS RR
to ensure that both are used. When one LB fails, the VIP moves to the other
one.

If you stick to the following principles, you should never encounter issues :
- DNS = load balancing, no availability at all
- HA = availability, no load balancing at all.
=> use DNS to announce always available IP addresses

Cheers,
Willy

[Angelo Höngens]

(sorry for top posting, damn Outlook)

Just to second Willy's story, this is how a lot of people do it, including us. We use pacemaker for high availability, and dnr rr for loadbalancing.

For example we have a 4-node cluster running varnish and haproxy. In this case I have 4 virtual ipv4-addresses and 4 virtual ipv6 addresses on the cluster. We use pacemaker to keep the virtual ip's up, and we use dns round-robin to balance the load. We get nice equal load balancing this way, and if a node is down (or I want to do maintenance), the vip's move to other nodes, and they take the extra load.
--
With kind regards,


Angelo Höngens

Systems Administrator

------------------------------------------
NetMatch
tourism internet software solutions

Ringbaan Oost 2b
5013 CA Tilburg
T: [phone]
F: [phone]

mailto:A.Hongens-isB3h8G6/***@public.gmane.org
http://www.netmatch.nl
------------------------------------------

-----Original Message-----
Sent: donderdag 6 januari 2011 8:20
To: David
Subject: Re: Haproxy failover: DNS RR vs Virtual IP (heartbeat,
keepalived)
Hi David,

Post by David
Hi,
Let's say I have an architecture where a couple of servers are put
behind a haproxy instance for load balancing, to serve content at
www.example.com. For reliability/availability purpose, I want to have
more than one haproxy instance to ensure continuous servicing when

one

Post by David
LB1 LB2
| |
------------------------------------
| | |
Server1 Server2 Server3
The issue is how to "distribute" the load across both load balancers.

I

Post by David
- DNS Round Robin: www.example.com is resolved to both LB1 and

LB2's

Post by David
IP. If e.g. LB1 crashes, clients will then look at the next entry,

LB2

Post by David
in this case
- High Availability IP (heartbeat, keepalive) between both load
balancers. Only one load balancer is proxying all the requests at a

time

Post by David
(I assume one load balancer has enough power to serve all our

traffic).

Post by David
I have been asked to implement the DNS RR method, but from what I

have

Post by David
read, method 2 is the one most commonly used. What are the pros/cons

of

Post by David
each ?

The first one is just pure theory. You may want to test it by yourself
to conclude that it simply does not work at all. Most clients will see
a connection error or timeout, and few of them will be able to perform
a retry on the other address but after some delay which will cause some
unpleasant experience. Also, most often the browser does not perform a
new lookup if the first one has already worked. That means that until
the browser is closed, the visitor will remain bound to the same IP.
Then you might think that it's enough to update the DNS entries upon
failure, but that does not propagate quickly, as there are caches
everywhere. To give you an idea, the haproxy ML and site were moved to
a new server one month ago, and we're still receiving a few requests a
minute on the old server. In general you can count on 1-5% of the visitors
to cache an entry more than one week. This is not a problem for a disaster
recovery, but it certainly is for a server failover because that means you
cannot put it offline at all.
High availability has the big advantage of always exposing a working
service for the same IP address, so it's a *lot* more reliable and
transparent to users. There are two common ways to provide HA under
- heartbeat
- keepalived
The first one is more suited to data servers, as it ensures that no more
than one node is up at a time. This is critical when you share file systems.
The second one is more suited to stateless servers such as proxies and load
balancers, as it ensures that no less than one node is up at a time. Sadly
people generally confuse them and sometimes use keepalived for NFS servers
or use heartbeat with haproxy...
High availability presents a small inconvenient though : the backup node
is never used so you don't really know if it works well, and there is a big
temptation not to update it as often as the master node. This is also an
advantage in that it allows you to validate your new configs on it before
loading them on the master node. If you want to use both LBs at the same
time, the solution is to have two crossed VIPs on your LBs and use DNS RR
to ensure that both are used. When one LB fails, the VIP moves to the other
one.
- DNS = load balancing, no availability at all
- HA = availability, no load balancing at all.
=> use DNS to announce always available IP addresses
Cheers,
Willy

[David]

Post by Angelo Höngens
(sorry for top posting, damn Outlook)
Just to second Willy's story, this is how a lot of people do it, including us. We use pacemaker for high availability, and dnr rr for loadbalancing.
For example we have a 4-node cluster running varnish and haproxy. In this case I have 4 virtual ipv4-addresses and 4 virtual ipv6 addresses on the cluster. We use pacemaker to keep the virtual ip's up, and we use dns round-robin to balance the load. We get nice equal load balancing this way, and if a node is down (or I want to do maintenance), the vip's move to other nodes, and they take the extra load.

Thanks for the information. Both Willy and you refer to DNS RR as a load
balancing solution, but I don't really understand that point: if
caching, etc... means hostname->load balancer resolution is random, the
load balancing will likely be very unbalanced, no ?

cheers,

David

[Angelo Höngens]

Round robin is not the same as random.

If a host name has 4 A records, then most DNS servers (if not all) will return it round-robin. So first a.a.a.1, then a.a.a.2, then a.a.a.1, then a.a.a.2, etc.. Of course there are multiple dns servers involved and thousands of clients over the world, but in the end we almost always get a perfect 25-25-25-25 balancing.
--
With kind regards,


Angelo Höngens

Systems Administrator

------------------------------------------
NetMatch
tourism internet software solutions

Ringbaan Oost 2b
5013 CA Tilburg
T: [phone]
F: [phone]

mailto:A.Hongens-isB3h8G6/***@public.gmane.org
http://www.netmatch.nl
------------------------------------------

-----Original Message-----
Sent: donderdag 6 januari 2011 9:45
Subject: Re: Haproxy failover: DNS RR vs Virtual IP (heartbeat,
keepalived)

Post by Angelo Höngens
(sorry for top posting, damn Outlook)
Just to second Willy's story, this is how a lot of people do it,

including us. We use pacemaker for high availability, and dnr rr for
loadbalancing.

Post by Angelo Höngens
For example we have a 4-node cluster running varnish and haproxy. In

this case I have 4 virtual ipv4-addresses and 4 virtual ipv6 addresses
on the cluster. We use pacemaker to keep the virtual ip's up, and we
use dns round-robin to balance the load. We get nice equal load
balancing this way, and if a node is down (or I want to do
maintenance), the vip's move to other nodes, and they take the extra
load.
Thanks for the information. Both Willy and you refer to DNS RR as a load
balancing solution, but I don't really understand that point: if
caching, etc... means hostname->load balancer resolution is random, the
load balancing will likely be very unbalanced, no ?
cheers,
David

[David]

Post by Angelo Höngens
Round robin is not the same as random.

Sure, but my understanding was that between caches and inconsistent
usage of the list returned by RR (sometimes shuffled), the behavior in
practice was far from being perfect round robin.

Post by Angelo Höngens
If a host name has 4 A records, then most DNS servers (if not all) will return it round-robin. So first a.a.a.1, then a.a.a.2, then a.a.a.1, then a.a.a.2, etc.. Of course there are multiple dns servers involved and thousands of clients over the world, but in the end we almost always get a perfect 25-25-25-25 balancing.

Hm, interesting, I have seen reports of different behaviors. What kind
of configurations may interfere with the ideal, near uniform balancing ?

thanks,

David

[Angelo Höngens]

Post by David

Post by Angelo Höngens
If a host name has 4 A records, then most DNS servers (if not all)
will return it round-robin. So first a.a.a.1, then a.a.a.2, then
a.a.a.1, then a.a.a.2, etc.. Of course there are multiple dns servers
involved and thousands of clients over the world, but in the end we
almost always get a perfect 25-25-25-25 balancing.

Hm, interesting, I have seen reports of different behaviors. What kind
of configurations may interfere with the ideal, near uniform balancing ?

Dunno.. what I do know is that the more users you get, the better the
balancing :)

Here you see a request graph of one of my clusters:

Loading Image...

As you see the distribution is not perfect, but for me it's enough.
--
With kind regards,


Angelo Höngens
systems administrator

MCSE on Windows 2003
MCSE on Windows 2000
MS Small Business Specialist
------------------------------------------
NetMatch
tourism internet software solutions

Ringbaan Oost 2b
5013 CA Tilburg
[phone]
[phone]

A.Hongens-isB3h8G6/***@public.gmane.org
www.netmatch.nl
------------------------------------------

[David]

Post by Angelo Höngens

Post by David

Post by Angelo Höngens
If a host name has 4 A records, then most DNS servers (if not all)
will return it round-robin. So first a.a.a.1, then a.a.a.2, then
a.a.a.1, then a.a.a.2, etc.. Of course there are multiple dns servers
involved and thousands of clients over the world, but in the end we
almost always get a perfect 25-25-25-25 balancing.

Hm, interesting, I have seen reports of different behaviors. What kind
of configurations may interfere with the ideal, near uniform balancing ?

Dunno.. what I do know is that the more users you get, the better the
balancing :)

That seems intuitive :)

Post by Angelo Höngens
http://files.hongens.nl/2011/01/06/varnishrequests.png
As you see the distribution is not perfect, but for me it's enough.

That looks indeed much better than what I would have expected following
what I read. Thanks a lot for the detailed info,

David

[Willy Tarreau]

Post by David

Post by Angelo Höngens
http://files.hongens.nl/2011/01/06/varnishrequests.png
As you see the distribution is not perfect, but for me it's enough.

That looks indeed much better than what I would have expected following
what I read. Thanks a lot for the detailed info,

You can get a fairly imbalanced distribution if you add/remove IPs at one
point. But after a few days, you start to get something looking smooth again.

Willy

[Anze]

DNS RR offers at least some measure of protection in case the whole data
center loses its connection(s) to the world, which happens from time to time.
Of course, if you have another data center to fall to. ;)

So ideally you would have keepalived-like solution inside each data center,
but DNS RR if you deploy your solution to multiple data centers.

Anze

Post by Angelo Höngens
Round robin is not the same as random.
If a host name has 4 A records, then most DNS servers (if not all) will
return it round-robin. So first a.a.a.1, then a.a.a.2, then a.a.a.1, then
a.a.a.2, etc.. Of course there are multiple dns servers involved and
thousands of clients over the world, but in the end we almost always get a
perfect 25-25-25-25 balancing.

-----Original Message-----
Sent: donderdag 6 januari 2011 9:45
Subject: Re: Haproxy failover: DNS RR vs Virtual IP (heartbeat,
keepalived)

Post by Angelo Höngens
(sorry for top posting, damn Outlook)
Just to second Willy's story, this is how a lot of people do it,

including us. We use pacemaker for high availability, and dnr rr for
loadbalancing.

Post by Angelo Höngens
For example we have a 4-node cluster running varnish and haproxy. In

this case I have 4 virtual ipv4-addresses and 4 virtual ipv6 addresses
on the cluster. We use pacemaker to keep the virtual ip's up, and we
use dns round-robin to balance the load. We get nice equal load
balancing this way, and if a node is down (or I want to do
maintenance), the vip's move to other nodes, and they take the extra
load.
Thanks for the information. Both Willy and you refer to DNS RR as a load
balancing solution, but I don't really understand that point: if
caching, etc... means hostname->load balancer resolution is random, the
load balancing will likely be very unbalanced, no ?
cheers,
David

[Brett Delle Grazie]

Hi David,

Post by Willy Tarreau
Hi David,

Post by David
Hi,
Let's say I have an architecture where a couple of servers are put
behind a haproxy instance for load balancing, to serve content at
www.example.com. For reliability/availability purpose, I want to have
more than one haproxy instance to ensure continuous servicing when one
LB1 LB2
| |
------------------------------------
| | |
Server1 Server2 Server3
The issue is how to "distribute" the load across both load balancers. I
- DNS Round Robin: www.example.com is resolved to both LB1 and LB2's
IP. If e.g. LB1 crashes, clients will then look at the next entry, LB2
in this case
- High Availability IP (heartbeat, keepalive) between both load
balancers. Only one load balancer is proxying all the requests at a time
(I assume one load balancer has enough power to serve all our traffic).

Another couple of options are (since I've just researched this very
thing not more than 5 months ago):

(3) ClusterIP: is a multicast MAC address clustering method for a single
IP address. The down side to this is that all traffic is sent to _all_
nodes. This must be used in conjunction with one of the cluster
controller's below (Corosync + Pacemaker or Heartbeat 3.x + Pacemaker)
since this is implemented in iptables and needs 'controlling software'
to tell the node which parts of traffic to listen to.
http://security.maruhn.com/iptables-tutorial/x8906.html

(4) Common Address Redundancy Protocol (CARP): Protocol originally
developed under BSD. Often used in Firewalls. Alternative to Cisco's
proprietary HSRP protocol. Linux has a user-space implementation (uCARP)
http://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol

(5) RedHat Cluster Suite: Similar to Corosync + Pacemaker but was
developed separately - moving slowly towards using Pacemaker as a
resource controller. In RHEL6, Pacemaker can be used as the resource
controller.

(6) Corosync + Pacemaker or Heartbeat 3.x + Pacemaker (these are the
newer cluster control software) - These are extremely good if you have
multiple things to control as part of a cluster. Heartbeat 2.x was
split into Heartbeat 3 + Pacemaker to streamline development. Corosync
is the 'cluster membership' and is required if you are using shared file
systems, Pacemaker is a generic resource controller.

The best solution would depend upon your requirements but an optimal one
(with no requirement for a server to be dormant until failure - i.e.
pure HA) is (for 2 node load balancer):
2 IPs registered in DNS-RR
2 virtual IPs which are those listed in DNS-RR.
HAproxy running on both nodes
Cluster controlling software (RHCS or Corosync + Pacemaker or Heartbeat
+ Pacemaker) managing virtual IPs and HA proxy on both nodes. According
to current advice from linux-cluster and pacemaker mailing lists - stay
away from Heartbeat 2.x - I know this is supplied by most distributions
but it is buggy and doesn't handle certain corner cases well.

Hope this helps.

Post by Willy Tarreau

Post by David
I have been asked to implement the DNS RR method, but from what I have
read, method 2 is the one most commonly used. What are the pros/cons of
each ?

The first one is just pure theory. You may want to test it by yourself
to conclude that it simply does not work at all. Most clients will see
a connection error or timeout, and few of them will be able to perform
a retry on the other address but after some delay which will cause some
unpleasant experience. Also, most often the browser does not perform a
new lookup if the first one has already worked. That means that until
the browser is closed, the visitor will remain bound to the same IP.
Then you might think that it's enough to update the DNS entries upon
failure, but that does not propagate quickly, as there are caches
everywhere. To give you an idea, the haproxy ML and site were moved to
a new server one month ago, and we're still receiving a few requests a
minute on the old server. In general you can count on 1-5% of the visitors
to cache an entry more than one week. This is not a problem for a disaster
recovery, but it certainly is for a server failover because that means you
cannot put it offline at all.
High availability has the big advantage of always exposing a working
service for the same IP address, so it's a *lot* more reliable and
transparent to users. There are two common ways to provide HA under
- heartbeat
- keepalived
The first one is more suited to data servers, as it ensures that no more
than one node is up at a time. This is critical when you share file systems.
The second one is more suited to stateless servers such as proxies and load
balancers, as it ensures that no less than one node is up at a time. Sadly
people generally confuse them and sometimes use keepalived for NFS servers
or use heartbeat with haproxy...
High availability presents a small inconvenient though : the backup node
is never used so you don't really know if it works well, and there is a big
temptation not to update it as often as the master node. This is also an
advantage in that it allows you to validate your new configs on it before
loading them on the master node. If you want to use both LBs at the same
time, the solution is to have two crossed VIPs on your LBs and use DNS RR
to ensure that both are used. When one LB fails, the VIP moves to the other
one.
- DNS = load balancing, no availability at all
- HA = availability, no load balancing at all.
=> use DNS to announce always available IP addresses
Cheers,
Willy

[Angelo Höngens]

Post by Brett Delle Grazie
The best solution would depend upon your requirements but an optimal one
(with no requirement for a server to be dormant until failure - i.e.
2 IPs registered in DNS-RR
2 virtual IPs which are those listed in DNS-RR.
HAproxy running on both nodes
Cluster controlling software (RHCS or Corosync + Pacemaker or Heartbeat
+ Pacemaker) managing virtual IPs and HA proxy on both nodes. According
to current advice from linux-cluster and pacemaker mailing lists - stay
away from Heartbeat 2.x - I know this is supplied by most distributions
but it is buggy and doesn't handle certain corner cases well.
Hope this helps.

And if you want to have your pacemaker+corosync cluster up within the
next hour, follow this howto on my site ;)

http://blog.hongens.nl/guides/setting-up-a-pacemaker-cluster-on-centosrhel/
--
With kind regards,


Angelo Höngens
systems administrator

MCSE on Windows 2003
MCSE on Windows 2000
MS Small Business Specialist
------------------------------------------
NetMatch
tourism internet software solutions

Ringbaan Oost 2b
5013 CA Tilburg
[phone]
[phone]

A.Hongens-isB3h8G6/***@public.gmane.org
www.netmatch.nl
------------------------------------------