Design Proposal: http-agent-check, explict health checks & inline-mode

[Robin H. Johnson]

Hi,

This is something I have a vague recollection of existing somewhere, but
didn't find any leads in documentation or source.

Right now, if you want to use load feedback for weights, you either need
something entirely out-of-band from the servers back to HAProxy, or you
have to use the agent-check option and run a separate health agent.

The agent-check protocol is described only in the configuration.txt
'agent-check' section, and is conveyed entirely over pure TCP, no HTTP.
It supports conveying useful health including weight and DRAIN/MAINT
states.

The http-check behavior only supports matching strings or status codes,
and does not convey any load feedback.

I would like to propose a new http-agent-check option, with two usage
modes.
1. health-check mode: this connects like the existing agent-check, but
sends does HTTP request & response rather than pure TCP.

2. inline mode: if the server has best-case knowledge about it's status,
and HTTP headers are used for the feedback information, then it
should be possible to include the feedback in an HTTP response header
as part of normal queries. The header processing would detect & feed
the data into the health system during normal traffic.

Question: where & how should the feedback information be encoded in the
response?
1. HTTP payload
2. Single HTTP header
3. Multiple HTTP headers

[Robin H. Johnson]

Post by Robin H. Johnson
Right now, if you want to use load feedback for weights, you either need
something entirely out-of-band from the servers back to HAProxy, or you
have to use the agent-check option and run a separate health agent.

With that you mean "external-check command" ?
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#external-check%20command

No, I mean 'agent-check' per
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-agent-check

This is an agent that runs on the realserver, not the load balancer.

...

Post by Robin H. Johnson
I would like to propose a new http-agent-check option, with two usage
modes.
1. health-check mode: this connects like the existing agent-check, but
sends does HTTP request & response rather than pure TCP.
2. inline mode: if the server has best-case knowledge about it's status,
and HTTP headers are used for the feedback information, then it
should be possible to include the feedback in an HTTP response header
as part of normal queries. The header processing would detect & feed
the data into the health system during normal traffic.

Interesting Ideas.
Are there any LB's out there which already uses this concept?

I haven't looked specifically, but I am aware of a lot of other
dynamic-realserver weight work (mostly in the keepalived/ipvs world,
like feedbackd and lvs-kiss from the early 2000's).

The inline mode is probably deserving of seperate work, I think it might be possible
to implement it with the existing Lua codebase.

Post by Robin H. Johnson
Question: where & how should the feedback information be encoded in the
response?
1. HTTP payload
2. Single HTTP header
3. Multiple HTTP headers

I would like to have it in the one header per value 'Server-State-*' as the X-
Prefix is Deprecated.
https://tools.ietf.org/html/rfc6648
Deprecating the "X-" Prefix and Similar Constructs in Application Protocols
Server-State-Load
Server-State-Users
Server-State-Health
Server-State-...

Multiple headers easier to write new parsers for I agree, but supporting
the other variants might be worthwhile.

I'm thinking to maybe implement my lua-check first, then write a simple
HTTP checker within Lua.