CLI proxy for master process

[Willy Tarreau]

The problem is that at the moment it's not possible to connect to the stats
socket of a process which is leaving. Sometimes it's really useful to debug and
see the session which are still connected on the old process. And that's the
ultimate goal of this feature (not covered yet, but soon :-) )

This precisely is the case I'm personally interested in : I don't upgrade
often (I'm not a good example to follow but on the other hand I'm well aware
of the need to upgrade for my use cases), and occasionally during the reload
on the new version I see the old process not quitting. The developer in me
cannot help but think "grrrr I should have kept a connection to this stats
socket to see what's happening". When this happens, it's after a huge version
jump so it's hard to tell if it's an old fixed bug or not, of course.

With the ability to consult old processes, I could connect to the master,
enter an old process and start to debug it (show sess, etc), or even
selectively kill certain connections. This can a be very convenient
feature. I don't use the master-worker model for now (as I'm a happy and
lucky systemd-less user so I have the choice) but as I told William, this
definitely is one feature that could make me switch to the master-worker
model.

With nbproc, there's also the ability to connect to all processes via a
single connection that some people will appreciate. Those running 4 or
even more processes are probably fed up with having to reconnect to each
of these individual sockets when troubleshooting something. Here you can
access everything from the same socket. For example when you're searching
a connection using "show sess | socat | fgrep src=10.11.12.13", instead
of doing it in a for loop in shell, you could easily have a single command
that sends this to each process and provides you with the info you're
looking for, wherever it comes form.

I'm sure that over time new ideas will emerge around this. We just need
to be reasonable not to go too far too quickly or it will be hard to go
back and take a different route if needed.

Cheers,
Willy

[Willy Tarreau]

BTW what's nb in "nb(thread|proc)"?
[ ] No block
[ ] never been

"NumBer" :-)

This one is not derived from french, it's not like
"option independant-streams" which I messed up years ago!

Willy

[Lukas Tribus]

Hello,

Hi Aleks,
With a nbproc setup, the first goal is to be able to access multiple stats
sockets from one socket.
In a more "modern" nbthread setup, it's possible to have only one worker, but
we still fork a new process upon a reload.
The problem is that at the moment it's not possible to connect to the stats
socket of a process which is leaving. Sometimes it's really useful to debug and
see the session which are still connected on the old process. And that's the
ultimate goal of this feature (not covered yet, but soon :-) )
It also implements a "show proc" which lists the PIDs of the processes.
Regards,

Btw could stdout logging leverage some of this infrastructure? I
assume that if the master process would write to stdout
(synchronously), we would not stall our workers because of slow log
readers?

https://www.mail-archive.com/***@formilux.org/msg17436.html
https://github.com/dockerfile/haproxy/issues/3
https://github.com/docker-library/haproxy/pull/39


It looks like in the docker world, everything other than stdout
logging is unacceptable - at least for official docker image, and I'm
wondering if the master/worker process model along with this
infrastructure could theoretically help there, without sacrificing the
performance of the event loop or log reliability.



Regards,
lukas

[David Birdsong]

love socklog!

i use it combined w/ s6-log https://skarnet.org/software/s6/s6-log.html

Hi.
In the meantime you can use Socklog [1] or fluent-bit [2] to listen to
syslog and write to stdout as I use it in my Image. Pay attention that
Fluent-bit writes by default json format to stdout.
https://gitlab.com/aleks001/haproxy18-centos/blob/master/containerfiles/container-entrypoint.sh#L94
The image setup is documented in this post.
https://www.me2digital.com/blog/2017/08/openshift-egress-options/#haproxy-generic-proxy
I have also written a blog post about syslog in the container world as
this is a non trivial topic, imho.
https://www.me2digital.com/blog/2017/05/syslog-in-a-container-world/
It would be nice to have a log to stdout option in HAProxy, but until this
is available there are many solutions available out there to solve the
issue.
Regards Aleks
[1] http://smarden.org/socklog/
[2] https://fluentbit.io
-------- UrsprÃŒngliche Nachricht --------
Gesendet: 5. November 2018 17:08:09 GMT+00:00
Betreff: Re: CLI proxy for master process
Hi Lukas,

Post by Lukas Tribus
Btw could stdout logging leverage some of this infrastructure? I
assume that if the master process would write to stdout
(synchronously), we would not stall our workers because of slow log
readers?

Well, it will just move the problem from the worker to the master, which
will
be slowed down and can prevent the ability to do maintenance operations
from the
master CLI. At the moment we don't have anything to forward the logs from
the
workers to the master which is another problem, but we could use the
sockpair
between them, but this is a stream socket which will lead to the same
issue.

Post by Lukas Tribus
https://github.com/dockerfile/haproxy/issues/3
https://github.com/docker-library/haproxy/pull/39
It looks like in the docker world, everything other than stdout
logging is unacceptable - at least for official docker image, and I'm
wondering if the master/worker process model along with this
infrastructure could theoretically help there, without sacrificing the
performance of the event loop or log reliability.

We are aware of the problem and we already discussed this issue several
time
with the other developers, the ideal design should be the use of a log
buffer
which will allow TCP logging, and stdout logging. We have the idea in mind
but
the development is not planned yet. But it will definitively be a useful
feature.