Discussion:
How to deal with proxying shoutcast / ICY protocol
David Young
2011-03-10 02:36:11 UTC
Permalink
Hi folks,

First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.

I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.

A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).

I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.

An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3

Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.

I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).

Can anybody enlighten me?

Many thanks,
David

-- Relevant haproxy.cfg --

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog

# Log on start of connection, not completion
option logasap

# log errors separately
option log-separate-errors

option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3


#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend


backend test_backend
balance roundrobin
server localhost localhost:3128



-- Telnet to haproxy infront of single squid tcp backend --

DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$


-- Telnet to squid directly --

DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close

00/Ȯt?????sƸD?,?4A?Q`F??BA????B??@(???u??kTC??????? <binary stream
continues...>
Hank A. Paulson
2011-03-10 03:32:01 UTC
Permalink
Is ICY really listening on localhost:3128 ?
If you telnet directly to that, does it work?
Post by David Young
Hi folks,
First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.
I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.
A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).
Can anybody enlighten me?
Many thanks,
David
-- Relevant haproxy.cfg --
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog
# Log on start of connection, not completion
option logasap
# log errors separately
option log-separate-errors
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend
backend test_backend
balance roundrobin
server localhost localhost:3128
-- Telnet to haproxy infront of single squid tcp backend --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$
-- Telnet to squid directly --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1:<BR>This stream requires<a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close
continues...>
Willy Tarreau
2011-03-10 06:34:40 UTC
Permalink
Hi David,
Post by David Young
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
If you get a 502 while you're in "mode tcp", then you're not in "mode tcp",
but still in http. Haproxy cannot emit HTTP error codes in TCP mode.

Looking at your config, I understand what happens : your "defaults" section
is http, which means that all subsequent sections will have "mode http" by
default. Your frontend is explicitly TCP but not your backend, which is
still HTTP. So once the connection is forwarded to the backend, it's still
parsed and processed in HTTP mode.

One thing to keep in mind : if you set a mode in a defaults section, do not
use a different one in subsequent sections, or sooner or later you'll get
trapped. Better redefined a new defaults section to make things clearer. I
like to use two sections "defaults http" and "defaults tcp" for this.

Regards,
Willy
Florescu, Dan Alexandru
2011-03-10 08:43:50 UTC
Permalink
Hi David,

I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :)

I too am performing LB between proxies.
The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1.

If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail.
It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case).
so: mode tcp in backend and you're ok.

I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend.


-----Original Message-----
From: David Young [
Sent: Thursday 10, March 03, 2011 4:36
To: ***@formilux.org
Subject: How to deal with proxying shoutcast / ICY protocol

Hi folks,

First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.

I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.

A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).

I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.

An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3

Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.

I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).

Can anybody enlighten me?

Many thanks,
David

-- Relevant haproxy.cfg --

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog

# Log on start of connection, not completion
option logasap

# log errors separately
option log-separate-errors

option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3


#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend


backend test_backend
balance roundrobin
server localhost localhost:3128



-- Telnet to haproxy infront of single squid tcp backend --

DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$


-- Telnet to squid directly --

DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close

00/Ȯt?????sƸD?,?4A?Q`F??BA????B??@(???u??kTC??????? <binary stream
continues...>


The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.ro
David Young
2011-03-10 09:22:01 UTC
Permalink
Hi Willy, Dan,

Thanks for pointing out that I was still in http mode - that was an
oversight, and should solve my problem for now. Long term, I'd rather
use http mode over tcp for flexibility, but unfortunately I can't get
away from having to support Shoutcast (and any other dodgy protocol you
can think of!) :(

Dan - I guess we're stuck making backend decisions on destination IP
only - any reason you're using Squid 3.0 over 3.1?

I'll test in the lab tomorrow and report back results :)

D
Post by Willy Tarreau
Hi David,
I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :)
I too am performing LB between proxies.
The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1.
If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail.
It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case).
so: mode tcp in backend and you're ok.
I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend.
-----Original Message-----
From: David Young [
Sent: Thursday 10, March 03, 2011 4:36
Subject: How to deal with proxying shoutcast / ICY protocol
Hi folks,
First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.
I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.
A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).
Can anybody enlighten me?
Many thanks,
David
-- Relevant haproxy.cfg --
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog
# Log on start of connection, not completion
option logasap
# log errors separately
option log-separate-errors
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend
backend test_backend
balance roundrobin
server localhost localhost:3128
-- Telnet to haproxy infront of single squid tcp backend --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$
-- Telnet to squid directly --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close
continues...>
The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
Florescu, Dan Alexandru
2011-03-10 09:58:36 UTC
Permalink
Post by Willy Tarreau
Post by David Young
I guess we're stuck making backend decisions on destination IP
Well, true enough, BUT: when you use a proxy in, let's say Winamp, and add a stream server and play it, the Destination IP seen by Haproxy will not be the stream server's one (external IP), it will be the Proxy server's one (internal IP) - because you are in fact establishing a connection to Squid and it connects afterwards to the intended destination.
This is why I can't use this method.
Post by Willy Tarreau
Post by David Young
any reason you're using Squid 3.0 over 3.1?
It's a production environment with several hundred people using it at the same time, so I can't upgrade just yet.




-----Original Message-----
From: David Young
Sent: Thursday 10, March 03, 2011 11:22
To: ***@formilux.org
Subject: Re: How to deal with proxying shoutcast / ICY protocol

Hi Willy, Dan,

Thanks for pointing out that I was still in http mode - that was an
oversight, and should solve my problem for now. Long term, I'd rather
use http mode over tcp for flexibility, but unfortunately I can't get
away from having to support Shoutcast (and any other dodgy protocol you
can think of!) :(

Dan - I guess we're stuck making backend decisions on destination IP
only - any reason you're using Squid 3.0 over 3.1?

I'll test in the lab tomorrow and report back results :)

D
Post by Willy Tarreau
Hi David,
I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :)
I too am performing LB between proxies.
The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1.
If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail.
It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case).
so: mode tcp in backend and you're ok.
I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend.
-----Original Message-----
From: David Young [
Sent: Thursday 10, March 03, 2011 4:36
Subject: How to deal with proxying shoutcast / ICY protocol
Hi folks,
First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.
I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.
A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).
Can anybody enlighten me?
Many thanks,
David
-- Relevant haproxy.cfg --
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog
# Log on start of connection, not completion
option logasap
# log errors separately
option log-separate-errors
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend
backend test_backend
balance roundrobin
server localhost localhost:3128
-- Telnet to haproxy infront of single squid tcp backend --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$
-- Telnet to squid directly --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close
continues...>
The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://ww
David Young
2011-03-11 00:05:00 UTC
Permalink
Works as advertised, thank you :)

One thing is now missing from my setup, and that's the ability to
identify WHICH source IPs are accessing WHICH sites.

Without haproxy, I'm able to do this with squid's access.log, and with
"mode http", I'm able to see this in the haproxy logs.

When I enable "mode tcp" though, to work around this silly shoutcast
issue, I get source IPs _only_ in the haproxy.log, and my squid access
log reports every request has having a source IP of my haproxy host.

Any ideas?

Thanks,
D
Post by David Young
Hi Willy, Dan,
Thanks for pointing out that I was still in http mode - that was an
oversight, and should solve my problem for now. Long term, I'd rather
use http mode over tcp for flexibility, but unfortunately I can't get
away from having to support Shoutcast (and any other dodgy protocol you
can think of!) :(
Dan - I guess we're stuck making backend decisions on destination IP
only - any reason you're using Squid 3.0 over 3.1?
I'll test in the lab tomorrow and report back results :)
D
Post by Willy Tarreau
Hi David,
I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :)
I too am performing LB between proxies.
The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1.
If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail.
It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case).
so: mode tcp in backend and you're ok.
I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend.
-----Original Message-----
From: David Young [
Sent: Thursday 10, March 03, 2011 4:36
Subject: How to deal with proxying shoutcast / ICY protocol
Hi folks,
First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.
I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.
A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).
Can anybody enlighten me?
Many thanks,
David
-- Relevant haproxy.cfg --
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog
# Log on start of connection, not completion
option logasap
# log errors separately
option log-separate-errors
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend
backend test_backend
balance roundrobin
server localhost localhost:3128
-- Telnet to haproxy infront of single squid tcp backend --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$
-- Telnet to squid directly --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close
continues...>
The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.com/disclaimer.html
Willy Tarreau
2011-03-11 08:23:54 UTC
Permalink
Post by David Young
Works as advertised, thank you :)
One thing is now missing from my setup, and that's the ability to
identify WHICH source IPs are accessing WHICH sites.
Without haproxy, I'm able to do this with squid's access.log, and with
"mode http", I'm able to see this in the haproxy logs.
When I enable "mode tcp" though, to work around this silly shoutcast
issue, I get source IPs _only_ in the haproxy.log, and my squid access
log reports every request has having a source IP of my haproxy host.
Any ideas?
Well, at this point, I think it would make sense to start thinking about
the possibility to implement HTTP derivative protocols such as shoutcast
and icap. We need to carefully study their specificities and their
compatibility with HTTP so that we figure if we have to configure
exclusive support for one or the other, or if we can dynamically adapt
without taking risks nor opening security issues.

For this, we have to consider the following points :

- what does a normal shoutcast request look like
- what does a proxied shoutcast request look like
- do shoutcast servers accept HTTP/1.0 or 1.1 requests
- what does a shoutcast response look like
- can it be sent in response to a valid HTTP request, and what is the
risk of wrong identification (eg: assume shoutcast instead of http or
conversely)
- do shoutcast clients accept different responses
- do some proxies accept "hybrid" requests such as shoutcast requests
with many standard HTTP headers

There are probably other points that will come after those ones, those are
just a few thoughts. Maybe we'll figure a way to easily build a basic
implementation, or maybe it will require some deeper work.

Willy
Florescu, Dan Alexandru
2011-03-11 08:24:10 UTC
Permalink
I do not think you can see this with mode tcp, because of the lack of X-Forwarded-For.
Your request could have been possible with mode http with the following settings:

in haproxy -> option forwardfor (I see you already have this enabled)
in squid -> (in this order)
logformat squid [%{%d/%m/%Y-%H:%M:%S}tl] %{X-Forwarded-For}>h through %>a %Ss/%03Hs %rm/%Hs %ru %Sh/%<A %mt Referer:%{Referer}>h
cache_access_log /var/log/squid3/access.log squid
and disable cache_access_log if it is enabled.

From the logformat above, the essential parts are:
%{X-Forwarded-For}>h = read the X-Forwarded-For header in the http request
%>a = display originating server or peer IP address

for more reference consult http://devel.squid-cache.org/customlog/logformat.html
Do try this with mode tcp as I am currently unable to.



-----Original Message-----
From: David Young
Sent: Friday 11, March 03, 2011 2:05
To: ***@formilux.org
Subject: Re: How to deal with proxying shoutcast / ICY protocol

Works as advertised, thank you :)

One thing is now missing from my setup, and that's the ability to
identify WHICH source IPs are accessing WHICH sites.

Without haproxy, I'm able to do this with squid's access.log, and with
"mode http", I'm able to see this in the haproxy logs.

When I enable "mode tcp" though, to work around this silly shoutcast
issue, I get source IPs _only_ in the haproxy.log, and my squid access
log reports every request has having a source IP of my haproxy host.

Any ideas?

Thanks,
D
Post by David Young
Hi Willy, Dan,
Thanks for pointing out that I was still in http mode - that was an
oversight, and should solve my problem for now. Long term, I'd rather
use http mode over tcp for flexibility, but unfortunately I can't get
away from having to support Shoutcast (and any other dodgy protocol you
can think of!) :(
Dan - I guess we're stuck making backend decisions on destination IP
only - any reason you're using Squid 3.0 over 3.1?
I'll test in the lab tomorrow and report back results :)
D
Post by Willy Tarreau
Hi David,
I'm kind of amazed that we have both the same problem ( http://www.formilux.org/archives/haproxy/1103/4397.html ) in almost the same circumstances, and we both asked for help at the same time :)
I too am performing LB between proxies.
The issue with ICY is also in Squid. You need the upgrade_http0.9 ( http://www.squid-cache.org/Doc/config/upgrade_http0.9/ ) option. But this one works only with Squid 2.7 and 3.1, not with 3.0. I myself have a 3.0 for the main Squid, so I thought I would balance it with Haproxy (elegant solution) to a Squid 3.1.
If I try streaming directly on the 3.1 Squid, it will work, but through Haproxy it will fail.
It will work through Haproxy if you enable mode tcp. Now, I see you have enabled mode tcp in frontend, but you didn't do this for the backend aswell. You also have a mode http in the defaults section, so if you don't explicitly specify it in the backend, it will get the default-specified mode (http in your case).
so: mode tcp in backend and you're ok.
I myself have another issue, I cannot send the requests for ICY/radio streaming through my second squid because I don't have any relevant options to use in ACLs, so all requests are sent to the default_backend.
-----Original Message-----
From: David Young [
Sent: Thursday 10, March 03, 2011 4:36
Subject: How to deal with proxying shoutcast / ICY protocol
Hi folks,
First-time poster here - we've been working on implementing haproxy to
perform load balancing between our backend squid proxies.
I stumbled across an issue today when I realized that I can't listen to
streaming radio via my haproxy instance, but that it works fine if I
direct my request to one of by backend proxies directly.
A bit of searching lead me to the issue of HTTP/1.1 vs ICY as the
probable cause
(http://www.squid-cache.org/Versions/v3/3.1/RELEASENOTES.html#ss2.10).
I thought that if I configured haproxy in "mode tcp" instead of "mode
http", that it'd behave more like a simple loadbalancer, and just pass
the request onto my squid backend verbatim, but that doesn't seem to
have worked either.
An example URL I'm having trouble with is http://66.225.205.47/;stream.mp3
Even in "mode tcp", if I request that URL through by browser from
haproxy, I get a 502 error, whereas if I requested it directly from the
backend squid instance which services my request anyway, I get the
expected headers and stream.
I thought I'd try telnetting to the haproxy / squid ports respectively,
to test, and I received differing output (below).
Can anybody enlighten me?
Many thanks,
David
-- Relevant haproxy.cfg --
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option dontlognull
option http-server-close
option httplog
# Log on start of connection, not completion
option logasap
# log errors separately
option log-separate-errors
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is
not found
timeout client 300000
timeout server 300000
maxconn 60000
retries 3
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend test_frontend *:5001
mode tcp
option tcpka
default_backend test_backend
backend test_backend
balance roundrobin
server localhost localhost:3128
-- Telnet to haproxy infront of single squid tcp backend --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 5001
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
Connection closed by foreign host.
DavidBook:~ davidy$
-- Telnet to squid directly --
DavidBook:~ davidy$ telnet webscan-dev.blahblah.net 3128
Trying x.x.x.x...
Connected to webscan-dev.blahblah.net.
Escape character is '^]'.
GET http://66.225.205.47/;stream.mp3
ICY 200 OK
Date: Thu, 10 Mar 2011 02:25:55 GMT
icy-notice1: <BR>This stream requires <a
href="http://www.winamp.com/">Winamp</a><BR>
icy-notice2: SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
icy-name: Family Friendly WBGL
icy-genre: Contemporary Christian
icy-url: http://www.wbgl.org
Content-Type: audio/mpeg
icy-pub: 1
icy-br: 96
X-Cache: MISS from webscan-dev.blahblah.net
X-Cache-Lookup: MISS from webscan-dev.blahblah.net:3128
Via: 0.0 webscan-dev.blahblah.net (squid/3.1.11)
Connection: close
continues...>
The information contained herein is intended for its addressee(s) only and it is privileged or otherwise confidential. Any unauthorized distribution, amendment or disclosure hereof is strictly forbidden by the law. Please find complete and translated versions at http://www.rompetrol.co
Loading...