Discussion:
[PATCH] MEDIUM: lua: Add stick table support for Lua
Thierry Fournier
2018-08-21 07:45:28 UTC
Permalink
Hi guys,
I've attached a patch to add stick table support to Lua. Operations are
- Show basic table info
- Key lookup
- Table dump, optionally using data/column filter
One side note, the code provides support for multiple filters
(4 by default) while CLI doesn't support that, nor it complains about
multiple "<data.type> <op> <val>" clauses.
Hi Adis,

This is a great feature. I look this ASAP.
Also, if this patch is accepted, maybe we can use provided helper
functions in other places in the code.
It's always funny to reply to self, right sending email to public I've
spotted a bug. New patch attached.
SMT_T_SINT should be treated as ordinary signed integer, and shoud use
lua_pushinteger() on it?
There are a function which convert haproxy type in Lua types: hlua_arg2lua(),
and SINT is converted with lua_pushinteger(). I guess that stick tables SINT
are the same.
On many places in the code it is noted as "64" bit integer, but in
struct stktable_type stktable_types[SMP_TYPES] = {
[SMP_T_SINT] = { "integer", 0, 4 },
The lua_pushinteger function takes a lua_Integer which is defined as known C
type (int, long or long long). So, when the function lua_pushinteger() is
called, the compilator perform a conversion between the type passed as argument
and the expected type. So I’m confident with the good behavior.

br,
Thierry
Best regards,
Adis
<0001-MEDIUM-lua-Add-stick-table-support-for-Lua-read-only.patch>
Thierry Fournier
2018-08-21 09:32:34 UTC
Permalink
Hi Adis,

Thanks for this patch, it is a very useful class.

Some remark about the documentation and the formats:
----------------------------------------------------

js:function:: StickTable.info()
js:function:: StickTable.lookup(key)

Maybe the specification and an example of the returnes values
will be welcome, because I guess that the table keys are hardcoded.
Something like :

{
type = <string> -- list of given string,
length = <int>,
...
}

or

{
expire = <integer> -- Specifiy the unit (ms or ?)
}

js:function:: StickTable.dump([filter])

The exact list of allowed operators will helps the user to use
your class. It seems that string or regexes operators are not
allowed, only the integer operator are taken in account. This
list is "eq", "ne", "le", "lt", "ge", "gt".

Same remarqk for allowed data type. Maybe a link to the HAProxy
documentation will be sufficient for the data type.

I see in the code that the filters can exceed 4 entries. This
limitation must be written in the doc.

I miss also the relation between oprators and between the content
of operators. I mean AND or OR. How I understand your example:

+ local filter = {
+ lt={{"gpc0", 1}, {"gpc1", 2}},
+ gt={{"conn_rate", 3}},
+ eq={{"conn_cur", 4}}
+ }

( gpc0 >= 1 AND gpc1 >= 2 ) OR (conn_rate > 3) OR (conn_cur = 4)
or
( gpc0 >= 1 OR gpc1 >= 2 ) OR (conn_rate > 3) OR (conn_cur = 4)
or
( gpc0 >= 1 AND gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)
or
( gpc0 >= 1 OR gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)

Are you sure that the syntax <operator>=<list of left, right operands>
is a good format ? Maybe something like the following, with the operator
as argument between the two operands. lines are implicitly OR, and columns
are AND:

+ local filter = {
+ {{"gpc0", "lt", 1}, {"gpc1", "lt", 2}},
+ {{"conn_rate", "gt", 3}},
+ {{"conn_cur", "eq", 4}}
+ }

It is read:

( gpc0 >= 1 OR gpc1 >= 2 ) AND (conn_rate > 3) AND (conn_cur = 4)

You can also implement a parser for natural language, in order to understand
directly the previous string ? This algoritm is very simple to implement:

https://en.wikipedia.org/wiki/Shunting-yard_algorithm

Idea of extension for the future: Maybe it will be safe to compile
sticktable filter during the initialisation of the Lua code, to avoid
runtime errors ?


Other point with the doc of this function, you must specify that the
execution of this function can be very long for millions of entry, even
if a filter is specified because the stick table is entirely scanned.


some remarks about the code and the logs:
-----------------------------------------

The line 182 of the patch contains space in place of tab.

Line 274 of your patch, I don't see any HA_SPIN_LOCK(STK_TABLE_LOCK
I don't known very well the thread, so maybe there are useles, maybe no.

Line 311: I see the decrement of the refcount without ATOMIC function
and whitout lock. Once again, I don't known very well the thread but
I send a warning. Maybe locks are useles, maybe no.

Line 286 of your patch. It seems that smp.flags is used uninitialized.
Maybe you should apply this change:

- smp.flags |= SMP_F_CONST;
+ smp.flags = SMP_F_CONST;

l.365, 369: The user doesn't have context about the error. there are the
first entry of the table, the second ? Which operator doesn't exists ?

L.380, 384: Which line is wrong ?

L.431: Your release the lock, so the next element relative to the current
"n", can disappear and the ebmb_next() can return wrong memory.

That's all.
Thierry
Hi guys,
I've attached a patch to add stick table support to Lua. Operations are
- Show basic table info
- Key lookup
- Table dump, optionally using data/column filter
One side note, the code provides support for multiple filters
(4 by default) while CLI doesn't support that, nor it complains about
multiple "<data.type> <op> <val>" clauses.
Also, if this patch is accepted, maybe we can use provided helper
functions in other places in the code.
It's always funny to reply to self, right sending email to public I've
spotted a bug. New patch attached.
SMT_T_SINT should be treated as ordinary signed integer, and shoud use
lua_pushinteger() on it?
On many places in the code it is noted as "64" bit integer, but in
struct stktable_type stktable_types[SMP_TYPES] = {
[SMP_T_SINT] = { "integer", 0, 4 },
Best regards,
Adis
<0001-MEDIUM-lua-Add-stick-table-support-for-Lua-read-only.patch>
Thierry Fournier
2018-08-23 08:53:15 UTC
Permalink
Hi

[...]
Post by Thierry Fournier
I miss also the relation between oprators and between the content
+ local filter = {
+ lt={{"gpc0", 1}, {"gpc1", 2}},
+ gt={{"conn_rate", 3}},
+ eq={{"conn_cur", 4}}
+ }
Are you sure that the syntax <operator>=<list of left, right operands>
is a good format ? Maybe something like the following, with the operator
as argument between the two operands. lines are implicitly OR, and columns
+ local filter = {
+ {{"gpc0", "lt", 1}, {"gpc1", "lt", 2}},
+ {{"conn_rate", "gt", 3}},
+ {{"conn_cur", "eq", 4}}
+ }
Actually, I was playing with some other ideas, and it was useful to be
able to "preselect" filter operators.
However, the CLI doesn't even support more than one, maybe we don't need
local filter = {
{"gpc0", "lt", 1},
{"gpc1", "lt", 2},
{"conn_rate", "gt", 3},
{"conn_cur", "eq", 4}
}
The default operator would be AND, and we would not support other
operators (to keep the things simple). e.g. example use case for the
filter would be to filter out on gpc0 > X AND gpc1 > Y
If this sounds good, I can update/simplify the code.
Ok, it sounds good. I think this kind of syntax is easily understandable
and it allow a good way for filtering values.
Post by Thierry Fournier
Idea of extension for the future: Maybe it will be safe to compile
sticktable filter during the initialisation of the Lua code, to avoid
runtime errors ?
I'm returning runtime errors since it can be easy to mix up data from
the client side (most probably data would come as json table, then
transformed to Lua table)
ok

[...]
Post by Thierry Fournier
Line 274 of your patch, I don't see any HA_SPIN_LOCK(STK_TABLE_LOCK
I don't known very well the thread, so maybe there are useles, maybe no.
hlua_stktable_lookup() uses stktable_lookup_key() which does have locks,
so I guess that it should be fine then?
sure !


[...]
Post by Thierry Fournier
l.365, 369: The user doesn't have context about the error. there are the
first entry of the table, the second ? Which operator doesn't exists ?
L.380, 384: Which line is wrong ?
Yes, it is somwehat cryptic. I've tried to avoid returning user supplied
data in the error messages. We can revisit this if/when we change the
filter table format.
ok
Post by Thierry Fournier
L.431: Your release the lock, so the next element relative to the current
"n", can disappear and the ebmb_next() can return wrong memory.
I was under impression that we only have to acquire lock and increment
ref_cnt (so we can be sure our current node n is not deleted)
ebmb_next() is called only when we're holding lock, first and every
other iteration, i.e.
HA_SPIN_LOCK(STK_TABLE_LOCK, &t->lock);
eb = ebmb_first(&t->keys);
for (n = eb; n; n = ebmb_next(n)) {
...
ts->ref_cnt++;
HA_SPIN_UNLOCK(STK_TABLE_LOCK, &t->lock);
...
HA_SPIN_LOCK(STK_TABLE_LOCK, &t->lock);
}
Or I didn't get your point?
ok, you probably right.


Thierry
Willy Tarreau
2018-08-23 13:43:59 UTC
Permalink
Hi Thierry,

On Thu, Aug 23, 2018 at 10:53:15AM +0200, Thierry Fournier wrote:
(...)
Post by Thierry Fournier
Ok, it sounds good. I think this kind of syntax is easily understandable
and it allow a good way for filtering values.
Does this mean I should merge Adis' patch or do you want to verify
other things ? Just let me know.

Thanks,
Willy
Thierry Fournier
2018-09-27 14:52:29 UTC
Permalink
I Adis,

Sorry for the delay, I processed a quick review, and all seems to be ok for me!

BR,
Thierry
Post by Willy Tarreau
Post by Willy Tarreau
Hi Thierry,
Have you had the time to review my patches?
Thierry,
Reviving this thread, do you have any objections about latest version of
the patch (Lua stick table patch)
Best regards,
Adis
Willy Tarreau
2018-09-29 18:15:55 UTC
Permalink
Hi Adis,
Post by Thierry Fournier
I Adis,
Sorry for the delay, I processed a quick review, and all seems to be ok for me!
BR,
Thierry
Great, happy to hear that, I hope guys will merge it soon.
OK, just merged now.

Thanks to you both!

Willy
b***@gmail.com
2018-10-10 22:17:18 UTC
Permalink
Post by Thierry Fournier
Hi Adis,
Post by Thierry Fournier
I Adis,
Sorry for the delay, I processed a quick review, and all seems to be ok for me!
BR,
Thierry
Great, happy to hear that, I hope guys will merge it soon.
OK, just merged now.
Thanks to you both!
Willy
Hi Adis,

nice feature, thank you. Are there plans for adding write access also?

Currently i've a use case for this in Lua (get/set some sort of shared
lock) and i'm planning to use HAProxy maps as a workaround instead of
stick tables (or writing entries to stick tables from Lua via tcp to
admin socket).

----------------------------------------------------------------
Best Regards / Mit freundlichen Grüßen

Bjoern

Loading...