Feature Proposal: Setting Variables Using Macros

[Friday 13 November 2009] [17:21:32] <MichaelDaum> CDot, I just came across the need to deal with macro parameters of the
same name returned as an array (in perl)
[Friday 13 November 2009] [17:21:50] <MichaelDaum> like cgi params in an array context
[Friday 13 November 2009] [17:21:54] <CDot> MichaelDaum: "returned"?
[Friday 13 November 2009] [17:22:09] <MichaelDaum> the Attrs.pm overrides them ... last come wins
[Friday 13 November 2009] [17:22:11] <CDot> oic; %X%=a,b,c,d
[Friday 13 November 2009] [17:22:17] <CDot> yeah, probably
[Friday 13 November 2009] [17:22:32] <MichaelDaum> more like this: MACRO{filter="this" filter="that"}
[Friday 13 November 2009] [17:22:52] <CDot> yep. Well, Attrs.pm can easily be extended.
[Friday 13 November 2009] [17:22:53] <MichaelDaum> and then in the macroHandler: my @filter = $params->('filter');
[Friday 13 November 2009] [17:23:40] <CDot> what does %filter% mean in that context?
[Friday 13 November 2009] [17:23:42] <MichaelDaum> problem is how to deal with normal code that does an array access: $params->{filter} ...
you still need to return the last elem from the array not the array ref
[Friday 13 November 2009] [17:23:56] <MichaelDaum> filter: an arbitrary param
[Friday 13 November 2009] [17:24:13] <MichaelDaum> s/filter/random param name/
[Friday 13 November 2009] [17:24:25] <CDot> yeah, but if I say %MACRO{filter="this" filter="that"}% what is the value of %filter%?
[Friday 13 November 2009] [17:24:48] <MichaelDaum> the last in the row -> back warts compatibility
[Friday 13 November 2009] [17:26:08] <CDot> ok, so what other cases are going to bite
[Friday 13 November 2009] [17:26:20] <CDot> %SEARCH{type="query" type="keyword"}%
[Friday 13 November 2009] [17:26:26] <MichaelDaum> you actually might want to retrieve the list in wiki app space
[Friday 13 November 2009] [17:26:34] <CDot> yes
[Friday 13 November 2009] [17:26:44] <MichaelDaum> something along the lines of %@filter%
[Friday 13 November 2009] [17:26:53] * CDot throws up
[Friday 13 November 2009] [17:27:20] <CDot> %LISTPARAM{"filter"}%
[Friday 13 November 2009] [17:27:27] * MichaelDaum throws up
[Friday 13 November 2009] [17:27:48] * CDot cleans the vomit off his keyboard and tries again.... %[filter]%
[Friday 13 November 2009] [17:28:00] <MichaelDaum> actually %listparam{}% is not bad
[Friday 13 November 2009] [17:28:14] <CDot> well, it's analagous to %URLPARAM
[Friday 13 November 2009] [17:28:43] <MichaelDaum> what about %PARAM{}%
[Friday 13 November 2009] [17:28:46] <CDot> how does URLPARAM handle lists?
[Friday 13 November 2009] [17:28:58] <MichaelDaum> URLPARAM multiple="on" separator=","
[Friday 13 November 2009] [17:29:13] <MichaelDaum> URLPARAM format="$item," is broken btw
[Friday 13 November 2009] [17:29:36] <MichaelDaum> does return thisthat, instead of this,that,
[Friday 13 November 2009] [17:30:04] <CDot> so %PARAM{"filter" multiple="on" format="$item" separator=","}%
[Friday 13 November 2009] [17:30:18] <MichaelDaum> so %PARAM{"filter" multiple="on" }% would be the proper analogy
[Friday 13 November 2009] [17:30:25] <MichaelDaum> snap
[Friday 13 November 2009] [17:30:27] <CDot> guess so
[Friday 13 November 2009] [17:31:00] <CDot> question is, if you know you want the second one, how do you refer to it?
[Friday 13 November 2009] [17:31:18] <CDot> we have no way of representing, or handling, a list value
[Friday 13 November 2009] [17:31:22] <MichaelDaum> %PARAM{"filter" multiple="on" skip="10" limit="5"}% 
[Friday 13 November 2009] [17:31:32] <CDot> * Set VALUE = scalars only, chaps
[Friday 13 November 2009] [17:31:53] <CDot> * Set @VALUE = [ 1, 2, 3 ]
[Friday 13 November 2009] [17:32:18] <CDot> * Set %VALUE = { a => b, c => d }
[Friday 13 November 2009] [17:32:18] <MichaelDaum> what about %SOMEVAR{value="1" value="2" value="3"}%
[Friday 13 November 2009] [17:32:28] <CDot> shoot me, someone, before I re-invent perl
[Friday 13 November 2009] [17:33:06] <MichaelDaum> %SET{name="filter" value="this" value="that"}%
[Friday 13 November 2009] [17:33:22] <CDot> hmmm.
[Friday 13 November 2009] [17:34:14] <MichaelDaum> the counter part would be a %GET
[Friday 13 November 2009] [17:34:21] <MichaelDaum> instead of a %PARAM
[Friday 13 November 2009] [17:34:29] <CDot> well, it would be consistent. %SET{"filter" value="this" value="%filter%"}%
[Friday 13 November 2009] [17:35:06] <CDot> make %GET handle urlparams as well, and I could learn to like it.
[Friday 13 November 2009] [17:35:06] <MichaelDaum> %SET has got some nice advantages over * Set =
[Friday 13 November 2009] [17:35:20] <CDot> %SET has *huge* advantages, IMHO
[Friday 13 November 2009] [17:36:32] <MichaelDaum> let me copy-paste this to a proposal
<CDot>   oh, also, %SET cannot affect permissions settings. So the terminology, I guess, is "Preferences" (where * Set used to set default values) and "Macros" where preferences provide default values, but those values may be overridden in %SET to provide a new value that can only be referenced using %GET) sounds dangerously like SpreadShitPlugin    [17:20]
<MichaelDaum>   naw no new namespace they go to preference land. not sure if acls are stored there actually    [17:21]
<CDot>   also, if JoePluginAuthor writes getPreferenceValue("MUNGE") they will retrieve * Set MUNGE but not %SET{"MUNGE" which could be confusing    [17:23]
<MichaelDaum>   should be avoided my @list = getPreferenceValue retrives the list ... in scalar context it is the last list elem
acls themselves are lists    [17:25]
<CDot>   erm, you can't have getPreferencesValue return the result of a %SET
it *must* return a static preference; otherwise the value is variant over the different handlers, isn't it?    [17:26]
<MichaelDaum>   wantarray?    [17:27]
<CDot>   * Set preferences are still scalar values    [17:27]
<MichaelDaum>   oic    [17:27]
<CDot>   if you want an array from it, you should have to split(',', Foswiki::Func::getPreferencesValue("MUNGE"))    [17:28]
<MichaelDaum>   which is too fragile    [17:28]
<CDot>   very fragile
Foswiki::Func::makeList($scalar) -> @list    [17:28]
<MichaelDaum>   so can Func::setPreferenceValue override ACLs?
if not then %SET has good changes not to either
<CDot>   on setPreferenceValue; not sure about that.... checking
No. setPreferencesValue sets a SESSION level preference. Session level is not checked for ACLs.    [17:45]
<MichaelDaum>   okay so session level is the natural place where all %SET goes?    [17:47]
<CDot>   I guess so
though it's still variant over the duration of the session, which worries me slightly
could be a devil to debug

-- MichaelDaum - 13 Nov 2009

This is a very important proposal. For instance if TINYMCEPLUGIN_INIT and WYSIWYGPLUGIN_STICKYBITS hashes, it would be much, much easier to override and adjust only parts of such large configuration variables.

I hope we can also think about how list types in metadata might be implemented.

I am having to do very, very ugly work to mimic hashes:

Field name	Hash equivalent
Foo_Bar	Foo['Bar']
Foo_Bar_Tree	Foo['Bar']['Tree']

FormFieldListPlugin provides a really tedious way to extract individual and groups of fields that I need.

-- PaulHarvey - 13 Nov 2009

Be very, very careful to understand the implications of this.

A %SET makes a macro value variant over the lifetime of a request. Preferences are currently invariant - once set, they do not change. To understand what this means, consider this scenario. For illustration purposes, let's say our topic contains a * Set X=1 in the topic body, %SET{"X" value="2"} in the topic body, %SET{"X" value="3"} in the edit template and %SET{"X" value="4"} in the view template.

Currently, for any request on the topic, %X% will have the value 1. That means when we are writing plugins and core code, we don't have to worry what stage in the processing pipeline we are referencing a macro; the value is invariant for the lifetime of the request.

Now with %SET, %<nopX% will still have that value 1 until macros start to be processed. Then, in view, it will have the value 1 until the topic body SET is encountered, where the value will change to 2. Then, when the template furniture is processed, the value will change to 4. When editing it will go from 1 to 3.

I don't have a problem with that in principle; plugin handlers are executed sequentially, and body and template macros processed in a defined order. However we are effectively nailing that order down, because if we ever changed it - tried to process the template macros before the topic body, for example, we would get the wrong values. This has the potential to adversely affect future optimisations in a big way - for example, "precompiling" templates.

Paul, in your specific example, TINYMCEPLUGIN_INIT will have the "preference" value until the macro that "extends" the value is processed. I'm not sure where you could put that SET macro such that it gets processed before the initPlugin that reads the value of TINYMCEPLUGIN_INIT.

Potentially confusing, fragile, very geeky. But worthy of an experiment.

-- CrawfordCurrie - 14 Nov 2009

Right. I (and I'm sure most others) have pondered how hash variables for wiki apps might be done. My thought experiments always went along the lines of an extension to the * Set (and META:PREFERENCE) syntax. Accessing values mimicking existing programming languages, Eg. %FOO['bar']%

-- PaulHarvey - 14 Nov 2009

Compiling templates hasn't been proposed or even specified anywhere yet. I don't think that %SET and %GET add more complexity to this job than there already is.

For example we already have a problem to distinguish two sorts of TMPL:P: one that is fully expandable during perl compile time, i.e. it does not depend on any sideeffects. The other one is a parametrized TMPL:P which does not get during template expansion. This latter one is acrually just a normal macro that only looks like the template equivalent. This construction shows that there are already problems to find out how deep a template can already be preprocessed.

While I share the concerns that people might not understand which value a variable has at what point of the rendering pipeline, I see this as more or less of the same complexity that every programming language has. I also think that %SET and %GET are intuitive enough so that people will make use of it.

-- MichaelDaum - 14 Nov 2009

If %SET and %GET are parsed inside-out, just like all other macros, then I should be able to do something like this to add a new value to an existing list:

%SET{"MYLIST" value="newValue" %GET{"MYLIST" format="value=\"$item\"" separator=" "}% }%

That would be very useful indeed!

I am not sure that it is clear yet where %SET (and %GET) are intended to have effect. * Set takes effect in SitePreferences, WebPreferences, user-topic and the topic being rendered. Should %SET be processed in all of these locations, too? If so, then there may be a performance hit, because inside-out processing must be applied to each of those topics, every time. If any of those topics use an expensive %MACRO, then all other topics are also affected.

Putting %SET together with %FOREACH, %SEARCH, and %INCLUDE - I will be able to shoot many more holes in my feet

-- MichaelTempest - 14 Nov 2009

Well, %SET and %GET are intended to be processed in the normal parsing flow. I can't see the performance problem. They could be implemented in a simple plugin interfacing with the core via Foswiki::Func::setPreverenceValue() and Foswiki::Func::getPreferenceValue() ... except for the multi-value attribute parameters. This needs a core change to Foswiki::Attrs.

Appending values to a list is an interesting problem that might need to rethink the current %SET syntax.

-- MichaelDaum - 14 Nov 2009

If the only %SET and %GET that are processed are those in the current topic (directly, or via %SEARCH, %INCLUDE, etc), then there is no performance problem.

There may be a performance problem if %SET and %GET in SitePreferences (or any of the other topics from which preferences are extracted) are also processed when rendering other topics. But I think you said this is not the case, in which case there is no performance problem.

-- MichaelTempest - 15 Nov 2009

* Set is statically evaluated, so of course a * Set from a %INCLUDE is not evaluated as it is outside the scope of static evaluation. %SET on the other hand, would be evaluated, allowing you to use it in a %INCLUDE. Could be interesting.

-- CrawfordCurrie - 16 Nov 2009

AcceptedBy14DayRule ?

-- MichaelDaum - 30 Nov 2009

as I mentioned in CleanerSyntaxForMetaDataAccess, this proposal becomes much simpler when existing core functionality is used. SET need to be implemented, but GET can be done using a natural combination of TOM syntax and the in trunk FOREACH macro (which I originally called FORMAT, as that is what it really is).

the todo for GET becomes adding a little more clever-ness to the Format refactor, which I'm quite happy to do.

oh, and, yes, if you make the SET, the docco for it and the unit tests, I'll do the GET bits while I'm refactoring the foreach code, and we can call it extremely AcceptedBy14DayRule.

-- SvenDowideit - 25 Feb 2010

Sven, please let me do this job! I don't want to back off to unit tests and docu only.

Part of it is allowing multiple values in an attribute list of the same name, which GET then fetches and formats.

-- MichaelDaum - 25 Feb 2010

I'm not stopping you in any way, I'm only offering you a way to integrate the formatting into the rest of the core, and a shorter development path as we need to get 1.1 features done, and this one, dispite our common desire to see it, hasn't even started development.

-- SvenDowideit - 25 Feb 2010

Michael, when integrating your existing plugin to the core, please bear in mind that we have a number of ways to get a preference variable (implemented and proposed)

• %BLAH% (implemented)
  • Traditional, current scope only
• %QUERY{"$ BLAH"}% (proposed)
  • from CleanerSyntaxForMetaDataAccess (I used GET in the proposal, but will refer to QUERY here to avoid a clash with your version of GET)
  • limited to variables in the current scope, and also picks up URL params
• %EXPAND{"BLAH"}% (implemented in trunk)
  • from http://trunk.foswiki.org/bin/view/System.VarEXPAND (trunk)
  • Controllable scope, this one seems to overlap most with this proposal
• %SHOWPREFERENCE% (implemented in trunk)
  • from http://trunk.foswiki.org/bin/view/System.VarSHOWPREFERENCE
  • debugging tool, shows where preferences are defined

This proposal adds

• %GET{"BLAH"}% (proposed)

We're getting carried away. Let's reset a minute, and examine our real requirements. We want:

1. to maintain compatibility with existing SEARCH, META and FORMFIELD macros
2. to be able to debug where a preference is defined
3. to be able to retrieve the value of a meta-datum
4. to be able to retrieve the value of a meta-datum in a different topic
5. to be able to retrieve the raw value of a preference in the current scope
6. to be able to retrieve the raw value of a preference in a different scope (another topic)
7. to be able to set the raw value of a preference
8. to be able to post-process the value of a preference for display (e.g. s/\n/
   /g)
9. to be able to post-process the value of a meta-datum for display

If I missed any, please add them.

Now we have a bunch of implementations and proposals that address some of these requirements (Y = full support, N = no support, though you might have expected it, , p = partial (subset) support, blank = no applicable)

Macro	1	2	3	4	5	6	7	8	9
META	Y		p	N					p
FORMFIELD	Y			p	p				p
SEARCH	Y		p	p	p	p	p	Y	Y
SHOWPREFERENCE	Y	Y
QUERY	Y		Y	Y	Y	N
SET	Y						Y
GET	Y				Y	N		Y
EXPAND	Y				Y	Y		N
FOREACH	Y							Y	Y

If I missed any, please add them. There seems to be a lot of overlap and a holes in the preference getting support, and in the value formatting.

Michael, as I understand it, Sven is saying that some combinations of these macros can simplify this for us (ignoring META, FORMFIELD and SEARCH), we can combine rows:

Macro	3	4	5	6	8	9
QUERY + FOREACH	Y	Y	Y		Y	Y
GET + EXPAND + FOREACH			Y	Y	Y	Y

Sven, in this proposal, Michael is (by implication) putting forward the case that all macros that recover preferences/meta should support the format header etc. params. So, the conflict I see here (if there is one) is between there two points of view - post processing for formatting, or formatting in the core macro.

Personally I see the FOREACH approach as the cleanest and most general. However I have concerns about the inside-out-left-right order of evaluation, and the use of comma-separated lists:

1. What happens when the value (preference or meta) contains a macro call?
2. What happens when a value contains a comma?

So pragmatically, Michael's approach appears to be the low hanging fruit. It also in principle deals with some of the corner cases that FOREACH misses, such as newline, though there doesn't appear to be any shareable code that implements such features.

We need some serious thought on this. I'm going to spend a few hours today experimenting with FOREACH to see where the holes are (if there are any)

-- CrawfordCurrie - 26 Feb 2010

You're still missing what the extractFormat feature is all about. FOREACH is only one way to access the formating code. The next part of that refactor - which is scheduled for 2.0, is to use that same code as a Foswiki::Func and internally for all uses of format, header, sort, paging etc.

not just Macros that recover preferences, but all macros that have output.

What I'd like to do, is have Michael use and extend the existing code which is destined to become the one Macros formatting system, rather than duplicating, and making something that is subtly different from it.

I really don't think we should force our users to learn 2 'master' macros to access TOM addressable elements, nor do I beleive that its to our advantage to add another format codebase.

(please make unit tests fo your experimentation )

-- SvenDowideit - 27 Feb 2010

First of all: I am not integrating SetVariablePlugin with this work. SET is not meant to provide persistence. It is meant to be a TML conform way of saying <space><space><space>* Set FOO = BAR. If at all this is comparable with %CALC{"$SET()"}%, which does establish a namespace of variables completely separate from preference variables. %CALC in itself is a beast as its evaluation order conflicts with the rest of the TML language in many places, thus rendering it a PITA to get sorted out, with users working around evaluation artifacts more than getting real work done.

Second: there is a severe limitation in Foswiki::Attrs only allowing one value for a parameter of the same name. For example, the value of param in %INCLUDE{"topic" param="bla" param="blub"}% is not specified formally. It only happens to be one of the two (dunno by heart which one, i guess the last one wins). Having params of the same name to a parametrized function call is however a missing feature when you'd like to pass lists. Today, you'd format a list value as comma separated list and then separate the list again in the body of the called function. That's awful.

That's why I'd like to extend Foswiki::Attrs to capture all values of param in a list instead of overriding the last definition. Overriding the former value bla with blub would still work as is for backwards compatibility.

Now, to retrieve all values of param, we need a kind of access function. I proposed GET for that.

Note, that the nature of SET and GET is

• to only operate on the namespace of preference variables, not meta data of any other kind

and

• to modify variables on "session" level, that is not on user, web, plugin, or topic level.

The required functions are all available in the Foswiki::Func API already, that is setSessionValue() and getPreferenceValue().

So far a straight-forward quite simple contribution to the core.

Both EXPAND and QUERY weren't on the radar when I proposed this.

A similar extension to Attrs has been proposed on the other project recently.

-- MichaelDaum - 27 Feb 2010

OK, that's what I understood. I appreciate that EXPAND and QUERY weren't on the radar, and I don't actually think QUERY is relevent to this discussion (except where it impinges on the FOREACH debate). EXPAND fulfils a different function, to expand a macro (not just a preference) in variable scope.

So, the difference between %GET{"BLAH"}% and %BLAH% is the ability to format the result of the GET as a list? So we come back to the debate regarding a functional FOREACH approach versus format support in individual macros. Can you comment on this please?

With regard to lists; I don't have a problem with extending Attrs to support multiple values. Note that perl semantics when assigning a list to a scalar is for the last value in the list to be taken as the scalar value. I assume we are talking here about the scalar value of a list value being the join(',',@list), which raises the spectre of commas in values again. Preference values are stored as scalars, so this is the same issue as bites FOREACH, and I don't see that GET is any better that FOREACH in this respect. How do you propose to deal with this?

-- CrawfordCurrie - 28 Feb 2010

From what I know about FOREACH so far, it tries to unify formatting by proposing constructions like this:

%FOREACH{
   "%INNERMACRO%" 
  format="..." 
  header="..." 
  footer="..." 
  separator="..."
}%

%INNERMACRO% must format its list result separating it using comas to please %FOREACH%.

(BTW: a list like "orange , apples" fails to split the list in a robust way; there's no way to define a split pattern or any other means to decompose the result of INNERMACRO into a proper list. There are other shocking weirdnesses in the code. Let's put that all aside for a moment as I don't want to discuss this here.)

In general, the way INNERMACRO and FOREACH pass around the list - that is via text substitutions on the wiki application level - is horrendously inefficient.

Because:

• (1) the INNERMACRO presumably already has got some sort of perl structure representing a list
• (2) it then stringifies it to a list to return its result
• (3) finally FOREACH parses this string and converts it back to a list for the purpose to reformat the result in a different way.

Fragile, error prone, inefficient.

INNERMACRO should use internal APIs to format its list results right away using the FFs.

Even better: instead of communicating a list using an inefficient stringification and decomposing cycle in the middle, the internal list object is bound to a %VARNAME% (in session layer) and GET (or FORMAT) can access it directly using its name as an id.

%GET{"VARNAME" <- not %VARNAME% !!!
  header="..."
  format="..."
  separator="..."
  footer="..."
}%

A plain %VARNAME% - without a GET would expand to a comma separated list for simplicity given it is a list object.

-- MichaelDaum - 28 Feb 2010

Very good points, and your observations about FOREACH are accurate, but you have just moved the list parsing problem back a stage. %GET still has to recognise where a preference represents a list. At the moment if I define:

• Set BLAH = blah , blah ,blah

Then I get a preference with a value "blah , blah ,blah". What should I expect if I %GET{"BLAH" format="$item" separator=","}% (or, for that matter, %FOREACH{"%BLAH%" format="$item" separator=","}%? How do I express a list that contains a value that contains a comma?

Sven has proposed standardising the code that handles format, header and I full support that idea. So we can reasonably assume that %GET and %FOREACH will end up using the same code to generate their output. The question remains, though, as to what code they use to parse their input.

-- CrawfordCurrie - 28 Feb 2010

Right. But the point is to prevent having to parse the input back and forth. I want MACROS to talk to each other more directly and process result sets internally, instead of going through the TML parser.

Now, there are some lists that do need to be parsed. FOREACH does not provide sufficient means to do so. FORMATLIST of FilterPlugin does and has been proven to be very flexible for a lot of different kinds of lists. However, it lacks storing these lists in a shared namespace for post processing.

But, there are also some lists that better are not stringified and remain lists internally, represented by a result set ID. We should work on specifying this as a means to process these lists repeated times.

Crawford, you've done something of that kind in FormQueryPlugin? . SolrPlugin does it as well. So it is time to get a good result set framework in place.

-- MichaelDaum - 28 Feb 2010

We all want lists handled internally, and passed between macros and stored in preferences. But we are quite a long way from that. Until we are a lot closer, we are faced with the question of "what is the string representation of a list?", and that is the key *question I'm trying to get you to help answer. Right now, by ignoring/fudging around the question, we are faced with lists that are defined as: split( /\s*,\s*/ ) - which of course precludes the use of commas in tokens. Maybe that's sufficient; if it is, we need to standardise on it, in the same way as we standardised on the formatting tokens.

-- CrawfordCurrie - 28 Feb 2010

Only allowing lists to that simple kind is very limiting.

Let's recap the docu of FORMATLIST from FilterPlugin as that's the most advanced list parser that we have atm.

(Not that I think that this is a discussion that belongs here. However you asked for not "fudging around the question". So here you are.)

%FORMATLIST{
   "some list"
   <list parser arguments>
   <list filter arguments>
   <list formatting arguments>
}%

List parser arguments are:

• tokenize="...": regex to tokenize the list before spliting it up, tokens are inserted back again after the split stage has been passed
• split="...": the split expression (default ",")

List post processing and filter arguments

• exclude="...": remove list items that match this regular expression
• include="...": remove list items that don't match this regular expression
• limit="...": max number of items to be taken out of the list (default "-1")
• skip="...": number of list items to skip, not adding them to the result
• sort="on,off,alpha,num,nocase" order of the formatted items (default "off")
• reverse="on,off": reverse the sortion of the list
• unique="on,off": remove dupplicates from the list

List formatting arguments:

• pattern="...": pattern applied to each item (default "\s(.*)\s")
• format="...": the format string for each item (default "$1")
• header="...": header string
• footer="...": footer string
• separator="...": string to be inserted between list items
• selection="...": regular expression that a list item must match to be "selected"; if this matches the $marker is inserted
• marker="...": string to be inserted when the selection regex matches; this will be inserted at the position $marker as

Here's what happens inside:

1. list construction phase
   1. all standard escapes in the list are replaced and the result is executed (expandCommonVariables)
   2. the list is processed using the tokenize parameter by temporarily removing matching substrings before proceeding to the next step
   3. the list is split according to the split pattern
   4. tokens gathered in step (2) are inserted back into the list
2. list processing phase
   1. the list is sorted according to the sort parameters
   2. if specified its order is reverted
   3. any unwanted list items not matching include or matching exclude are removed from the list
   4. if unique was switched on, duplicate items are removed.
   5. the list is reduced to the specified sublist as specified by skip and limit
3. output phase
   1. for each list item pattern is matched and grouping is applied as specified in the pattern parameter.
   2. each item is formatted using the format parameter and pushed onto a result stack; groups caputred via pattern are inserted via $1, $2, ...
   3. if a list item matches the selection expression marker is inserted (we've got that in WEBLIST as well; avaliable here for more general purposes)
   4. the result stuck is joined using the join parameter and a header and footer is added

Here's a more interesting list and a best-practice for using FORMATLIST as well:

%FORMATLIST{"
    "oranges=2;apples=15;grapes=1,5"
   split="\s*;\s*"
   pattern="(.*)=(.*)"
   format="   * $1 is $2"
   separator="$n"
   footer="$n found $count items"
}%

There are a lot of situations where you can encode lists this way with items consisting of tupples of information bits that belong together.

Another interesting thing to work around list separators that might occur in the list items as well is the tokenize parameter. This can be used to sort of protect items from being misinterpreted as separate list items.

What would make a helluffa sense is to split out a PARSELIST from FORMATLIST and add an id parameter to both. So PARSELIST could store a list under the given id and multiple FORMATLISTs could format it in various ways, multiple times.

In fact, I'd like to see is namespace of IDs being shared among all macros. Behind a specific ID we have a result object with a standard interface to iterate over all contained items. For ease of use we could use TIEs for that. But that's already too impl specific. Let's stop this here and see if we can formally define ResultSets.

One central design factor is to hide lazy retrieval inside a result set, that is: never ever fetch all of the result set in advance trying to load it all into memory. That's what SEARCH in trunk currently does - a cardinal flaw.

-- MichaelDaum - 28 Feb 2010

We have talked about result sets before in SearchResultsPagination.

-- ArthurClemens - 28 Feb 2010

That's only covering the output phase of some search specific result structure.

-- MichaelDaum - 28 Feb 2010

I agree that that topic shows a limited view of what has been in our heads, and you have summarized it above:

• Querying involves handling lists, or collections (ordered or unordered)
• Collections involve a result set, then a sort step, a filter step and a display step
• Result sets should be stored (memory/db/disk) for quick retrieval or to manipulate them

-- ArthurClemens - 28 Feb 2010