Item9814: Default SEARCH result order includes an implicit sort on the web= parameter

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Urgent	Being Worked On	Engine	SEARCH

On 09/10/10 03:11, Crawford Currie wrote:

> > While trying to test the DBIStoreContrib, I noticed that some of the
> > testcases (e.g. Fn_SEARCH::test_paging_with_limit_*) rely on the
> > ordering of the web= parameter. Specifically, if it's
> > web="System,Main,Sandbox" then the results are returned in that order,
> > even though the default sort is alphabetic, and "Main,Sandbox,System" is
> > implied (and documented for topics, though not for webs).
> >
> > I don't think this is right. The order of the results should not depend
> > on the order of the web names in the web= parameter, any more than it
> > depends on the order of the topic names. Raising to you because you
> > wrote these testcases, and may have other ideas. If this is designed
> > behaviour, then I can always post-sort the SQL search results, but it's
> > really counter intuitive to have to do that.

On 09/10/10 05:40, Sven Dowideit wrote:

> looking at it in the code, I didn't break the 'default' if the user
> doesn't set an order.
>
> rather I broke the (also) undefined second level ordering when the user
> specifies the order to be something other than topic - and this is
> noticable when the requested order key is non-unique.
>
> not urgent, but annoying - as we either have to force all store to order
> by %USERREQUESTEDORDER%, topicname
>
> or accept that some stores have different internal order
>
> same thing for webs - if any less steeped in the technical details of
> the implementation have an opinion, I'd like to hear it now, so we can
> just set one or the other as 'correct'

The problem is related to ordering - default and explicit.

Way back in time, (tm)wiki was lazy about search ordering in the default cases. The search algorithm would visit one web at a time, and the results would be presented in the order the webs were visited. This behaviour was never documented. Also, because of the way the algorithm worked, topics would be returned in the sub-order that they were specified in the topic= parameter. This was actually in direct contradiction of the documentation, and has problems when wildcard lists of topic names are used (it requires a full enumeration of topic names to establish the ordering).

The importance of this ordering becomes apparent when you consider a search like this:

>%<nop?SEARCH{"blah" web="WebC,WebA,WebB" topic="TopicC,TopicA,TopicB"}%

VarSEARCH suggests an alphanumeric ordering, which should return the order WebA.TopicA, WebA.TopicB, WebA.TopicC, WebB.TopicA, WebB.TopicB, WebB.TopicC, WebC.TopicA, WebC.TopicB, WebC.TopicC. However we would actually see WebC.TopicC, WebC.TopicA, WebC.TopicB, WebA.TopicC, WebA.TopicA, WebA.TopicB, WebB.TopicC, WebB.TopicA, WebB.TopicB

On the face of it this is quite seductive - you can control the ordering of results using the web= and topic= parameters.

So, the question is, how important is this behaviour? If it's important, then it:

1. needs to be covered in unit tests
2. needs to be documented
3. needs to be fully implemented
4. will be a comlpete bastard to maintain when we expand order= and groupby= to support multi-level sort.

We also need a decision about how to handle duplicate topic names and web names in the web= and topic= lists. Does:

%SEARCH{"blah" web="WebA,WebA"}%

Return each result in WebA twice? Again, this would a be a right bastard to maintain.

%SEARCH is such an important macro that we can't afford to be vague about this.

I'm going to change it so that duplicates are ignored and there is only one default ordering - alphanumeric, with web being the major sort key and topic the minor key - and all other orderings should have to be explicitly stated in the order= and groupby= parameters (TBD)

-- CrawfordCurrie - 09 Oct 2010