Feature Proposal: Simply SEARCH parameters

Motivation

combining web and topic or excludetopics doesn't quite work.

Its actually impossible to search 3 webs, and only exclude the WebHome topic of one of them, because the spec of topic and excludetopic are topic name only..

web="Name" <br /> web="Main, Know" <br /> web="all"=	Comma-separated list of webs to search. You can specifically exclude webs from an all search using a minus sign - for example, =web="all,-Secretweb". The special word all means all webs that do not have the NOSEARCHALL preference set to on in their WebPreferences. Note that AccessControls? are respected when searching webs; it is much better to use them than NOSEARCHALL.	Current web
topic="WebPreferences" <br /> topic="*Bug"	Limit search to topics: A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names.	All topics in a web
excludetopic="Web*" <br /> excludetopic="WebHome, <nop>WebChanges"	Exclude topics from search: A topic, a topic with asterisk wildcards, or a list of topics separated by comma. Note this is a list of topic names and must not include web names.	None

Description and Documentation

I'm refactoring the code area to use iterators, and it looks to me like the restriction is totally arbitrary - I expect to find other such un-necessary limitations over time.

Examples

Impact

Implementation

I'm not 100% sure yet - mostly, I'm migrating the arrays of topicnames into iterators of web.topic names - though I'd love to have them be iterators of nodes ....

-- Contributors: SvenDowideit - 25 Mar 2009

Discussion

Heh, I wished you'd rant about negotiated parameters as well, e.g. nonoise and friends.

-- MichaelDaum - 25 Mar 2009

you're welcome to rant about them, and even to do something about them. If there is a nice analysis of how we can replace them while still allowing existing users to cross grade, I'm sure we'd all welcome it...

-- SvenDowideit - 25 Mar 2009

For a much scarier problem, I need to remove&replace Foswiki::Func::searchInWebContent. it has a totally grep based interface, and relies on an implementation detail that isn't going to grow with us.

for reference:

=begin TML

---+++ searchInWebContent($searchString, $web, \@topics, \%options ) -> \%map

Search for a string in the content of a web. The search is over all content, including meta-data. Meta-data matches will be returned as formatted lines within the topic content (meta-data matches are returned as lines of the format %META:\w+{.*}%)
   * =$searchString= - the search string, in egrep format
   * =$web= - The web to search in
   * =\@topics= - reference to a list of topics to search
   * =\%option= - reference to an options hash
The =\%options= hash may contain the following options:
   * =type= - if =regex= will perform a egrep-syntax RE search (default '')
   * =casesensitive= - false to ignore case (defaulkt true)
   * =files_without_match= - true to return files only (default false). If =files_without_match= is specified, it will return on the first match in each topic (i.e. it will return only one match per topic, and will not return matching lines).

The return value is a reference to a hash which maps each matching topic
name to a list of the lines in that topic that matched the search,
as would be returned by 'grep'.

To iterate over the returned topics use:
<verbatim>
my $result = Foswiki::Func::searchInWebContent( "Slimy Toad", $web, \@topics,
   { casesensitive => 0, files_without_match => 0 } );
foreach my $topic (keys %$result ) {
   foreach my $matching_line ( @{$result->{$topic}} ) {
      ...etc
</verbatim>


sub searchInWebContent {

    my ( $searchString, $web, $topics, $options ) = @_;
    ASSERT($Foswiki::Plugins::SESSION) if DEBUG;
    my $webObject = Foswiki::Meta->new( $Foswiki::Plugins::SESSION, $web );
    return $webObject->searchInText( $searchString, $topics, $options );
}

I intend to replace it with something that will allow the Extension developer to call query searches, and to allow the full ResultSet? enabled power..

-- SvenDowideit - 26 Mar 2009

Noone raised concern.

And as far as I can see the proposal at the top is backwards compatible. So no concern from me either.

Accepted by 14-day rule

-- KennethLavrsen - 12 Apr 2009

I'm pretty sure this was implemented last year - but I will need to add some more unit tests to be sure its how we want it.

-- SvenDowideit - 31 Mar 2010

Wow, exactly one year later! Anyway, the proposal text describes something which hasn't been done yet: allowing fully-qualified "Web.Topic" names in the topic= and excludetopic= parameters.

At work we have this problem where we have things that are best described as "pick lists", that end up being a list of web.topic names exceeding 4,000 items. To search on that list, there's a couple of solutions if the topics are all within the same web:

   1 %SEARCH{"Colour='Red' AND (name='Item1' OR name='Item2' .... OR name='Item4000')" type="query"}%
   2 %SEARCH{"Colour='Red' AND name IN (item1, item2, ... item4000)" type="query"}%
   3 %SEARCH{"Colour='Red'" type="query" topic="Item1,Item2,...Item4000"}%

Using this exact query structure, querying a topic list of 1,000 topics I obtained the following timings with ab n -4 and MongoDBPlugin Foswikirev:11733 on a web of 25,000 topics:

1. ~5.2s
2. n/a (MongoDBPlugin doesn't do IN yet)
3. ~5.2s

And a dumb query which looked simply like: Colour='Red' over the entire 25,000 topics responded in ~0.8s.

Anyway, the issue is that VarSEARCH's topic parameter doesn't understand fully-qualified web.topic names yet (and QuerySearch provides no fully-qualified webtopic name to query against either, I guess I can do web + '.' + name), and this would be really nice to have.

-- PaulHarvey - 31 May 2011