Feature Proposal: add match operator to query language

Motivation

While there's a like operator ~ there's no regex match. For instance, this is needed to search for proper words inside a string (see examples below).

Description and Documentation

Until now, we did not implement this obvious operator as we assumed that it would make it harder to translate the query language to SQL ... once we decide to do. However, today's database engines do have support for posix regex. So I'd consider this a thin argument not to have this very useful operator.

Examples

%IF{
  "'foo bar baz' =~ '\bbar\b'"
  then="yes"
  else="no"
}% -> yes

%IF{
  "'foo barbaz' =~ '\bbar\b'"
  then="yes"
  else="no"
}% -> no

%IF{" 
   Tag =~ '\bbased-on-novel\b'" 
   then="This movie is based on a novel"
}%

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

package Foswiki::Query::OP_match;

use strict;

use Foswiki::Query::BinaryOP ();
our @ISA = ('Foswiki::Query::BinaryOP');

sub new {
    my $class = shift;
    return $class->SUPER::new( name => '=~', prec => 500 );
}

sub evaluate {
    my $this = shift;
    my $node = shift;

    return $this->evalTest(
        $node,
        \@_,
        sub {
            my $regex;
            eval { $regex = qr/$_[1]/ };
            if ($@) {
              throw Foswiki::Infix::Error( 'illegal regex', $_[1] );
            }
            defined( $_[0] ) && defined( $_[1] )
            && $_[0] =~ m/$regex/s ? 1 : 0;
        }
    );
}


1;

All unit tests in Fn_IF then pass.

-- Contributors: MichaelDaum - 26 Mar 2010

Discussion

Btw, here's an interesting link comparing different flavors of regular expressions and their database support.

-- MichaelDaum - 26 Mar 2010

Absolutely we should allow them everywhere. Otherwise we're back in the twiki hell of things working in some places but not in others. For consistency I'd actually prefer they worked in attr strings as well, but that's maybe a step too far.

I seem to recall raising this issue of differing regex support; IIRC you assured me that "everyone supports POSIX".

-- CrawfordCurrie - 26 Mar 2010

Just to clarify.

This is a proposal to add regex operator to query. We already have plain regex search type, but it looks everywhere in topic text.

This will enable targeted regex where we only match regex to a specific form field or parent or whatever AND can combine this with AND and OR and additional conditions.

Great enhancement of query. Very visible enhancement to application builders.

The rest of the discussions about the escaping is an existing bug/problem in query which just becomes more important with regexes.

So people should ignore the escape discussion and only judge the proposal on the addition of =~ operator.

I recommend not raising concern. There are two top developers on it, the feature is very important, and further enhance our feature set compared to the old project. So bear over with the late arrival. I let this pass as an exception because the code is already available and both Michael and Crawford implement it so it will be stable for feature freeze, and because it is a killer feature.

-- KennethLavrsen - 26 Mar 2010

excellent.

I presume there will be docco showcasing this new operator in a query SEARCH too. and a unit test - to support the non-core QueryAlgorithm implementors.

-- SvenDowideit - 27 Mar 2010

By that you mean a unit test for the operator in the context of a SEARCH? I'd regard that as essential, in the same way as a unit test for IF is essential. At the moment we have unit tests for the query syntax, but none that apply to "real" topics.

Note that one thing this test must check is the compatibility of the regular expressions. I believe it's important that the RE match in query searches observes the same RE syntax as type="regex". There are many flavours of regular expression (see http://www.regular-expressions.info/refflavors.html) and there is no lowest common denominator. Notably Javascript has elected to support Perl syntax regexes, and the PCRE lib means they are supported by many C programs as well (this is what NativeSearchContrib uses).