Item1363: port and enhance KinoSearch

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Normal	Closed	Extension	KinoSearchContrib	Main.AndrewJones

wbniv@r17311:~/foswiki.org/trunk$ core/tools/twiki2foswiki SearchEngineKinoSearchAddOn
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/System
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/SearchEngineKinoSearchPlugin.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/KinoSearch.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/SearchEngineKinoSearchAddOn.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki/VarKINOSEARCH.txt
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/data/TWiki
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Stringifier.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/KinoSearch.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Index.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifyBase.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/DEPENDENCIES
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/MANIFEST
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/HTML.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/Text.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/XLS.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/PDF.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_abiword.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_antiword.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/DOC_wv.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/PPT.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/build.pl
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/Config.spec
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Plugins/SearchEngineKinoSearchPlugin.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Plugins
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store/SearchAlgorithms/Kino.pm
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store/SearchAlgorithms
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Store
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/TWiki
A         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/System
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn/KinoSearchResult.jpg
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn/KinoSEARCH.jpg
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki/SearchEngineKinoSearchAddOn
D         /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/pub/TWiki
core/tools/twiki2foswiki: line 76: cd: /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Plugins/SearchEngineKinoSearchAddOn: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read MANIFEST: No such file or directory
sed: can't read build.pl: No such file or directory
sed: can't read build.pl: No such file or directory
sed: can't read DEPENDENCIES: No such file or directory

-- WillNorris - 24 Mar 2009

---

rewrote tests from using {session}->{store} to Foswiki::Func calls

GETTING CLOSER!!!

current state:

wbniv@r17311:~/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn$ perl build.pl test
Loaded config from /home/wbniv/.buildcontrib
Running tests in /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm
Running: perl -w -I /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /etc/perl -I /usr/local/lib/perl/5.8.8 -I /usr/local/share/perl/5.8.8 -I /usr/lib/perl5 -I /usr/share/perl5 -I /usr/lib/perl/5.8 -I /usr/share/perl/5.8 -I /usr/local/lib/site_perl -I . /home/wbniv/foswiki.org/trunk/core/test/bin/TestRunner.pl /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm
Options:
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Update started
Updating index finished
Update started
Updating index finished
Update started
Updating index finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Indexing started
Indexing finished
Use of uninitialized value in split at SearchTests.pm line 119.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 387.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 388.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 403.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 418.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 422.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 423.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 424.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 425.
Use of uninitialized value in substitution (s///) at /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib/Foswiki/Contrib/SearchEngineKinoSearchAddOn/Search.pm line 426.
Failed to perl -w -I /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/lib -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/ -I /home/wbniv/foswiki.org/trunk/core/lib/CPAN/lib/ -I /etc/perl -I /usr/local/lib/perl/5.8.8 -I /usr/local/share/perl/5.8.8 -I /usr/lib/perl5 -I /usr/share/perl5 -I /usr/lib/perl/5.8 -I /usr/share/perl/5.8 -I /usr/local/lib/site_perl -I . /home/wbniv/foswiki.org/trunk/core/test/bin/TestRunner.pl /home/wbniv/foswiki.org/trunk/SearchEngineKinoSearchAddOn/test/unit/SearchEngineKinoSearchAddOn/SearchEngineKinoSearchAddOnSuite.pm: 11 at /home/wbniv/foswiki.org/trunk/core/lib//Foswiki/Contrib/Build.pm line 820, <DATA> line 1.

-- WillNorris - 2 Jun 2009

We might also want to take some of the enhancements from TWiki.org (see http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item6177).

We can certainly take the extra stringifier's from http://develop.twiki.org/svn/twiki/trunk/SearchEngineKinoSearchAddOn/lib/TWiki/Contrib/SearchEngineKinoSearchAddOn/StringifierPlugins/. The other enhancements might need reviewing, as some of them either don't make sense or don't seem to be implemented in the best way.

I am quite happy to help out on this (and on Item8246), but I don't want to touch anything without checking with you first.

-- AndrewJones - 07 Sep 2009

Task list for porting and enhancing Kino:

• Get working under 01x00 and trunk
• Strinigifiers and tests for .docx, .xlsx, .pptx
• Try and get unit tests to work (need to learn a lot more about unit testing Foswiki)
• Move program paths into Configure (so we don't need to assume antiword, etc is in the path)
• Use Foswiki::Sandbox rather than =system=
• Improve the docs and default templates
• Check under FastGGI
• Integrating KinoSearch into Foswiki's internals =SEARCH=

Other tasks for the future, but might not be completed before the initial port

• Use rest handlers
  • Might have to wait until Foswiki 1.1, as in 1.0 rest handlers need to be authenticated (afaik)
• Remove the =%SPLIT%='s from templates

-- AndrewJones - 09 Sep 2009

Using KinoSearch as a SearchEngine might get problematic, since Kinosearch is - by design - not a fulltext search engine: ie. you simply cannot find substrings in tokens.

-- OliverKrueger - 10 Sep 2009

Yeah true, but it could fall back to the default search as needed (i.e. when using regex search).

It looks like a work in progress by SvenDowideit so I will leave it well alone for now.

-- AndrewJones - 10 Sep 2009

Still can't get the unit tests to work, but apart from that this is pretty much ready. Will do a bit more testing and try some more with the tests, but this should be released by the end of the week

-- AndrewJones - 14 Sep 2009

All tests passed (86)

-- AndrewJones - 24 Sep 2009

After 64 checkins, the Kino Search is now ported as KinoSearchContrib and KinoSearchPlugin.

I posted an announcement on the Foswiki blog.

Closing the task.

-- AndrewJones - 24 Sep 2009

Removed old SearchEngineKinoSearchAddOn from svn.

-- AndrewJones - 28 Sep 2009