versions query w/SEARCH, RCS store = hang
I've not had much luck with the
query in SEARCH
It works fine for individual topics with QUERY
, but SEARCH
is just too slow unless you only have a few dozen topics.
I think we need a strategy before we ship this. At a minimum, disable
with RCS stores, or alternatively we introduce a new feature to make SEARCHes abort long-running queries in a graceful manner (can we just return a shorter-than-normal resultset, and log/emit an error?).
The query I ran on trunk.foswiki.org was something like:
Which ran for several minutes before Koen killed the process for me.
- 11 Dec 2010
Yeah, well, TBH I'm not that surprised. A versions query has to load a shitload of information just to query, and that's not efficient. As you say, the RCS store just isn't built for this kind of query.
Should this be a fix just for a versions query, or is there a more general problem, that a user should be able to put a limit on the amount of time spent on a query? A general mechanism would work for other types of bad query.
- 11 Apr 2011
I don't have any good ideas on how to control a hypothetical "search timeout" feature. Configure setting? Macro param? URL param? What should be the default?
On my own wiki, I have some reports that just take 10s of seconds, and a dot graph that takes minutes - I need to be able to run those from a cron job, where I save the output back into "cache" topic which is refreshed every hour.
Let's say we have a default SEARCH
timeout of 10s - I need a way for my cron-job scenario to override it so that it will run to completion from CLI.
Additionally: I don't know about general Foswiki practice, but the biggest troubles on my wiki are nested searches. If there's 200 SEARCHes as a result of an outer SEARCH
, how do we apply a "timeout" in that case, if each individual SEARCH
is still on the order of ~2s?
Hmm, so at first glance it seems that a general solution is a can of worms.
If I get time to invest in this, I think I'd rather try to ship a SearchAlgorithm which continues to work with RCS store but perhaps caches just the
part of every topic version in
somehow (reproducing the
directory layout, but
would be a directory)
- 11 Apr 2011
I have added a caveat emptor
to the QuerySearch
topic to warn of the performance risks. Having done this I think it is valid to regrade this report from 'Urgent' to 'Enhancement'.
- 21 Jan 2013
I think that caching of the RCS log information plus the %META would be good as a basic feature of the RCS based Store. Anything that needs to dip into the rcs log records, for ex. the Attachment display of the comment field, is horribly slow. Avoiding RCS to access the topic metadata history without a full rcs pass would be very helpful. Maybe store it along side the
- 21 Jan 2013