Item5351: WebCreateNewTopic filters out international characters

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Normal	Closed	Engine

WebCreateNewTopic filters out international Characters. While typing a new topic name international characters are automatically removed and the next character after the international one is capitalised. As capital letters have a special meaning in topic names this is quite confusing.

This sounds a bit like Item4419 but the solution suggested there does not solve the problem.

-- TWiki:Main/MartinMayer - 11 Feb 2008

The problem is two-fold:

• webtopiccreator.js in PatternSkin uses removePunctuation() to sanitize topic names
• removePunctuations (from TWikiJavascripts/twikiString.js) basically strips out all but MIXED_ALPHANUM_CHARS, which are [A-Za-z\d]

-- TWiki:Main.AndrewPantyukhin - 19 Apr 2008

The javascrripts use the same regexes as TWiki core. Updates to TWiki regexes should be reflected in the javascripts, but not the way around.

-- TWiki:Main.ArthurClemens - 19 Apr 2008

I don't know much JavaScript and I can't grasp the rationale behind twikiString.js. We've got character class definitions as global vars in twiki.js and unicode_chars.js. For some reason, though, twikiString.js does not use them, but introduces a curious OO-based definitions. It looks like someone was trying to do some refactoring, but never finished it. Anyway, twikiStringUnicodeChars.js is an attempt to convert unicode_chars.js to the same OO interface, but it only contains one class (UPPER_ALPHA_CHARS), and is not included from PatternSkin (while unicode_chars.js is).

I may be wrong, but I'd say we need to forget about the OO-thing and just run s/twiki.StringConstants.getInstance().// in twikiString.js. That way it'll use the complete definitions.

-- TWiki:Main.AndrewPantyukhin - 24 Apr 2008

This little patch fixes the problem: http://heka.cenkes.org/sat/diffs/non-ascii.wikiword.diff

-- TWiki:Main.AndrewPantyukhin - 29 Apr 2008

I applied Andrew's patch after an hour of testing

It works great. We non-English can again create I18N topics (non UTF8)

• KennethLavrsen - 24 Dec 2008

(from Japan:) This will break twiki.String because it now won't function without unicode_chars.js. Running TestCaseTWikiJavascripts will reveal the errors in the new code.

I see I did not finish foswikiStringUnicodeChars.js which is the proper replacement for unicode_chars.js (I did not include the lowercase chars).

Proper fix:

• use foswikiStringUnicodeChars.js
  • added documentation
  • updated test in TestCaseTWikiJavascripts
• delete unicode_chars.js

-- ArthurClemens - 26 Dec 2008