Item438: replace horrid UTF8 regex's and replace with Encode

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Normal	Being Worked On	Engine		Foswiki:Main.OlivierRaginel

from Tasks.Item401

I'm just wondering (and apparently I'm not the only one), why we're using a regexp where we could directly use Encode.

I know Encode is another module to require, thus another piece of code that gets loaded, but anyway some modules already require it (such as Wysiwyg).

To my humble opinion, if we want to go UTF-8, we will have to use some proper tool to do it, and thus Encode seems the appropriate choice.

Re-inventing the wheel using regexp can work, but...

Also, Encode uses XS, thus is much quicker than a regexp to achieve the same.

• OlivierRaginel - 08 Dec 2008

Funny: http://develop.twiki.org/trac/changeset/17776 Item6146: Adding Encode as a required CPAN module

Encode was first released with perl 5.007003 (patchlevel perl/15039, released on 2002-03-05)

But according to people using it, it makes no sense doing UTF-8 with anything older than perl 5.8.3.

• OlivierRaginel - 09 Dec 2008

switching to Encode is a very sane sounding thing

-- SvenDowideit - 12 Dec 2008

Just for the record, I've contacted RichardDonkin for inputs on this.

My plan is to re-write most encodings (Check Item 5437), using Encode, for all encoding issues.

But first I need input from the big players

-- OlivierRaginel - 12 Dec 2008

Got input from SvenDowideit and RichardDonkin, thus I can start working on it.

Planning this for 1.1 anyway, so first bug fixes for 1.0

-- OlivierRaginel - 15 Dec 2008

No commits against 1.1.0 - bumping this up to a major release.

-- GeorgeClark - 13 Mar 2011