Item12180: Implementation for AJAXOnDemandCheckersForConfigure

This is the implementation item for AJAXOnDemandCheckersForConfigure.

-- TimotheLitt - 22 Oct 2012

The main commit is rev 15847. There are still a few loose ends, but this seems functional enough to share. It turned into a somewhat larger project... Note also that this depends on several other items that were broken out so they could be considered for patch releases.

Thanks to everyone who was patient during the time it took to get everything sorted out, and special thanks to George Clark for his testing of intermediate code drops and his encouragement.

I'll address the loose ends list and any other discoveries as they come up, but don't anticipate another commit on this scale...

-- TimotheLitt - 03 Nov 2012

On my computer I am getting deep recursion errors:

configure: Deep recursion on subroutine "Foswiki::Configure::Checkers::PATH::check" at /Users/arthur/Sites/foswiki.trunk/core/lib/Foswiki/Configure/Checkers/ScriptDir.pm line 19.
configure: Deep recursion on subroutine "Foswiki::Configure::Checkers::ScriptDir::provideFeedback" at /Users/arthur/Sites/foswiki.trunk/core/lib/Foswiki/Configure/Checkers/PATH.pm line 67.

Mac OS X 10.7.5, perl 5.16.0.

-- ArthurClemens - 03 Nov 2012

Recursion is fixed.

-- ArthurClemens - 03 Nov 2012

Feedback.pm -> sub deliver: this has text and html hardwired in the backend, and should be using a template. It is very hard to put the message somewhere else now.

-- ArthurClemens - 03 Nov 2012

Sorry about the recursion problem. I think your Foswiki.spec got mangled in an svn update. But it shouldn't have caused the infinite recursion. You should check your .spec file. The recursion is fixed.

Feedback.pm is primarily a transport mechanism for messages generated by others (provideFeedback routines.)

It does have a few error cases and one status case - the unsaved items message you asked for - where it generates messages.

I haven't seen documentation on the template mechanism, nor am I really anxious to learn yet another pseuo-scripting language.

If you produce templates that suit you, I'm happy to call the parser routines with the right argument lists.

Note that what's happening on the other end is replacement of <div> contents (and sometimes element values).

So the templates do have to contain (or be contained) in a compatible wrapper.

The placement on the screen is entirely controlled by the existing templates; Feedback.pm simply supplies the replacement data.

Oh, except for a class of error data - anything that doesn't look like the wire protocol initiates a modal error window; this allows html formatting for things like authentication timeouts (and in development, perl crashes).

Everything I did has a CSS class so you can adjust colors, placement, backgrounds, whatever. What I provided functions, but certainly is not elegant. I hope you can help improve the look.

Let me know how I can help.

-- TimotheLitt - 03 Nov 2012

I was just in the process of refactoring the templates to use Text::Template, so you just can use Perl inside templates.

As it is now, and what I want to keep, is that the backend code provides data to the template, and the template draws. The backend should not have to know about any html structure.

I have done this refactoring a previous time - pulling frontend code out of the backend - and it has been a hell of a job. I really want to keep this separated and clean.

The choice of templating engine was "better to use any templating than none". But the currently used FreeMarker templating is too slow, and we are all Perl developers, so Text::Template is a better choice.

For now you seem to understand the template basics, and it would help to use the current engine while I am working on the transition to Text::Template.

-- ArthurClemens - 03 Nov 2012

I support your goal.

There are two places where Feedback.pm creates text. One is trivial - a completely static message. I can handle that.

The other is where I conditionally say 'nothing pending" or "some unsaved" and for the latter, optionally which ones. (Which is very helpful for debugging.) That involves a conditional and an optional formatted list.

If you write a template for the latter, I will update Feedback.pm to use it and I'll do the static one. OK?

One small concern: this is invoked every time a checked field changes. E.g. if you click or tab out of a check-on-change field, this is run. So it needs to be efficient.

Two other things:

I created autocheck.png as a visual indication that a field is check-on-change. It's OK, but I don't think it's up to your standards for design. Perhaps you can come up with something better. If you change it, the size is hardcoded in legend.html and UIs/Value.pm (Yes, I know, it's not a template. But that's how that part of configure works.)

George noticed that your template changes dropped legend.html - was that intentional?

-- TimotheLitt - 03 Nov 2012

I took a stab at templating Feedback.pm. I still have a mystery to solve, but I guess you're off the hook for creating a template for me, though I doubt what I did is optimal.

-- TimotheLitt - 03 Nov 2012

Arthur: Your template changes have the Logout link and the "unsaved changes" status scrolling off the top of the screen.

This isn't good. The feedback status needs to be visible no matter what item you've scrolled down to change, and Logout needs to be easy to find - In a fixed place on all configure screens.

Remember that simply typing in a value box will update the status.

I see that you left a fixed bar on the bottom for "save changes" and "change password". How about moving these two items there?

Also, if I click on 'Change password' button on the Introduction screen, no boxes appear for the required input. (Nor are they on the bottom bar.) This needs to be fixed.

I also miss the Legend - I hope it's coming back.

Thanks for your efforts; I'm sure they'll improve things.

-- TimotheLitt - 03 Nov 2012

The interface is currently too unresponsive after changing. Line 1063 has a timer to postpone calculation, however this is only called after all interface elements have been searched. The blocking code starts at line 964 at $(KeyIdSelector).closest.

Ideally the function doFeedback should be refactored into smaller functions. There would be a separate function with $(KeyIdSelector).closest that can be called from a timeOut.

-- ArthurClemens - 07 Nov 2012

I keep breaking it down, it keeps growing back. It actually has a number of private functions, but it does get unweildy. The ajax closures add to the fun.

I added that timer for you!

The timer doesn't work as you describe. It has nothing to do with the current request.

It simply tells the NEXT update not to run. That only applies to status updates; actual feedback (by button click or field change) can't be deferred. (Well, we could have a queue, but that won't help, just complicate things.) It has to happen, and it has to be in order.

The actual AJAX calls are asynchronous. doFeedback just marshalls the POST data for the request, and unmarshalls (distributes) the results.

The code at KeyIdSelector scans upward from the button that's clicked to the form (usually not far), then scans just that form's controls to build the POST data. It is painful to convert to ASCII forms/multi-part, but that's the way it's done.

The work has to be done, and it has to be the state when the user clicks. The only thing we can do is to reduce the number of controls; e.g. by breaking the screen into separate forms - perhaps one for each section. But then there are challenges when unsaved state exists in more than one form.

I have done some work to minimize the amount of data transferred by dynamically disabling some controls, which helps.

I spent some time yesterday experimenting with compressing the POST data, but I don't think there's a good way to do that. (The compression is easy, but the browser-server channel doesn't like pure binary.) I had it working, except that some data would get lost. (And yes, it went into the Ajax call,l but didn't come out.)

Fixed in HTML.next (5, they say). But no good browser-tolerant mechanism that I can find works today.

As some consolation, remember that prior to this work, updates took seconds -- even minutes -- as configure had to rebuild the whole screen.

But I'm open to other ideas. And I'll keep thinking. This is pretty good, though perfect would be better ..

-- TimotheLitt - 08 Nov 2012

I have done some more optimizations of the client javascript - I didn't measure the results, but it does feel a little faster.

But I suspect the real delay is in talking to the server. Making configure persistent would help a lot. It would take some work to make it mod_perl-safe, but it doesn't seem impossible. Not today.

The controls should not be blocked. If you have a slow network/server, you could increase the timer value so the status updates stay out of the way a bit longer. It's currently set to 1.5 seconds - a more-or-less random value. Long enough to help, not so long that the lag in updating status is obvious.

If it makes a big difference for you, I could make it adaptive. I'd rather only add the complexity if it actually helps.

-- TimotheLitt - 08 Nov 2012

I have checked-in some experimental adaptive bandwidth management code, which adjusts update frequency based on sampled queue depths.

If I click as fast as possible on a checkbox, I can get about 10 changes into the default timeout period, which increases as a result - then glides downward as activity decreases.

The interface seems responsive with my initial settings, but there are some knobs to tweak. I doubt they'll make a huge difference.

Of course, when an actual check is done, it takes longer - quite a bit more Perl code gets loaded and executed. But that's why we have the spinner.

On another subject: I changed an icon and added a button to the status bar. The new icon came from glyphicons_free, though I changed it a little, but I'm not wedded to it. But I see you already saw that. Thanks!

-- TimotheLitt - 10 Nov 2012

Usability concerns: selecting/unselecting languages will immediately trigger actions on the language cache ... for each checkbox clicked. That's pretty annoying. Better would be to make a selection and then click a "go do it" button.

I also get a

Use of uninitialized value $Foswiki::cfg{"DataDir"} in concatenation (.) or string at ....foswiki/trunk/core/lib/Foswiki/Plugins/GenPDFWebkitPlugin/Config.spec line 7

-- MichaelDaum - 31 Jan 2013

Howdy! Working on some checkers for Item12285 (commits on Item12391, Eg. distro:1828327c92d2) I notice that you have to manually go in and validate Web server environment under Configuration Audit. I guess I can live with that but the thing that needs fixing is that none of the ERROR or WARN messages generated during the CGISetup.pm's analyzeWebserver checks are ever propagated to the user.

I know they're working because if I place a die $content; just before the return statement in that method, we can see the errors/warnings decorated in the HTML page that is rendered inside the ajax-loaded response div thingy.

Suggestions?

-- PaulHarvey - 12 Feb 2013

Regrading this from "Enhancement" to "Urgent" because it is blocking 1.2.0 release.

-- CrawfordCurrie - 25 Jul 2013

I think it's time to set this mega-task "Waiting for release", and open individual tasks for remaining issues. We are loosing track of what needs to be fixed to make this a usable system.

-- GeorgeClark - 18 Oct 2013

Commenting is disabled while not logged in