CertificatedStatisticsUpdate

Automatic statistics update

Webstatistics is a bit awkward to update automagically per the instructions in TWikiSiteTools? , especially when X.509 certificate authentication is enabled. As noted there, the geturl.pl script is anonymous.

Here is a shell script that is suitable for use as a cron job to automagically update statistics. Or you can run it by hand (instead of the bin/statistics script.).

Like geturl.pl, it will access the update statistics script. Unlike geturl.pl, it is capable of using host and client X.509 authentiation.

Host authentication ensures that the update goes to the correct host.

Client authentication identifies the script to the wiki. With the X509Plugin? installed, it will automagically log the scrip in under its Wiki username. Thus it will have the correct access privileges. (N.B. TWikiAdminGroup? is usually required for updating the statistics for Main...).

The script should be installed in /twiki/tools. It will determine the configuration from ../lib/LocalSite.cfg. Running it on my site is as simple as tools/runstatistics -- but it has enough options to meet most any requirements. Note that the defaulting mechanism will take advantage of the code supplied in question11, but does not require it.

It would probably help others if it were put in the distribution.

I don't monitor this topic, but can be e-mailed with concerns.

Enjoy

-- TimotheLitt - 16 Dec 2008

runstatistics

#!/bin/sh
#
# Automagic update for web statistics
#
# Run by URL since we can't login under apache - and in any case, we want
# certficate-based authentication to work
#
# /twiki/bin/statistics - Current month for all webs
# /twiki/bin/statistics/Main - Just the main web
# Can also specify ?logdate=200801;webs=proj1,proj2,... (Would update proj1, proj2 for Jan 2008
#
# Since we use https authentication, wget uses administrator certificates...
# We also must specify our CA so that wget can validate our server
#
# wget can succeed even if the update fails, so we check for the expected
# completion text.
cd `dirname $0`

CFG="../lib/LocalSite.cfg"

#
# Get useful stuff from configuration
#
DefaultUrlHost='http://localhost:80'
ScriptUrlPath='/twiki/binx'
grep -P '^\$TWiki::cfg{(DefaultUrlHost|ScriptUrlPath|SmimeCertificateFile|SmimeKeyFile)}' $CFG | sed -e's|\$TWiki::cfg{\(.*\)} = \(.*\)\;|\1=\2|' >$p.tmp
. $p.tmp
rm $p.tmp
#
WHOST="$DefaultUrlHost$ScriptUrlPath"
#
CCSWS=
if [ -n "$SmimeCertificateFile" ] && [ -n "$SmimeKeyFile" ]; then
    CCSWS="--certificate=$SmimeCertificateFile --private-key=$SmimeKeyFile"
fi
#
# Would be nice to get this from LocalSite.cfg...
HCSWS=
HCSWS='--ca-directory=/etc/pki/tls/ca/'

P="`basename $0`"
WQ='-q'
Q=
T='cat'
WA=

while [ -n "$1" ];  do
    case "$1" in
        -d)
            # debug
            WQ=
            T="tee $P.log"
            ;;
        -l)
            # logfile date to process
            shift
            if [ -z "$Q" ]; then Q="?logdate=$1" ; else Q="$Q;logdate=$1" ; fi
            ;;
        -w)
            # webs to process
            shift
            if [ -z "$Q" ]; then Q="?webs=$1" ; else Q="$Q;webs=$1" ; fi
            ;;
        -c)
            # Client certificate
            shift
            CCSWS="$CCSWS --certificate=$1"
            ;;
        -k)
            # Client private key
            shift
            CCSWS="$CCSWS --private-key=$1"
            ;;
        -X)
            # Suppress X509 client certificate use
            CCSWS=
            ;;
        -a)
            # CA certificate bundle for host verification
            shift
            HCSWS="$HCASWS --ca-certificate=$1"
            ;;
        -A)
            # CA certificate directory for host verification
            shift
            HCSWS="$HCASWS --ca-directory=$1"
            ;;
        -x)
            # Suppress host verification
            HCSWS='--no-check-certificate'
            ;;
        -W)
            # wget advanced usage switches
            shift
            WA="$WA $1"
            ;;
        *)
            cat <<EOF
Usage: $P [-d] [-w web[,web...]] [-l logdate] [-X] [-x] [-c cert] [-k key] [-a cafile] [-A cadir] [-W wgetswitch]
Updates wiki statistics by touching the statistics update url.

The hostname and script url are extracted from LocalSite.cfg.

-w - comma-separated list of webs to update.  Default is all.
-l - logifile dates to update,  Format is yyyymm

https considerations:
  o wget will authenticate the host certificate using the default openssl CA.
    To specify a private CA (or a restricted list),  specify your CA directory with -A.or your bundle with -a
  o To disable host certificate authentication - not a good idea - use -x
  o Your client certificate and key are specified with -c and -k.  They default to the S/MIME signing certificates in LocalSite.cfg (if you have them).
  o Client authentication is enabled if -c and -k are specified or defaulted.  You can disable it with -X,
  o Some environments may need to specify more advanced wget switches.  -W lets you do that.
-d enables debugging and will write $P.log.
EOF
            exit 1;
            ;;

    esac
    shift
done

# After all that, the actual work is quite simple:

/usr/bin/wget $WQ $CCSWS $HCSWS $WA $WHOST/statistics$Q -O - | $T | grep $WQ 'End creating usage statistics'
#
ZZ="$?"
#
if [ $ZZ != 0 ]; then
    echo "wiki update $P failed with status $ZZ on `date -R`"
else
    if [ -z "$WQ" ]; then  echo "wiki update $P succeeded on `date -R`" ; fi
fi
if [ "$T" != 'cat' ]; then echo "Check $P.log for details" ; fi

---

Interesting. I guess that this script is a general solution to the "I want to run this logged-in" problem, and applies equally to other scripts such as view and publish?

(I wanted to email to tell you I had moved this page, but I can't find an email address for you )

-- CrawfordCurrie - 17 Dec 2008 - 13:27

Note that for 1.1.5, the statistics script has been changed to require POST. Since it is updating topics it probably should not be running with GET.

-- GeorgeClark - 23 Feb 2012