#!/bin/bash set -a USAGE='foswiki-upgrade-check [options] install_dir old_release new_release This allows you to check what a foswiki upgrade would do, and perform it with interactive merging. Arguments are: - install_dir your foswiki directory, where the current instance runs - old_release a directory containing the distributed release for it - new_release a directory containing the distributed new version Quickstart: If your foswiki is currently at version 1.0.4, runs in /www/fw, and you want to upgrade to 1.0.5. Untar the orginal 1.0.4 and 1.0.5 distribs in /tmp, and run: # [1] to see what may need manual merge, run: # (it will also check that the file permissions allow the upgrade) foswiki-upgrade-check /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5 # [2a] then, to perform the upgrade on the same server, run: foswiki-upgrade-check -y /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5 # [2b] or, if fowsiki runs in /www/fw on host "server" as user www-data, run: foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5 Without options, it just lists the conflicting files that may require merging. Run with -y to actually perform the upgrade. The script will try to do as much as possible automatically, and will prompt you to interactively resolve merge conflicts it cannot solve via external graphical merge programs. It creates files to perform a rollback, but it is advised to backup your installation yourself, just to be sure... In the interactive merge programs, - left window is the old_release - middle window is install_dir - right window is new_release For kdiff3 (default), for each conflict choosing the version of the old_release is Ctrl-1, install_dir is Ctrl-2, new_release is Ctrl-3, so you can drive the merging process entirely via the keyboard, e.g: Ctrl-3, Ctrl-S, Ctrl-Q Also, be sure to run "kdedialog" to disable kdiff3 verbose debug output If the interactive merge programs window does not appear on a file listed with conflicts, it is because all the merges could actually be performed automatically for this file by the external tool. REMOTE MODE: if install_dir is a remote dir (in the form user@host:dir), a remote server installation is performed as in the exemple below. E.g: foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Fw-1.0.4 /tmp/Fw-1.0.5 In this mode, the script goes to the server, checks what files should be ugraded, bring them back on the local machine to perform the interactive merge conflict resolutions, and prompts for uploading the upgrade to the server. The account "user" on the host should have write permission to files in the wiki installation. As connections will be done via ssh and rsync, you may want to install your public key as authorized_keys on this account for not having to enter your password many times. Options: -v verbose: lists all operations that will be done -a list all files that would be modified, not just the to merge ones. -s do not merge, but show the diffs for all conflicts (use kdiff3) -y yes, really "do it" performs the copy & merge you must have write permission to the install_dir files. Please make a BACKUP before!!! If you do not want to make a full backup, you can backup the files listed with the -a option that will be changed by the upgrade. Note that if the install_dir is remote, the changes will only be made on a local copy, and you will be invited to rerun the command with -ry to perform the actual upgrade on the remoter server. -Y same as -y, but on each conflict, prompts for what to do: interactive merging, keeping original, or using new file. Useful to force new files for all plugins that have been updated via the web interface. -bl FILE "Blacklists" files in FILE (listed one per line). These files will be kept untouched in case of merge conflicts -wl FILE "Whitelists: files in FILE (listed one per line). These files will be overriden by the one in new_release in case of merge conflicts. Intended use is to list conflict into a file, by running first without options, edit the file to keep only the lines describing the files you want to force upgrade of (typically distributed plugins that you may have upgraded from the configure interface), and use this file as whitelist for -y runs. -T DIR uses DIR instead of tmp to store temporary files -l list all files of the installation that will be probed for upgrade -kd3 use kdiff3 to perform interactive merges (default) -xxd use xxdiff to perform interactive merges -tkd use tkdiff to perform interactive merges -mld use meld to perform interactive merges -mfd DIR do not merge interactively, but instead in case of conflict take file from DIR. Useful if DIR is a copy of your prod instance you upgraded previously to test things and do not want to repeat manual process and copy what you did there -nom no merge, for all conflicts keep your files -mo FILE merge only the file FILE into a copy, and exit. Useful to check things, or re-do the merge for one file if you have regrets. Remember to use a backup of the old install_dir for this. -bd DIR specify backup dir -f force run: does not check that dirs are actually full foswiki install Do not forget to fix files permissions after un upgrade, for instance by a chown -R apache:apache ., chown -R www-data:www-data ., chmod -R a+rwX . depending on your installation The "new_release" dir can also be the distributed "upgrade" package, but this is not recommended. It is advised to use the full release for the new version as otherwise small discrepencies in the page headers may appear on susequent upgrades, forcing cumbersome manual merges for just the %TOPICINFO line. Temporary files are kept for reference in /tmp/fuc.$USER feel free to remove this directory one everything is working. v1.8 2011-12-07 remote mode was buggy v1.7 2011-11-08 new remote mode and (hidden) -ry option new -mo and -bl & -wl, blacklist & whitelist options better check for file and dirs permissions, and binary files v1.6 2011-04-20 -Y option for contributions by George Clark auto-merges if the conflict is only in the topicinfo line checks that install_dir files are writable v1.5 2010-09-23 -f force option to be able to use the script on any data v1.4 2010-03-28 -a and -l options to help backuping v1.3 2010-02-26 dir arguments can now be relative v1.2 2009-09-22 -s option, kdiff3 merge is now automatic if no conflicts detects case where file is new but different avoid merging binary files, just copy over. v1.1 2009-08-18 -nom was not working v1.0 2009-04-26 published on http://foswiki.org/Support/HowDoIUpgradeSafelyACustomizedFoswikiInstallation ' V () { if $verbose; then echo "== $@" >&2; fi; } VR () { echo "## $@" >&2; } doit () { if $doit; then "$@"; fi; } dolist () { if ! $doit; then if $dolist; then echo $1; fi; fi; } rsync-mirror () { rsync -aHSx --delete --force "$@"; } err () { echo "***ERROR: $*" >&2; exit 1; } merge=merge_kdiff3 merge_conflicts=true doit=false dolist=false listall=false verbose=false mfd= showdiff= force=false prompt_merge=false remote=false tmp=/tmp/fuc.$USER topicinfo=/tmp/fuc.$USER.topicinfo remote_yes=false blacklist= whitelist= merge_once= B=/tmp/foswiki-backup-$(date +%Y-%m-%d.%Hh%M) # Options processing while test "_${1#-}" != "_$1" -a "_${1//-/}" != "_";do case "$1" in -kd3) merge=merge_kdiff3;; -tkd) merge=merge_tkdiff;; -xxd) merge=merge_xxdiff;; -mld) merge=merge_meld;; -mfd) merge=merge_mfd; merge_conflicts=false; mfd=$2;shift;; -nom) merge=merge_none; merge_conflicts=false;; -s) showdiff=show_kdiff3;; -Y) doit=true; prompt_merge=true;; -T) tmp="$2";shift;; -bl) blacklist="$2";shift;; -wl) whitelist="$2";shift;; -ry) remote_yes=true;; -mo) merge_once="$2"; shift;; -a) dolist=true;; -l) listall=true;; -y) doit=true;; -v) verbose=true;; -f) force=true;; -bd) B="$2"; shift;; *) echo "$USAGE"; exit 1; esac;shift; done; if test "_$1" = "_--";then shift; fi if test $# != 3; then echo ERROR: 3 arguments needed; echo "$USAGE"; exit 1; fi $force || for d in "$@"; do if test ! -e $d/lib/Foswiki.pm; then if test "$d" = "$1" && [[ $d =~ : ]]; then #remote VR "Remote mode: preparing to upgrade remote server" VR " at $d" if ! ssh "${d%:*}" "test -e ${d##*:}/lib/Foswiki.pm"; then echo ERROR: Cannot find a foswiki remote directory at "$d"; exit 1 fi rshell=$(ssh "${d%:*}" 'echo $SHELL') if test "$rshell" != '/bin/bash'; then echo ERROR: "Account ${d%:*} does not use bash as shell, but $rshell" exit 1 fi else echo ERROR: "$d" is not a foswiki directory. Use -f to force use.; exit 1 fi fi done if test -n "$mfd" -a ! -d "$mfd"; then echo ERROR: "$d" is not a directory.; exit 1 fi I="$1"; O="$2"; U="$3" if [[ $I =~ : ]]; then remote=true elif [[ $I =~ ^[^/] ]]; then I="$PWD/$I"; fi if [[ $O =~ ^[^/] ]]; then O="$PWD/$O"; fi if [[ $U =~ ^[^/] ]]; then U="$PWD/$U"; fi if $listall; then cd $I; for d in "$O" "$U";do ( cd $d; find * -type f );done|sort|uniq \ | while read f; do if test -e "$f"; then echo "$f";fi;done exit 0 fi # merge args are is the file, merge into installation merge_kdiff3 () { kdiff3 -m --auto $orig $I/$1 $U/$1 -o $I/$1; } show_kdiff3 () { kdiff3 $orig $I/$1 $U/$1; } merge_tkdiff () { tkdiff -a $orig -o $I/$1 $I/$1 $U/$1; } merge_xxdiff () { xxdiff --show-merged-pane -O -m -M $I/$1 $I/$1 $orig $U/$1; } merge_meld () { meld $orig $I/$1 $U/$1; } merge_mfd () { cp $mfd/$1 $I/$1; } merge_none () { :; } # from now on, we compare in a case-independent way for case and =~ shopt -s nocasematch is_binary () { local ext if [[ $1 =~ [.]([^.]+)$ ]]; then # known extensions of text files ext="${BASH_REMATCH[1]}" if [[ $ext =~ ^(cfg|cgi|changes|css|diff|htaccess|html|htm|js|json|log|patch|php|pl|pm|tex|tmpl|txt|types|xml)$ ]];then return 1 fi fi # else use diff that returns 0 for null files, 1 for text, 2 for binaries diff --line-format= /dev/null "$1" >/dev/null 2>&1 if test $? = 2; then return 0; fi return 1 } perform_merge () { local rep i="$1" case "$i" in *.txt) # only for text files, that are potential wiki topics # auto-merge for differences in only the TOPICINFO line if $merge_conflicts && grep -q '^%META:TOPICINFO[{].*[}]%$' $U/$i; then if diff -q -I '^%META:TOPICINFO[{].*[}]%$' $I/$i $U/$i >/dev/null; then doit cp $U/$i $I/$i V " Only diff between Installed and New was in TOPICINFO, forcing new" return elif diff -q -I '^%META:TOPICINFO[{].*[}]%$' $O/$i $I/$i >/dev/null; then doit cp $U/$i $I/$i V " Only diff between Installed and Old was in TOPICINFO, forcing new" return elif diff -q -I '^%META:TOPICINFO[{].*[}]%$' $O/$i $U/$i >/dev/null; then V " Only diff between Old and New was in TOPICINFO, keeping installed" return fi V $i forcing new TOPICINFO line of new_release into installed grep '^%META:TOPICINFO[{].*[}]%$' $U/$i >$topicinfo grep -v '^%META:TOPICINFO[{].*[}]%$' $I/$i >>$topicinfo mv $topicinfo $I/$i fi ;; esac if $prompt_merge; then while true; do echo -n " [Enter, 1, m] = Merge, [2, o] = keep Original, [3, n] = use New file: " read rep &2 fi if ! cmp -s $i $I/$i; then if cmp -s $O/$i $I/$i; then V $i LOCALLY_SAME, DIST_CHANGED, COPY dobackup $i doit cp $U/$i $I/$i dolist $i elif cmp -s $i $O/$i; then V $i LOCALLY_CHANGED, DIST_SAME. KEEP else if test ! -e $O/$i; then V $i NEW FILE, BUT DIFFERENT IN LOCAL AND UPGRADE orig=/dev/null else V $i CONFLICT, MERGE orig=$O/$i fi if test -n "$blacklist" && fgrep -xq "$i" "$blacklist"; then if $doit; then echo "$i blackisted, untouched"; fi elif test -n "$whitelist" && fgrep -xq "$i" "$whitelist"; then if $doit; then echo "$i whitelisted, forcing update" dobackup $i doit cp $U/$i $I/$i fi elif $doit; then dobackup $i if is_binary "$i"; then doit cp $U/$i $I/$i echo "WARNING: copied $U/$i over your $I/$i . Revert if wrong" else echo "Merging $i"; perform_merge $i fi elif test -n "$showdiff"; then if is_binary "$i"; then ls -l $orig $I/$i $U/$i else $showdiff $i fi else echo $i fi fi else V $i SAME, IGNORE fi else V $i NEW FILE, COPY if ! wdir=$(dir_writable_of $I/$i); then echo "###WARNING: dir not modifiable: $wdir" >&2 fi if $doit; then { echo -n "rm -f \""; quote_quotes "$i"; echo "\""; } >>$B.del.sh dir=$I/$i; dir=${dir%/*} if test ! -d $dir; then mkdir -p $dir; fi cp $U/$i $I/$i fi fi done } remote_upgrade () { local o_doit o_I fuc=$(which foswiki-upgrade-check) local dest="${I%:*}"; RI="${I##*:}" if ! $remote_yes; then VR "Preparing local upgrade of remote server" VR "Note that nothing will be actually done on the server by this command" $doit && VR "You will be prompted later to run another command to apply them" rm -rf $tmp/I $tmp/O $tmp/U mkdir -p $tmp/I ssh $dest "mkdir -p $tmp/O $tmp/U $tmp/N" scp -q $fuc $dest:$tmp rsync-mirror $O/ $dest:$tmp/O/; rsync-mirror $U/ $dest:$tmp/U/ VR Running foswiki-upgrade-check on the server to list the conflicting files. ssh $dest "chmod -R a+rwX $tmp 2>/dev/null; cd $RI; $tmp/foswiki-upgrade-check $RI $tmp/O $tmp/U >$tmp/fwu.list; $tmp/foswiki-upgrade-check -a $RI $tmp/O $tmp/U | tar cfz $tmp/wiki-backup.tgz -T -; $tmp/foswiki-upgrade-check -l $RI $tmp/O $tmp/U | tar cfz $tmp/fwi.tgz -T -" VR Copying the server files to upgrade on your local workstation scp -q $dest:$tmp/fwi.tgz $dest:$tmp/fwu.list $tmp cd $tmp/I; tar xfz $tmp/fwi.tgz VR 'Performing the regular "foswiki-upgrade-check" locally' o_I="$I"; I=$tmp/I if $doit; then VR "Performing the merge locally only..." upgrade VR "Uploading your resolved conflicts to $dest:$tmp/N/" cd $tmp/I rsync -aHSx --delete --force -R $(cat $tmp/fwu.list) $dest:$tmp/N/ VR "Changes are now ready to apply on server at $o_I" VR "To actually perform them now, re-run the same command, using -ry" VR "(remote yes) instead of just -y" VR "i.e:" VR "$fuc -ry $o_I $O $U" else upgrade fi I="$o_I" exit 0 else VR "PERFORMING ACTUAL UPGRADE ###" VR "Performing the automatic merge on the server at $dest" ssh $dest "$tmp/foswiki-upgrade-check -y -bd $B -mfd $tmp/N $RI $tmp/O $tmp/U" VR Cleaning up scp -q $dest:$B.tgz $dest:$B.del.sh ${B%/*} 2>/dev/null cd / fi } upgrade_one () { local i="$1" LI=$tmp/LI; mkdir -p "$LI/${i%/*}" if $remote; then if ! scp -q "$I/$i" "$LI/${i%/*}"; then err "$i not found on $dest!"; fi else if ! cp "$I/$i" "$LI/${i%/*}"; then err "$i not found!"; fi fi o_I="$I"; I=$LI; o_doit=$doit; doit=true; orig=$O/$i VR "Merging only (no actual changes done) $i" perform_merge $i VR "Result merged in a local copy at: $LI/$i" I="$o_I"; doit=$o_doit } # return true if we can create the dir, and prints it dir_writable_of () { local dir="${1%/*}"; local d="$dir" while test ! -d "$d"; do d="${d%/*}"; test -z "$d" && d=/; done echo "$d"; test -w "$d"; return $? } quote_quotes () { echo -n "$1" | sed -e s'/"/\\"/g'; } dobackup () { local dest="$B/${1%/*}" if test ! -d "$dest"; then mkdir -p "$dest"; fi cp -a "$I/$1" "$dest" } if test -n "$merge_once"; then upgrade_one "$merge_once" else if $remote; then remote_upgrade; else upgrade; fi fi backup_header () { $1 && echo "#### BACKUP of changes done. To rollback, go to $I and use the commands:" } if $doit; then print_mess=true if test -d $B; then ( cd $B; tar cfz $B.tgz . ); rm -rf $B backup_header $print_mess; print_mess=false echo "#### tar xfz $B.tgz" fi if test -e $B.del.sh; then backup_header $print_mess; print_mess=false echo "#### sh $B.del.sh" fi if ! $print_mess; then echo "#### you can remove the backup file(s) when not needed by:" echo "#### rm $B.*" $remote && echo "#### both on the server and local machine" fi fi exit 0