How do I safely upgrade a customized Foswiki installation

For people having customized their Foswiki installation, or just cautious people that want to upgrade, I have made a (linux/unix) shell script (attached to this page) foswiki-upgrade-check to automate the process. This script will apply a Foswiki upgrade on your installation, performing only the safe copies (based on a reference directory containing the distribution on which your Foswiki installation is based, and a directory containing the new release) and prompting you for merging interactively xxdiff, or meld

Download: foswiki-upgrade-check

Usage:

foswiki-upgrade-check [options] install_dir old_release new_release
This allows you to check what a foswiki upgrade would do, and perform it
with interactive merging. Arguments are:
  - install_dir your foswiki directory, where the current instance runs
  - old_release a directory containing the distributed release for it
  - new_release a directory containing the distributed new version

Quickstart:
If your foswiki is currently at version 1.0.4, runs in /www/fw, and you want 
to upgrade to 1.0.5. Untar the orginal 1.0.4 and 1.0.5 distribs in /tmp, and
run:
# [1] to see what may need manual merge, run:
# (it will also check that the file permissions allow the upgrade)
foswiki-upgrade-check /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5
# [2a] then, to perform the upgrade on the same server, run:
foswiki-upgrade-check -y /www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5
# [2b] or, if fowsiki runs in /www/fw on host "server" as user www-data, run:
foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Foswiki-1.0.4 /tmp/Foswiki-1.0.5

Without options, it just lists the conflicting files that may require merging.
Run with -y to actually perform the upgrade. The script will try to do as much
as possible automatically, and will prompt you to interactively resolve merge
conflicts it cannot solve via external graphical merge programs.
It creates files to perform a rollback, but it is advised to backup your 
installation yourself, just to be sure...

In the interactive merge programs, 
  - left window is the old_release
  - middle window is install_dir
  - right window is new_release
For kdiff3 (default), for each conflict choosing the version of the 
old_release is Ctrl-1, install_dir is Ctrl-2, new_release is Ctrl-3,
so you can drive the merging process entirely via the keyboard, 
e.g: Ctrl-3, Ctrl-S, Ctrl-Q
Also, be sure to run "kdedialog" to disable kdiff3 verbose debug output

If the interactive merge programs window does not appear on a file listed with
conflicts, it is because all the merges could actually be performed
automatically for this file by the external tool.

REMOTE MODE: if install_dir is a remote dir (in the form user@host:dir), 
a remote server installation is performed as in the exemple below. E.g:
  foswiki-upgrade-check -y www-data@server:/www/fw /tmp/Fw-1.0.4 /tmp/Fw-1.0.5
In this mode, the script goes to the server, checks what files should be
ugraded, bring them back on the local machine to perform the interactive merge
conflict resolutions, and prompts for uploading the upgrade to the server.
The account "user" on the host should have write permission to files in the 
wiki installation. As connections will be done via ssh and rsync, you may want
to install your public key as authorized_keys on this account for not having
to enter your password many times.

Options:
  -v      verbose: lists all operations that will be done
  -a      list all files that would be modified, not just the to merge ones.
  -s      do not merge, but show the diffs for all conflicts (use kdiff3)
  -y      yes, really "do it" performs the copy & merge
          you must have write permission to the install_dir files.
          Please make a BACKUP before!!!
          If you do not want to make a full backup, you can backup the files
          listed with the -a option that will be changed by the upgrade.
          Note that if the install_dir is remote, the changes will only be 
          made on a local copy, and you will be invited to rerun the command
          with -ry to perform the actual upgrade on the remoter server.
  -Y      same as -y, but on each conflict, prompts for what to do: 
          interactive merging, keeping original, or using new file. 
          Useful to force new files for all plugins that have been updated 
          via the web interface.
  -bl FILE    "Blacklists" files in FILE (listed one per line). These files 
          will be kept untouched in case of merge conflicts
  -wl FILE    "Whitelists: files in FILE (listed one per line). These files 
          will be overriden by the one in new_release in case of merge
          conflicts. Intended use is to list conflict into a file, by running
          first without options, edit the file to keep only the lines 
          describing the files you want to force upgrade of (typically
          distributed plugins that you may have upgraded from the configure
          interface), and use this file as whitelist for -y runs.
  -T DIR  uses DIR instead of tmp to store temporary files
  -l      list all files of the installation that will be probed for upgrade
  -kd3    use kdiff3 to perform interactive merges (default)
  -xxd    use xxdiff to perform interactive merges
  -tkd    use tkdiff to perform interactive merges
  -mld    use meld to perform interactive merges
  -mfd DIR do not merge interactively, but instead in case of conflict take
          file from DIR. Useful if DIR is a copy of your prod instance you 
          upgraded previously to test things and do not want to repeat manual
          process and copy what you did there
  -nom    no merge, for all conflicts keep your files
  -mo FILE   merge only the file FILE into a copy, and exit. Useful to check
          things, or re-do the merge for one file if you have regrets. 
          Remember to use a backup of the old install_dir for this.
  -bd DIR specify backup dir
  -f      force run: does not check that dirs are actually full foswiki install

Do not forget to fix files permissions after un upgrade, for instance by a 
chown -R apache:apache ., chown -R www-data:www-data ., chmod -R a+rwX .
depending on your installation

The "new_release" dir can also be the distributed "upgrade" package, but
this is not recommended. It is advised to use the full release for the new
version as otherwise small discrepencies in the page headers may appear on
susequent upgrades, forcing cumbersome manual merges for just the %TOPICINFO
line.

Temporary files are kept for reference in /tmp/fuc.$USER feel free to remove
this directory one everything is working.

v1.8 2011-12-07 remote mode was buggy
v1.7 2011-11-08 new remote mode and (hidden) -ry option
                new -mo and -bl & -wl, blacklist & whitelist options
                better check for file and dirs permissions, and binary files
v1.6 2011-04-20 -Y option for contributions by George Clark
                auto-merges if the conflict is only in the topicinfo line
                checks that install_dir files are writable
v1.5 2010-09-23 -f force option to be able to use the script on any data
v1.4 2010-03-28 -a and -l options to help backuping
v1.3 2010-02-26 dir arguments can now be relative
v1.2 2009-09-22 -s option, kdiff3 merge is now automatic if no conflicts
                detects case where file is new but different
                avoid merging binary files, just copy over.
v1.1 2009-08-18 -nom was not working
v1.0 2009-04-26 published on http://foswiki.org/Support/HowDoIUpgradeSafelyACustomizedFoswikiInstallation

-- ColasNahaboo - 28 Mar 2010

Would it possible for you provide a pseudo-code version of the script? I'm considering re-implementing this for Windows. But the script is far from self-explanatory for a non-Unix person.

-- SeanMorgan - 17 Aug 2009

Yes, basically:

• Let's say that I is the installed wiki, O the old distribution used to install it, U the new upgrade or full distrib
• the script goes in the U dir, and (recursively) for all the files F (not dirs) it finds there, it decides what to do:
• if F does not exists in I, it copies F from U to I
• else, if F in I and U are the same do nothing, but if F in I is different,:
  • if F is the same in O and I, it means the file has changed in the new distrib but we didn't modify it locally. just silently copy F from U to I
  • else if F is the same in U and O, it means the file was not changed in the distrib but we changed it locally. we do nothing to keep our version
  • else we have a conflict. Either just print F, or if -y is given, we do a 3-way merge interactively between O, I, U versions of F and save result in I

-- ColasNahaboo - 19 Aug 2009

Thank-you for the explanation.

Related info: this tool comparison shows platform support, as well as feature support (i.e., the 3-way merge):

• http://en.wikipedia.org/wiki/Comparison_of_file_comparison_tools

With the exception of meld the tools you mentioned are all available on Windows.

-- SeanMorgan - 31 Aug 2009

It would be helpful for some additional tips/solutions for those without a graphical environment (which all of the above seem predicated on?) Are there command line diff tools that are compatible? The use case here would be a minimal VM instance, on debian or other.

-- CraigBowers - 01 Sep 2009

The "sdiff" tool that comes with the gnu diff utilities can do a side-by-side text based merge. For an example, the gentoo package upgrade tools (dispatch-conf) uses sdiff to interactively merge configuration files that are modified after each package upgrade. sdiff works pretty well. The "dispatch-conf" tool presents each "protected" file that changes with a simple diff. Among other options, you can choose to keep the new version, temporarily skip the new version and keep the old one, delete the new version, or do a side-by-side merge. With the merge, you can keep the left or right version of blocks of changed lines.

-- GeorgeClark - 02 Sep 2009

"sdiff" is not a 3-way merge. Just 2 ways. For headless servers, what I do is:

• run foswiki-upgrade-check on the server to list the conflicting files. take note of them for instance by foswiki-upgrade-check >/tmp/fwu.list
• copy the server install on your local workstation (by rsync -aHSx --delete --force SERVER:SRC/ DEST/ )
• do the foswiki-upgrade-check locally (in DEST/) with your graphical merge tool
• either re-upload your upgraded local copy to another place on the server (let's say /tmp/fwu ), or better just the resolved conflicts ( rsync -aHSx --delete --force -R `cat /tmp/fwu.list` SERVER/tmp/fwu/ ) if you copied /tmp/fwu.list from the server locally
• on the server, perform the automatic merge with foswiki-upgrade-check -y -mfd /tmp/fwu

uploaded v1.2 of the script

-- ColasNahaboo - 22 Sep 2009

uploaded v1.3 2010-02-26 dir arguments can now be relative

-- ColasNahaboo - 26 Feb 2010

uploaded v1.4 2010-03-28 -a and -l options to help backing up

NOTE it is recommended that you do not use the "Foswiki-upgrade-*" packages, as they omit some files that have actually changed (in minor ways like the hidden TOPICINFO metadata), it will generate a lot of conflict merges at next update that will not be automatically solvable.

-- ColasNahaboo - 28 Mar 2010

With Foswiki-1.1, there are a large number of .gz files that are not detected as binary. .gz needs to be added as a binary extension. Also I've found that most of the time - especially with all the extensions now shipped by default, that I just want to use the new version of the files - all the JQuery files, for example.

I've got a new version that I've attached that resolves the .gz issue, and adds a prompt in the shell. This allows the graphical merge to be skipped most of the time. Answer with 1, 2 or 3.

How do you want to handle conflict: /home/user/foswiki/Foswiki-1.1.0-RC1/data/Sandbox/WebNotify.txt ?
1) Merge files
2) Keep original
3) Use new file
#? 

-- GeorgeClark - 30 Sep 2010

Thanks Georges, I have incorporated your changes. The prompting for conflict is now enabled if -Y is given instead of -y, and I have changed it to stay on a line, and to default to interactive merge on a simple return:

Merging data/System/PatternSkin.txt
  [Enter, 1, m] = Merge, [2, o] = keep Original, [3, n] = use New file: 2
  Keeping original

New version v1.6 2011-04-20 uploaded as attachment to this topic.

-- ColasNahaboo - 20 Apr 2011

New version v1.7 2011-11-08 uploaded as attachment to this topic. Main new feature: remote mode to upgrade a remote server, generates files to perform a rollback.

-- ColasNahaboo - 09 Nov 2011

New version v.18 2011-12-07 uploaded as attachment to this topic. Bugfix to 1.7 that was always using update files in some cases. please upgrade

-- ColasNahaboo - 07 Dec 2011