BuildContrib Cookbook

This is a cookbook of recipes for using BuildContrib.

BuildContrib is a powerful, multi-faceted build system, used to build Foswiki extensions and the Foswiki core itself. It started out as a simple perl clone of ANT, but has gained a diverse range of other functionality since then.

Extensions should be checked into subversion, so that's assumed here. Anyone who develops outside subversion is asking for trouble, and is pretty much on their own. See HowToStartExtensionDevelopmentInSubversion for help getting started with it.

Insert the name and type of your module below, and hit "rewrite instructions" to rewrite the instructions for your module. See the Extensions web for a description of the different types of extension module.

Root of subversion checkout area
Extension name e.g. "Electric"
Extension type	PluginContribSkinAddOnWikiApp

Recipe 1: Setting up your dev environment

The best way to dev is in a subversion checkout. See SvnRepository for help in checking one out. (You can also use Git - See HowToUseGit.) Once you have a checkout area, follow the installation instructions to configure it as a runningFoswiki so you can test your code in the browser. Then set these environment variables.

• export FOSWIKI_HOME=/home/foswiki/core or the equivalent for your shell.
• export FOSWIKI_LIBS=$FOSWIKI_HOME/lib:$FOSWIKI_HOME/lib/CPAN/lib or the equivalent for your shell. Note that it's a path with two directories on it.

The beauty of using a checkout area is that if you make a boo-boo, it's easy to revert your changes, and you get the entire published Extension Library to crib, extend or work from. And you are always testing against the latest code.

Once you have a working Foswiki, you need to checkout BuildContrib, if you haven't already done that:

cd /home/foswiki svn co http://svn.foswiki.org/trunk/BuildContrib BuildContrib 

Now you have checked it out, use pseudo-install.pl to link it into the core, thus.

cd /home/foswiki/core && ./pseudo-install.pl BuildContrib 

If you want to be able to check your work into Subversion, you need to be authorized. See GettingStarted

Recipe 2: Creating a new extension

This recipe describes the process of building a new extension called ElectricPlugin, using the create_new_extension.pl script installed with BuildContrib.

1. cd /home/foswiki
2. perl BuildContrib/create_new_extension.pl ElectricPlugin

This will create the ElectricPlugin file hierarchy, including build.pl, DEPENDENCIES and MANIFEST. At this point, the Extension Topic (and all other files) are copied from Empty(Plugin/Contrib), and you will need to change those to reflect what you are working on.

Recipe 3: Making MANIFEST and DEPENDENCIES files

The MANIFEST file lists files that are released with the extension. Not all files in the source tree will be released; for example, it is unusual to release developer tests and support scripts such as those used for benchmarking.

The DEPENDENCIES file lists external dependencies, such as those on CPAN modules, or other Foswiki modules. The format of the DEPENDENCIES file is one dependency per line; dependency, version, type, and comment are comma-separated fields specified per dependency, defined below.

Storable,>=0,cpan,Required for saving attachment hash values and parameters
htmldoc,,external,Required

• Dependency name Package or module name to be installed. For Foswiki modules it is a fully qualified name, for example Foswiki::Plugins::SafeWikiPlugin, For CPAN modules, this should be the name as installed from the CPAN shell. For external packages, and for best compatibility with the .deb Debian packager, the name should have a unique hit when searched with apt-cache search <name>
• Version If a minimum version of the package is required, this should be a simple expression >10.1 for example
• Type This is one of perl, cpan or external. perl dependencies are only used for other Foswiki modules. Note that the Foswiki installer will not install external packages, however if the dependency matches a debian package, it will be added to the .deb package as a dependency.
• Description Description of the dependency. If the description starts with the word Optional, it will never be automatically installed.

After you run create_new_extension.pl the initial MANIFEST and DEPENDENCIES files are empty. You can generate draft contents thus:

1. cd /home/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
2. perl build.pl manifest
3. perl build.pl dependencies (needs the CPAN module B::PerlReq)

These targets "guess" the content of the files; you need to apply some intelligence to derive the actual content.

You can run these targets whenever you want, even after you have generated what you believe are correct MANIFEST and DEPENDENCIES files.

Recipe 4: Making unit tests

We strongly recommend using test-first development, and ensuring that you have unit tests in place for all your extensions. Unfortunately many people are too lazy, or too stupid (like the author of BuildContrib), to write tests for their extensions. If you are one of the wise:

1. mkdir -p /home/foswiki/ElectricPlugin/test/unit/ElectricPlugin
2. Create a testcase. The easiest way to do this is to base it on an existing testcase. For an example, see /home/foswiki/CommentPlugin/test/unit/CommentPlugin.
3. To run the tests, cd /home/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin and perl build.pl test
   • build.pl will print the command line used to run the tests, so you can easily copy-paste-modify the command line to home in on a failing test
   • do not ignore test failures - maintaining a quality test suite is a major help to yourself and others in maintaining extensions

Recipe 5: Debugging unit tests

Sometimes a unit test will fail (that's why you have them, after all) and you will need to debug. This is not so easily achieved with build.pl since it fires off separate processes to do the hard work. Fortunately there are some simple tactics you can use to help debug a test. Say that you want to know why, in your ElectricPlugin, test_Something reports a failure though it is supposed to work.

Recipe 5a: Isolate the failing test

1. grep test_Something /home/foswiki/ElectricPlugin/test/unit/ElectricPlugin/*.pm to find the .pm that contains the failing test
2. Run perl build.pl. Note the command line it prints as it starts the test run. It'll be something like this:
   perl -w -I/home/foswiki/ElectricPlugin/lib -I/home/foswiki/lib -I/home/foswiki/lib -I/home/foswiki/lib/CPAN/lib -I/etc/perl -I/usr/local/lib/perl/5.8.8 -I/usr/local/share/perl/5.8.8 -I/usr/lib/perl5 -I/usr/share/perl5 -I/usr/lib/perl/5.8 -I/usr/share/perl/5.8 -I/usr/local/lib/site_perl -I. /home/foswiki/core/test/bin/TestRunner.pl /home/foswiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm
3. Copy the command and paste it into the shell. Change the name of the .pm that is run to the one you found in step 1 that contains the failing test
4. Edit the source of the failing test, and change the names of all tests that are not failing to something that doesn't start with test. I usually s/sub test/sub detest/g
5. Run the command. Now the single failing test is the only one run, allowing you to focus in on the problem with print STDERR statements, or.....

Recipe 5b: Use a debugger

The following recipe will guide you to a Perl debugger session which stops at the first line of sub test_Something. It is less scary than it looks at first glance, you only need to type in what's in boldface.

1. $ cd /home/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin
2. $ export PERL5OPT=-d
   • ...or the equivalent for your shell. This environment variable contains options which well be passed to all invocations of perl from now on.
3. $ perl build.pl test
   • Fire off the tests.
4. Loading DB routines from perl5db.pl version 1.28
(...)
main::(build.pl:48): $build = new BuildBuild();
DB<1> c
   • Just continue, we are not going to find bugs in build.pl (we hope!)
5. Running tests in /home/foswiki/ElectricPlugin/test/unit/ElectricPlugin/TemplateToolkitPluginSuite.pm
(...)
main::(/home/foswiki/test/bin/TestRunner.pl:3):
3: require 5.006;
DB<1> b postpone ElectricPluginTests::test_Something
   • Daughter DB session, maybe in an new xterm. At the current line, your test suite has not yet been compiled, that/home/foswiki/core/test/bin/TestRunner.pl /home/foswiki/ElectricPlugin/test/unit/ElectricPlugin/ElectricPluginSuite.pm's what postpone is for.
6. DB<2> c
   • Start TestRunner.pl. Some test cases may run before test_Something (the ordering is hardly predictable), but eventually...
7. ElectricPluginTests::test_Something(ElectricPluginTests.pm:57):
57: my $this = shift;
DB<3>
   • Eventually, you're where you can start tracing. Phew! After you're done, you have to quit ( q) twice, for both parent and daughter session.

Final hint: If all your test cases fail, but perl never stops at the break points you set with this recipe, set a breakpoint at the init routine: b postpone ElectricPluginTests::set_up

Recipe 6: Testing your release package

You build and test your release package as follows. For this you need two shells, we'll call them "shell 1" and "shell 2", and a browser.

1. If you followed recipe 5, remember to ./pseudo-install.pl -uninstall ElectricPlugin first, or bad things will happen
2. In shell 1, cd /home/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin and then perl build.pl release.
3. In shell 2, cd /home/foswiki and perl ElectricPlugin/ElectricPlugin_installer. This will run your freshly built instaler script. It should find your just-built ElectricPlugin/ElectricPlugin.tgz and offer to install it. Let it.
4. Test in the browser, check the documentation etc etc

The reason for using 2 shells is so if you make a mistake, you can edit and rebuild in shell 1 without wearing your fingers out cd'ing back and forth.

You can use a similar approach to installing in another Foswiki install on the same machine - for example, if you want to test against an earlier release.

1. In shell 2, cd to the root of the other Foswiki installation (say /home/old)
2. export FOSWIKI_PACKAGES=/home/foswiki/ElectricPlugin to point to the package you just built
3. perl /home/foswiki/ElectricPlugin/ElectricPlugin_installer, and it will install in /home/old

Recipe 7: Uploading your work

Please don't skip Recipe 6! When an extension is uploaded, it becomes visible to the configure "Find Extensions" interface. Extensions that Foswiki advertises in the installer should at least install and run without errors using the Plugin topic. Extensions that don't cleanly install or function reflect badly on overall Foswiki quality. Note: If your extension will not work or is not tested in some environments, note this in the package form. And if it has not been thoroughly tested, or has known issues, please upload to the Extensions Testing Web.

The best way to upload your release package is to use the upload target.

cd /home/foswiki/ElectricPlugin/lib/Foswiki/Plugins/ElectricPlugin

and then

perl build.pl upload

It will offer to upload to foswiki.org (you can change this to upload to another repository, for example if you are using BuildContrib to build company-specific plugins). Use your Foswiki username and password. It will cache them to a local, unencrypted file, $HOME/.buildcontrib! If you can't use the upload target, then manually upload/attach the files that were built when you did perl build.pl release.

If your upload is large, or your upload link is very slow and takes longer than 3 minutes, it may fail with a 500 Internal Server Error. This is because of the 3 minute timeout on the fcgi process. Change the upload to use plain CGI by setting the Scripts path to http://foswiki.org/cgi

Note that the perl build.pl upload command will attempt to upload any attachments in the metadata of the Plugin topic. If your plugin has attached any files that you don't intend to upload, delete the metadata or the upload will fail with missing file errors.

-- Contributors: CrawfordCurrie, SvenDowideit, TWiki:Main.HaraldJoerg, GeorgeClark, NicolasWrobel

Discussion

In Tasks.Item8531, we finally stopped checking in .gz and compressed versions of js and css files, which is a great step forward.

But now, autogenerated MANIFESTs are much less useful than they used to be, and large plugins like TinyMCEPlugin, JQueryPlugin, anything with many .css and .js files is very tedious to confidently come up with a comprehensive MANIFEST.

So below are a couple of vim tricks that I finally got working, so that I can reliably rebuild the manifest automatically and apply the following regexes to get the minified/gzipped versions where appropriate

1. Assume extension directory has all minified and gzipped files removed
2. Generate a new MANIFEST using ./build.pl manifest
3. Insert .gz entries for any .css or .js file that does NOT have .uncompressed in their name:

    %s/^\(.*uncompressed.*\)\@!\(.*\)\.\(css\|js\)\(.*$\)/\1\2.\3\r\1\2.\3.gz/

4. Insert .gz AND minified entries for all .uncompressed. files:

   %s/^\(.*\)\.\(uncompressed\)\.\(js\|css\)\(.*\)$/\1.\2.\3\4\r\1.\3\4\r\1.\3.gz\4

Added a full recipe for updating TinyMCEPlugin at Development.TinyMCEPlugin

Probably, this is a sign that the gzipping/minificaiton work in BuildContrib? is incomplete

-- PaulHarvey - 17 Mar 2010