Feature Proposal: Formally support multiple extension versions

This proposal is being split into 3 separate proposals:

• This proposal - Support versions of plugins including dependencies and extension categories
• EasierMirroringOfExtensionRepositories - to cover enhancements to BuildContrib? and the new Local repository contrib.
• AddExtensionRepositoriesToFoswikiOrg - covering adding Testing and Archive locations to foswiki.org

Motivation

Extensions are critical to the operation of many Foswiki installations. While Core goes through a release process, with release candidates, and thorough testing, Extensions is a single stream of latest versions. In addition, Core extensions get released into Extensions web without necessarily going through the same process.

• Clicking "Install" from bin/configure always gets the latest version
• We have no simple way to get the same version as tested on a different server.
• There is no fallback by using bin/configure to select a previous version
• Limited version dependency support.
• No way to tie plugin versions to core versions.

Description and Documentation

Requirements

1. Allow new plugin versions to be released without overlaying the current stable version
2. bin/configure needs to find stable version and/or unstable versions.
3. Continue to use Foswiki for extension distribution.
4. Backwards compatible to previous Foswiki installs. We can't break bin/configure in the field.
5. Plugin topic should list versions of plugin including beta, stable and previous versions.
6. Plugin documentation topic of Beta, Stable and previous versions should also be available.
7. Users should be able to implement their own Extensions repositories

Potential Solutions

• Create a new subweb named Extensions/Beta This web will be used for developers to release extensions without overlaying "stable" version.
  • Separately addressed in Add Extension Repositories To Foswiki Org
  • Only fully addresses 1-4
  • Only supports a single beta version, and no back versions.
  • Could be extended to include an Archive/Obsolete category
  • Cleanup issues

• Implement versioning through Extension names: MyPlugin, MyPlugin-<version> or similar
  • Potential to address all requirements
  • Need to be creative with backwards compatibility
    • Versioning should not extend to the actual installed plugin
  • We don't have consistent versioning in plugins, as was hashed out in $REVISION$ / $VERSION$ discussions

• Implement versioning through the RCS system.
  • potential challenge - attachment versions are independent & disconnected from topic revisions.
  • Pointed out on IRC - need a better RCS system than currently in use

Examples

Impact

Implementation

-- Contributors: GeorgeClark - 11 Feb 2010

Discussion

Webs are best used as a sort of sub-wikis like a sub-domain in hosting language. I'd better not over-use webs for structuring content that belongs together as natural as stable and beta releases of the same extension. Instead, we should enhance the BuildContrib to allow to upload to the same extensions topic but attach the beta version with a different naming scheme.

Have a look at Drupal

-- MichaelDaum - 12 Feb 2010

I would strongly prefer to have access to a continuous stream of released versions. So strongly that I don't consider the Beta web approach a viable solution.

The Extensions repository was developed from a flat web of topics that had manually created zips attached. It has evolved into a semi-automated repository, but still has strings back to that old manual system. If we are going to re-engineer it, we should be prepared to cut those strings.

The most obvious solution is to create a gateway topic for each extension. However the extension releases themselves would each reside in topic named for the specific release. This would use simple TML to automatically find and include the latest release of the extension. For example, ActionTrackerPlugin could %SEARCH{"ActionTracker_*" scope="topic" and %INCLUDE the doc from most recently uploaded "package topic", ActionTrackerPlugin _201000020013 (the 12 digit numbering allows us to support dated releases and numbered releases in a consistently sortable way)

This would allow us to:

• upload the package topics without attaching the form (as we are forced to do at the moment), thus simplifying BuildContrib.
• use the gateway topic for user comments.

The main reasons I haven't done this before now are:

• the perceived need to maintain compatibility with already-released versions of configure.
• uncertain how to handle subsidiary topics (several extensions ship side .txt files)
• time to do it.

A variation on this theme is to allocate a specifically-named subweb for each release; thus:

• ActionTrackerPlugin.txt (gateway topic)
• ActionTrackerPlugin_201000020013/
  • ActionTrackerPlugin.txt (actual release topic)
    • attachment: ActionTrackerPlugin.tgz
  • VarACTION.txt (subsidiary documentation topic)

-- CrawfordCurrie - 13 Feb 2010

So if Extensions (or some new web to preserve backwards compatibility) becomes the Extension gateway, what do the gateway topics look like?

• Description of the extension (Included section from Stable revision)
• Screenshot
• (Include stable documentation topic - or is this too much noise.)
• User feedback - (Do we want users to be able to edit the topic that bin/configure uses to find extensions?)
• PackageForm
• Table of released versions: (Can this be in the PackageForm? somehow, or is this also a search?) - probably not. Can't currently embed tables in forms due to CRLF issues, and search would impact FastReport?

Plugin Release	Release Date	Status	Version	Revision	Core Dependency	Extension Dependency
MyPlugin? _201000020024	2/13/2010	Experimental			>=1.1
MyPlugin? _201000010013	1/13/2010	Release Candidate			>=1.1
MyPlugin? _200900010012	12/14/2009	Stable			<1.1, >1.0.4	>=ZonePlugin_20090001012
MyPlugin? _200900000011	2/1/2009	Obsolete			<=1.0.4

Plugin Release

MyPlugin? _[12-digit-number] generated during upload.

Version & Revision

extracted from plugin.pm if available.

Status

Specify the category of this particular release. Category can be promoted/demoted without changing the release, Version or Revision fields.

Core Dependency

Version of core that this release of the plugin requires. (Merge Core and Extension dependency fields?)

Extension Dependency

List of extension releases that this plugin requires, or block installation of this plugin.

• >= Plugin requires a release of the dependency greater than or equal to the specified release
• ! Version of the specified plugin must not be installed. Can include * wildcard. !BlahPlugin_* - Any version of the BlahPlugin will block this plugin.
• <= This plugin release requires an earlier release of a dependency. <=BlahPlugin_201000010001 - If a version newer than specified is installed, this version is blocked.

• Dependency Groups: (<core1.1? BlahPlugin? ) If core release less than 1.1, BlahPlugin is required.

bin/configure adds a checkbox to choose which versions to offer to install - Testing / Stable / Experimental?

Should _installer script continue to auto-resolve missing dependencies when invoked from bin/configure, or should bin/configure auto-select dependencies and/or BLOCK installation due to dependencies.

Would a status of "core" make sense. So that the version distributed with the Core distribution is also an available version?

Should the build.pl upload also "tag" the SVN with the plugin release so that source for a plugin release can be retrieved?

build.pl upload

• prompts the user to choose the release status.
• It insert the release information into the table of the gateway topic - using a REST handler.

If the list of versions is a search, that works find for viewing individual topics, but the impact on bin/configure installer would be difficult. I think still want something like FastReport? that will extract all of the installable versions in a single pass of the Extensions Gateway topics.

REST Handler functions?

• Insert new release into versions table
• Promote a release (Change status - Release Candidate -> Stable, or Stable -> Obsolete for example)
• Remove a release (Delete from table, should it also "trash" the associated topic and attachments?)

Should more than one version of each status be supported?

Notes on backwards compatibility - assuming that the "versioned" plugins are stored in a web below or along side of Extensions:

• Configure/UIs/EXTEND.pm hardcodes the download path to $repository->{pub} . $extension . '/' . $extension . $ext So for the "default" release, the archives must still be attached directly to the Extensions/Plugin topic.
• The shell Plugin_installer script is generated with the URL pointing to the download location of the plugin files, so plugins released with versioning support could download directly from a versioned respository
• SvenDowideit nightly debian packager does a for each against the Extensions directory. The auto-packaging needs to be coordinated with extensions redesign.

-- GeorgeClark -- 15 Feb 2010

configure depends on FastReport, which in turn depends on the format of extension pages ( description, version and release are generated by regex processing of the Extension front page)

Different versions of extensions will target different versions of Foswiki. It is this inability to tie to specific versions that has made it so difficult to improve the Func API. We must formalise the relationship between extension versions and Foswiki versions.

Sven, yes, authenticated repositories are supported, though only through encoding auth details in the URL.

-- CrawfordCurrie - 15 Feb 2010

I also have my worries about the beta web thing.

• I fear people start making lots of betas and hardly any real releases (the version 0.999998 syndrome)
• I fear the beta area becomes a garbage dump used by noone. Normal admins will stay away from betas. Developers run pseudo-installs and will try new code by simple "svn update".
• The need we have seen so far is to have plugin versions tied to Foswiki versions. Once Foswiki 1.1 is out we will wish to have the good old stable plugin versions available for those still on Foswiki 1.0.X. And have fully released more feature rich and not fully compatible plugins available for Foswiki 1.1.X. The current scheme cannot use this. A beta web does not solve this most urgent need. I'd rather see a solution that both allows betas and stable release versions for each Foswiki version or range of versions. A bit like we know from Linux distros. They have a repository per release generation and within each a stable, unstable/development branch and sometimes a bleeding edge (like in blood and bloody). We should create a solution inspired on this.

-- KennethLavrsen - 18 Feb 2010

I'm sorry for confusing things, but there were multiple ideas being discussed in this proposal - my fault, and I'm trying to fix it.

This topic now discusses how we might implement Extension versions, and hopefully addresses all of the reservations of the Beta/Testing web. However it's clearly a longer term effort.

I split off the alternate repository discussion into AddExtensionRepositoriesToFoswikiOrg and added a suggestion that we set a time limit - Topics will be deleted from the Testing / Beta web after some to be agreed upon limit. So Testing extensions either get released or trashed.

So short term - make it easier with limitations to release test versions / or coordinated interdependent extension releases. And design a long-term solution that comprehensively addresses extension versioning.

-- GeorgeClark - 19 Feb 2010

Documenting a few more challenges with current pages / searches. Version and Release extraction from the Plugin topics is unreliable. The current regex is:

$pattern(.*?Version\:?\s*\|([^\\n\r\|]*)\|.*) 

This results in reporting the first occurrence of Version: | or Version | in the topic to be the version. On some plugins, it ends up finding things like the Perl Version dependency, etc. So this needs to be improved. For BuildContrib? it finds the definition of the Version field used by the plugin.

-- GeorgeClark - 20 Feb 2010 George, can I kindly ask you to remember to sign your posts when a topic is in thread mode? It is a pain to find out who edited and add a the missing signature. -- KennethLavrsen - 09 Mar 2010

This proposal was split out and the splits were passed with no real comments.

What is this proposal now?

The problem is descrived but I do not see one single solution that I can decide if I like or not.

-- KennethLavrsen - 09 Mar 2010

Sorry Kenneth, I'll try not to omit the sigs

The split out pieces were the things that are simple to accomplish with no significant changes to core, and are more a workaround to the current Extensions architecture.

This piece is the difficult task. I guess this isn't really a proposal as much as a brainstorming discussion on how to redesign Extensions to truly support versions and dependencies on Extensions. But I couldn't see how to change this to brainstorming until a more concrete proposal can emerge.

This is probably a 2.x release discussion.

For this proposal I wanted to try to come up with a design for a new Extensions web and process that would allow:

• Multiple versions of an extension to exist simultaneously - both documentation and distribution files.
• Extension versions to clearly identify dependencies
• A way to clearly identify Extension status (Testing, Stable, Obsolete, ...)
• An easy way to "promote" extensions from Testing to Stable for example, without a re-release.
• Installer to clearly identify dependencies before committing to the install. (auto-select dependencies button in the ExtensionFinder? ?)
• Separate extension form from the extension to simplify release process.
• But still backwards compatible.

Or as a specific example - a way to release ZonePlugin, and all the co-dependent extensions, and give admins a choice as to whether to install a JQuery plugin that depends on ZonePlugin, or to install an older revision that does not have the dependency. Extension dependencies and overall Foswiki dependence on Extensions has become too complex for our current model of "last upload wins".

I'll try to refactor this into some more concrete proposals on how to proceed. I'll change to "Parked" status until I can clean it up and reorganize it.

-- GeorgeClark - 09 Mar 2010

It is urgent that we do something before the 1.1 release. JQueryPlugin is a core plugin that slightly changes its behaviour in 1.1, and we do not want new uploads to be used by 1.0.x admins unless they are aware of the changes and are prepared to do any fixup work for their own plugins/customisations.

Plugin documentation topics that must co-exist with other versions: this gets very messy, unless we have a subweb per extension "version"/branch (or do something creative with WorkflowPlugin? and marking revisions somehow).

Once we've decided on whether we really need this capability, or we accept the many hundreds of subwebs idea (or some other solution), we can move on to discussing implementation.

My thoughts:

• If we can make it so that only the "System topic" (FooPlugin? , FooContrib? , FooSkin? ) from an extension is ever uploaded to foswiki.org/Extensions, then we can quickly work out a new PackageForm? , FastReport? , etc. to enable multiple versions/branches.
• If we must be able to upload Extension documentation, then we either need hundreds of subwebs, or a rethink on the way Extensions are packaged. Work will be needed in BuildContrib? (for example, to re-write links to JQueryObject? as JQueryPlugin-Release01x00-JQueryObject)
  • A consequence of this is that the 1.1 solution will have to be an additional foswiki.org/Extensions/Release01x10 subweb for extensions that should not be used lightly by 1.0.x sites)

-- PaulHarvey - 01 Jun 2010

We now have the bare minimum in Foswiki 1.1 to allow FastReport? to serve up a report that's appropriate for a given Foswiki release (but without regard for the extension versions it may have installed).

GeorgeClark identified on IRC that we still have some challenges to resolve before we even begin to fix FastReport:

1. Extension.pm RELEASE and VERSION values differ from System/Extension.txt on those extensions that haven't used %$RELEASE% and %$VERSION% BuildContrib? macros. Configure looks at the module, FastReport looks at the topic.
2. Inconsistent version numbering schemes: v1.2.3, 1.2.3, 2009-09-01, 1234
3. Foswikirev:1234 and 2009-09-01 are insufficient for full dependency resolution (can't specify a release family like: < 1.8 AND > 1.5 - Foswikirev and date schemes are inappropriate if there is continued development of two different release families over a given time period)
4. Compatibility field is free-form text, not queryable

However, we can think about what we need now, and I think that's simpler than proper extension dependency resolution which we can keep working on later:

Make FastReport show the latest version of a given extension appropriate for the Foswiki release that has accessed FastReport

GeorgeClark mentioned on IRC that he would have a preference for each extension to have its own Subweb. I think that's probably the most future-proof structure; but I'm not sure what the performance would be like if FastReport must inspect 250 subwebs to build the report.

• Extensions
  • TinyMCEPlugin - ExtensionClassification? , DemoUrl? , ImageUrl? , etc.
    • TinyMCEPlugin/Release01x00 - DEPENDENCIES, version, release, date, license, copyright, author...
    • TinyMCEPlugin/Release01x10 - add ZonePlugin to DEPENDENCIES
  • JQueryPlugin
    • JQueryPlugin/Release03x10 - add ZonePlugin to DEPENDENCIES
    • JQueryPlugin/Release03x20
    • JQueryPlugin/Release03x..

I'm assuming QueryAcrossTopicRevisions will get implemented one day, and the report could eventually show configure prior versions of a given release family that could be installed

-- PaulHarvey - 04 Oct 2010