Feature Proposal: migrate from Subversion to git

Motivation

Subversion seems to be annoying a high proportion of our contributors, and many of them are going through the pain of using git svn.

To Do

It was agreed at FoswikiCamp2011 that this is a now, in principle, a do it.

Nonetheless I (JulianLevens) have not marked this as AcceptedProposal, various issues will still need to be resolved and agreed. For example, we discussed using GitHub? at the camp and realised that this would not provide support for our need for pre-commit hooks. Therefore, we will will host our own server gain this control. Other options may still needs to be discussed and agreed.

See the SCM Component for the list of related tasks. Please create specific tasks with SCM as a component to enable tracking of the various parts of this project.

Description and Documentation

There are a number of options:

1. host our own Git server
2. GitHub? recently added subversion commit support. While they are a closed source commercial service (free for opensource for now) they have alot of mindshare.
3. gitorious - an opensource git hoster
4. sourceforge - they have git, and in many ways it doesn't really matter where the 'master repos' is

Impact

1. we'd need to re-write our tools and docco - unless we use something like github svn support
2. it'll take time and a little risk to retrain some older brains
3. the move from cvs to svn took about 6months before it was decently settled - but at that point, fewer of the devs had used svn - many of our currently active devs already use git for something

It'd be good to have a scriptable interface for commit permissions (SF in its usual style is remarkably poor in this area - web UI only)

Development model

Git supports pretty much any development model. Because it is so trivial to create your own publicly shareable branches, it seems to me that the Foswiki project itself should minimize its workload, and only have the same kind of centralized development in the 'official' repository that we have in svn - trunk, release branches and release tags. This is already enough confusion for users that are not scm literate developers, but tracks the minimally bounded core focus - manage foswiki releases.

One improvement that could become interesting, is to discourage direct pushes to the master repository, and instead, have developers send a pull request to the Smoke Test farm (a set of systems with different Perl's, OS's and configurations), and if a changeset passes, the System pulls that changeset in.

That would ensure that everyone can sync to a codebase that has been tested as effectively as possible...

Implementation

PaulHarvey has been experimenting to make a git (svn) repo per extension. Attached is this quick & dirty script which does the work (doesn't actually download from svn - you have to do git svn fetch --all on each directory). You end up with a directory structure very similar to a trunk checkout, except that each extension is its own separate git-svn repo, with ability to switch between scratch/release/trunk branches, and dcommit/svn rebase from them independently.

And it's fast - all three branches of TinyMCEPlugin come to a 32MB .git directory.

The good:

• git log shows only TinyMCEPlugin history
• Switching branches is instant
• You can easily switch the branch of an individual plugin, rather than the whole checkout
• You can easily switch the branch of core, rather than the whole checkout
• If you branch an extension locally, you only have to deal with changes pertaining to the extension itself, rather than managing a branch of the whole Foswiki trunk

The challenges:

• Commit ids are all changed compared to traditional git svn clone -s checkout.
• All ~500 extensions consume 1011MB in bare form, whereas the .git on an average svn.foswiki.org clone is only around ~250MB
• pseudo-install changes?
• submodule strategy

We need Olivier to put these up as submodules of the foswiki github project; and then do another script which can pull from github and convert the github modules back to git-svn repos (I am mostly interested in having git checkouts which we can dcommit back to svn with).

-- Contributors: SvenDowideit - 14 Jun 2010

Discussion

clearly, I've given up on bzr - it started to annoy me, and as I seem to have begun using github for other things, its an annoyance/context change i don't care for.

oddly, tortoiseGIT doesn't like the github repository ( fatal: http://github.com/foswiki/foswiki/info/refs not found: did you run git update-server-info on the server? ) but is ok with the gitorious one.

-- SvenDowideit - 14 Jun 2010

I like the "social coding" aspect of github. TinyMCE (and many other projects) have picked up quite a few contributions from random forkers there. As you know, spocke even accepted a patch from me, where previously it sat unnoticed on sf.net for over 6 months.

Do we lose much if github/gitorious is merely tracking a hypothetical git.foswiki.org repo? Sounds like we need pros/cons for self-hosted vs gitorious vs github.

My thoughts:

• We should probably continue to host our own repo, unless there's some fantastic benefit that can't be had otherwise.
• We should definitely maintain github/gitorious/etc presence to encourage contributors.
• Would this mean nobody able to commit with svn?
• What's the "easiest" way to migrate svn folk? Has anybody evaluated easygit?

-- PaulHarvey - 14 Jun 2010

I am all for moving to git. svn is regularly taking my machine down for a normal svn up. I really don't care if we push to a github account or not. It will only hold the official trunk to pull from. Everybody can pull from each other as well. So risks are rather low, a lot lower compared to the current single point of failure.

Before moving to git we should think about which development model we should use. While I am pretty sure that best would be to keep a centralized approach (as the github question suggests), we should not discard alternatives too hastily.

-- MichaelDaum - 14 Jun 2010

I had previously opposed a move to git, because windows tools were not available. However, git clients are now available for windows so that argument is moot. I now definitely support a general move to git.

-- MichaelTempest - 14 Jun 2010

I also support such a move, and will be willing to help since I have some experience in moving projects from Subversion to Git.

-- AntonioTerceiro - 14 Jun 2010

I think I don't need to state where I stand on this Just to mention that I tried TortoiseGIT? today, and was able to clone the github repo without any issue.

I guess we now need to hear from the "less technical" ones of us (no harm intended, just not everybody is happy with like 10 git commands to do something, but I do love it).

Also, I've written the HowToUseGit, but I agree with MichaelDaum, if we were to move to git, we should define a clear way of working. An easy one is to branch per bug item, commit on the branch, and then merge to trunk once ready.

Finally, all the concerns I've raised about the use of $Rev$ (CDot loves them), and the like, are still valid. Even though I tried to make some hooks where I could to support git, I haven't really tested BuildContrib? upload and the like with it, so we might need some dev.

And just to dream... imagine committing your LocalSite? .cfg into git, and all your wonderful tools you're using daily but you don't want everybody to look at because you're ashamed of how they're written. Now imagine getting all the latest changes and merging them with your local repo using git pull. It requires some discipline, but I'm pretty confident it's easy to do.

-- OlivierRaginel - 14 Jun 2010

I have never used GIT. I have read about it and I must admit I do not really understand how it works. From the documentation some of you have added it seems to me that there is much more of a learning curve compared to subversion which is extremely simple.

I would not want to fight with a new software CM tool now that we should be focused on getting 1.1 done

And before all you geeks make the decision to do this - please consider how this will impact the occasional extension developer. Will they have an easier learning curve or a steaper learning curve?

You win something with GIT I am sure. But what do we loose? Who do we loose. THINK. Please. And please wait till after 1.1

-- KennethLavrsen - 14 Jun 2010

Hi Kenneth,

I understand you're quite busy at the moment, but probably to partially address your concerns I feel that one of us should evaluate easygit for svn users - it is designed to be as much of a drop-in replacement for svn as possible.

Not sure about windows support.

Git has certainly increased my productivity - I can maintain local ("embarrassing") branches amongst my three staging VMs with ease. In fact, just git clean -fd is a really nice way to clean the checkout compared to some find...exec command.

-- PaulHarvey - 15 Jun 2010

first up, there's no way we do anything before 1.1 is out. As I said, the move from cvs to svn took 6months - there is no way we're changing quickly.

secondly, the occasional extension developer usecase is one of the main things that is slowing down the currently active developers (ok, so Crawford hasn't actually commented ) and we clearly have a strong history of trying to accommodate them (and will continue to do so)

specifically, I don't think what we've done works for the occasional developers either! many have never used svn, and ideally we wouldn't force them to learn any tool.

additional notes:

1. TortoiseGIT? has an almost identical use interface to TortoiseCVS? and TortoiseSVN? so on windows, very little changes
2. I brought up github specifically because they seem to have enabled using svn to commit to the git repositories - to anyone that wants to delay learning git, should (testing needed) be able to work as they used to)

I've added and initial opinion wrt development model up top - please, don't call for discussion, actually discuss

-- SvenDowideit - 15 Jun 2010

@Kenneth, git is not that difficult to handle on normal day-to-day tasks. Even the base commands are named similarly (with some slight semantic changes of course). I am sure you will love it in the end.

Still, this is an essential shift in the tools this project uses. Nobody does that easily. Note also, that today git is rather mainstream as it slowly replaces svn in foss landscape. Not sure if there's any project that would start with svn. They go with one of the distributed VCSes using them in a centralized push development model to lower complexity. Linux-kernel is the exception using a pull model where Linus merges his repo with only a hand full of people he trusts. Distros run their own repos of course pulling from sources they trust from a very wide range of different projects, a modus operandi totally natural for them.

Moving from svn to git is rather a straight forward step that others have accomplished as well. There are little things like $Rev$ to adjust but that's minor. Compare this with the possibilities that unfold for foswiki itself once people get used to git on a broader base.

-- MichaelDaum - 15 Jun 2010

http://github.com/blog/626-announcing-svn-support

and

http://github.com/blog/644-subversion-write-support

neat huh.

-- SvenDowideit - 15 Jun 2010

It seems we all want it but nobody takes the lead. Any CommittedDeveloper?

-- MichaelDaum - 20 Jul 2010

mm, ok, I'll add myself (with date of commitment of my Birthday :0), as I've done several migrations before, including having a hand in the tmwiki cvs->svn one (Where's Ethermage dammit!)

added dream about using a build farm to pull changes into the master repository.

Anyone else that is willing to do the work - please add yourself too :).

That said however, I'd rather see more discussion of impacts, and an idea wrt how&what.. (and how we mitigate the risks)

(And for those not jumping up and down - I don't want to rush this, but subversion is annoying me)

-- SvenDowideit - 20 Jul 2010

This might help with the migration: http://github.com/rcaputo/snerp-vortex

From the author:

Snerp Vortex.

Anagram of Svn Exporter. It's a set of tools I wrote for the POE repository conversion. It converts POE's nearly 2900 revisions in about five minutes, so it's pretty cheap for me to scrap converted repositories and try again. And again.

-- AndrewJones - 29 Aug 2010

Given that we import a lot of javascript code, can I just say that it would be nice to have git to pull from Eg. http://github.com/tinymce/tinymce, http://github.com/jquery/jquery...

I am concerned that we haven't discussed how to utilise git submodules (I've not tried them yet). Possible benefits:

• Extension authors could host their own git repo, but still allow the official foswiki git repo to point to them.
• People checking out a trunk or release branch can choose to only checkout the extensions they are interested in
• With TinyMCE (and other libraries), we get access to all the proper change history of those other projects (git bisect could help us track down bugs introduced in a specific revision of an external project/module), instead of the sledge-hammer that is periodic import of new code.
• ohloh might finally be able to show that we are a perl project, not javascript (once it is clear that the libraries are from 3rd-parties

-- PaulHarvey - 04 Oct 2010

For those who might wonder, I'll try to explain a bit more what submodules are (I use them at work for chef).

Basically, they're pointers to another repository's tree, so, they just contain the SHA1 of the commit-id of the subrepo. Maybe that's not clear enough... Let's say you git submodule add tinymce, this would create a directory called tinymce, and some .gitsubmodule file with the commit id. Then when you want to merge / use a newer version, you go to the tinymce directory, which is a git repository, you git pull to get the latest version of the repo, you checkout the commit you want to use (should be a tag I guess), then you go back to the foswiki dir, and you git commit tinymce, which will simply update the SHA1 link.

So:

• yes, extension authors can host their own git repo like that
• yes, by default one would only get the main repository, and one would need to manually git submodule update $submodule
• Yes, we get access to the history, and we can bisect their repository with our tools, but we would need the sledgehammer to sometimes update the submodule SHA1. Also, bear in mind that some submodules need extra processing before being incorporated into our code. One might need to build or pack them.
• yes, this would make it clear what's our code, and what's YUI, JQuery, TinyMCE and the like

Sven, as this is long overdue, I guess I should add myself as a committed developer and help you out

-- OlivierRaginel - 16 Nov 2010

I am evaluating easygit. I'll see how it goes

-- MichaelTempest - 17 Nov 2010

What I would like to see, beforehand, is a proposal on the development model using pure GIT (not git svn stuff). This document should show both the "casual user" workflow (that will be, at the end, pretty similar to the current workflow) and the "hardcore developer" workflow (which includes branching and stuff... I think they are covered by the HowToUseGit but the workflow is not clear)

This should mitigate any fear of moving to git. The learning curve is steeper than svn, but for a simple workflow you'll end up with using 3 or 4 commands only which are pretty much the same as the svn counterparts.

Anyway, I'm all for moving to git.

-- RafaelAlvarez - 25 Nov 2010

Interposed question from an user: Did anyone try the Trademark Wikis GitPlugin on Foswiki yet? Wouldn't that rise the acceptance of the inevitable considerably?

-- FranzJosefGigler - 26 Nov 2010

Crawford migrated this plugin to Foswiki and uploaded it. He read the code a bit, and stopped. I read it for 5 minutes before concluding whoever wrote that had no idea whatsoever what git was, and its power. So, unless I'm totally wrong (and so is CDot), this plugin should be forgotten and stored in the category: Things you shouldn't do.

To reply to Soronthar, when I wrote HowToUseGit I tried to explain how I was using it, and why. It seems nobody understood, and found it too complex. So I stopped. I agree I should have split it into 2 like you suggest. But any Git SVN crash course would do:

Action	Subversion	git	Remark
Initial checkout	svn checkout url	git clone url	git gets the entire history
Get latest version	svn update	git pull	git gets the new stuff, and tries to merge the tree. SVN merges the files, one by one
Commit	svn commit	git commit -a	git will commit LOCALLY. You have to push your changes back before they're published
Add a new file	svn add	git add	identical
Remove a file	svn rm	git rm	identical
Move a file	svn mv	git mv	identical
Revert a file	svn revert path	git checkout path	identical
Revert a commit	svn merge -c -<rev> svn commit	git revert commit-id;	git revert creates a new commit that removes the changes of the identified commit. svn merge with a negative rev-id removes the rev into the working copy and requires a commit
Get the current status	svn status	git status	identical
Get a diff of changes	svn diff	git diff	identical, but git is faster and more powerful for that, afaik
View history	svn log	git log	identical
Show an individual commit	svn log -v -r nnnn svn diff -c nnnn	git show commit-id	git can show the complete commit atom with commit messages and the diff of changes using a single command. svn requires 2 commands.
Apply a commit from one branch into another	svn merge -c rev	git cherry-pick commit-id	As for a revert, git commits, whereas SVN does not
Amend file with checkins per line	svn blame	git blame	identical
Find which commit broke something	N/A	git bisect	git can apply checkouts between a "known good" and "known bad" release so the bad commit can be identified.
Publish back your changes	N/A	git push	Everything you do in git is local, so before you push, nobody will see anything. It pushes people to commit often, try things, then revert, and when they're ready to push, they clean their changelog

Feel free to amend / update, I wrote that in 5 minutes

Thanks George for amending. Honestly, reading any crash course should be much better and worth it, but at least you have the basics all in one table. A kind of cheat sheet

-- OlivierRaginel - 26 Nov 2010

I want to address some people who might be put off by reading on IRC about some of the problems myself and perhaps others have had with git... But it's worth noting that in my case, most of the problems I've encountered have been dealing with the git-svn side of things, where if it was a pure git workflow I'd likely not be exposing myself quite so much to git's more boring details.

Having said that, git-svn isn't that difficult to operate, and there's still no way I'd go back to a svn client even if foswiki.org stuck with svn for another 10 years.

-- PaulHarvey - 27 Nov 2010

Thanks to Oliver's excellent comparison git vs svn commands, it should be clear that for a contributor most common steps doesn't change that much at all. However there are still some things to lay down: most importantly building a release.

Another crutial point of moving to git is submodules, as Paul already outlined. What we lose when moving from svn to git is partoal checkouts, that is: using git you will no longer be able to check out a specific subdirectory of your choice anymore. This sounds as a disadvantage at first but stems from a fundamental difference in how these two VCSes work.

That's why there are "submodules" in git.

Before being able to MoveCodeRepositoryToGit, we will ultimately have to define which submodules we want to have and how this is related not only to daily work on the code but also how this is related to the release management and of course branching. I am not sure how switching to a different branch of the main module is propagated to the submodules part of the ckeckout.

It might very well be that even though git has got support for submodules, we will have to ease branching and releasing with a set of management scripts. We already have a bunch of them related to building a release. These need to be rewritten very carefully covering the submodule structure whatever they look like.

Related to slicing the source code into submodules is the directory structure in git: some of the practices we have become familiar working with the svn repo might need a review. For one of course, we will not have to keep branches and tags checked out in separate directories per default. But what does that mean for people that are used to run multiple branches simultaneously as virtual hosts or on different ports or paths from the same checkout area now, e.g. foswiki-1.0.x: port 8000, foswiki-1.1.x: port 8001, foswiki-trunk: port 80, all of these running from their respective checked out branch in a pseudo-install env. How does that look like using git? This also relates to how we will backport fixes in one branch to trunk or vice versa.

The last question is sort of on the fringe of two separate oportunities of using git: (1) code management of an open source project and (2) suporting virtual hosting using git.

As part of (2) it would be even nicer to have foswiki's revision control itself on git as well instead of RCS. So docu and wiki apps you write online are checked in directly and don't need an extra ckeckin to YAFR (yet another f* repository).

-- MichaelDaum - 27 Nov 2010

Michael, some of the stuff you said are either incorrect or misleading, unfortunately.

1. Partial checkout: If you mean that with SVN, you could, from anywhere, simply svn co http://svn.foswiki.org/trunk/SomePlugin, then yes, you cannot do that with git. But as soon as you have a local repository, you can clone it, or checkout whatever you want in whichever branch you want.
2. Several branches. I do that, and I think George explained how I do it in HowToUseGit. Basically, you git clone your main repository (which on linux takes almost no space, as git will create hard links for most of the files in .git/, and then you git checkout the branch you want. For example:
   • git clone github... -- create a directory foswiki with everything in it
   • git clone foswiki master -- to create a local copy. By default it should create hard links if possible. Play with the options if you want more flexibility
   • git clone foswiki release -- to create another local copy
   • cd master && git checkout master -- just to be sure
   • cd ../release && git checkout Release01x01 -- to checkout the release branch
   • git checkout master SomePlugin? -- to checkout a plugin residing only in trunk (so non-core), but test it in your release branch

The rest is perfectly sound. The GitPlugin? FranzJosef? mentionned was targetted exactly at that, but honestly the implementation is so quick & dirty I'd rather rewrite from scratch.

As per the structure, I would push to create a repository for each plugin. Then either we can have core as a special repo, with submodules, or have core as another submodule, and have a special foswiki repo consisting mostly of submodules. It can be seen as way more complex, but git eases the maintenance (something like git submodule update --init --recurse).

-- OlivierRaginel - 27 Nov 2010

Olivier, thanks for the clarification. -- Fixed my name

-- MichaelDaum - 27 Nov 2010

I've just started using repo-per-extension at work. I absolutely think this is the way to go - I suppose the only thing that might be more difficult are the 'global' changes we sometimes do in one commit; these would need a separate commit on every plugin

• Thank God there is git submodule foreach -- OlivierRaginel - 28 Nov 2010

-- PaulHarvey - 28 Nov 2010

Added a script which makes git-repo per extension.

• gitrepoperextension.pl.txt: gitrepoperextension.pl.txt

Let's use this to get each extension up on github as a submodule of foswiki, and then do yet another script so that developers can clone them back with svn tracking properly enabled so we don't lose ability to dcommit back to svn.

-- PaulHarvey - 19 Dec 2010

fun - next up, ./pseudo-install.pl does everything, including get the source, updates things - ala ./pseudo-install.pl rebase developer to update all the developer modules, or ./pseudo-install.pl checkout Release01x01 developer to convert all the developer modules to 1.1, while leaving the other plugins there for testing...

-- SvenDowideit - 19 Dec 2010

http://git.trin.org.au/foswiki/ is now finished sync'ing all ~500 extensions from svn into their own separate git repos, with self-contained scratch/trunk/release branches. There won't be any updates until I have finished the script which will intelligently run git svn fetch --all only on those extensions which have had changes.

-- PaulHarvey - 20 Dec 2010

From discussions with pharvey and babar on IRC:

• git repo organization:
  

• Foswiki core becomes a git super-module.
• pulling down the Foswiki repo gets the core structure, and an empty directory for each Extension
  • git clone --recurse will populate all of the Extensions, similar to a full SVN checkout
• git fetch in the sub-module will populate the sub-module from the repo.
• supermodule points to a specific commit of a sub-module
  • Need to update both the sub-module and super-module to apply a change

Issues?

• Extensions currently use $Rev$ as an increasing rev number. - no equivalent in git.
  • Core extensions change to sync with Foswiki version ex. 1.1.2
  • Core extension released outside of a Foswiki release gets a 4th octet. TinyMCEPlugin 1.1.2.1
  • Non-core extensions need to be converted to a x.x.x release numbering scheme
  • Possible alternative. Use a rest plugin to generate a repo-wide rev number during build.pl release for backwards compatibility
• per-extension fetch is complex for new developers
  • Use the --recurse option to pull the entire repo.
  • Consider adding git support to pseudo-install. So pi of a "empty" directory triggers the fetch to populate the extension
• How do we maintain multiple streams in a sub-module, adding a commit to the Release01x01 version without picking up trunk commits?

-- GeorgeClark - 20 Dec 2010

Branching and sub-modules. After further IRC discussion, each sub-module needs to branch however the pointer from super-module to sub-module is still based on a specific commit. This shows the relationship between the core branches and an individual core extension branches. The only relationship is the commit id. The branch tag is not meaningful when pointing from super to sub module.

All of this is subject to verification. This is somewhat conjecture.

-- GeorgeClark - 21 Dec 2010

Added a quick & dirty script to connect a bunch if git cloned repos back up to svn. Example:

git clone http://git.trin.org.au/foswiki/core.git
git clone http://git.trin.org.au/foswiki/TinyMCEPlugin.git
git clone http://git.trin.org.au/foswiki/TablePlugin.git
./fixclonedrepo.pl *
find . -maxdepth 1 -type d -exec bash -c 'cd {} && git svn rebase' \;

NB: http is a very slow transport for git, need to get these repos up on github.

-- PaulHarvey - 22 Dec 2010

• fixclonedrepo.pl.txt: Fix a git cloned repo so that it can dcommit/svn rebase again

• pseudo-install.pl.txt: a pseudo-install that tries to fetch with git if the module is missing

  ---

Some TODO's ...

• tools/check_manifest.pl is tied to svn commands to list the files in the repository. (Fixed by OlivierRaginel on trunk)
• lib/Foswiki/Net.pm uses the keyword substition of $Rev$ to set the agent string for net requests by Foswiki
• tools/build.pl sets the keyword LASTBUILD - this is not supported by git svn.

One big issue is the svn Keyword expansion on checkout - namely $Rev$ and $Date$.

• $Rev$ is used in the TOPICINFO meta version= string. The foswiki build script replaces it with "1", so it is only of use to developers running on a svn checkout
• $Rev$ and $Date$ are used in the module documentation - as in "API version $Date$ (revision $Rev$)" so that the perldoc tool shows version information.
• $Rev$ is set in many modules $VERSION strings
• $VERSION = '$Date$ $Rev$ ' is set in Foswiki.pm

Crawford requested that we replace the TOPICINFO rev info with a HTML comment in every topic. <!-- Foswiki Checkin: $Date$ --> If we go ahead with this

• Should these comments be removed prior to distribution, like BuildContrib? does with the Version= string?

The keywords are substituted by the svn client upon checkout, but git has no comparable capability. How about the following solution:

• For Topics and modules, define a %< macro that can be used by equally by SVN and GIT users to expand into revision information
  • Topics: create a formal document footer: "Foswiki Documentation: Revised: %macro
  • Modules could use it in the pod section for expansion by a plugin during perldoc processing, and in the $VERSION string
• Build.pl would replace these macros with static information from the repo during the build.
• Need to consider Dependency.pm - Dependency checking is the only place I can find $VERSION or $RELEASE being referenced.

Maybe a footer in every Foswiki distributed topic:

Foswiki distribution document: %REVSTAMP% %IF{"'%REVINFO{format="$rev"}%'!='1'" then="Local Revisions"}%

Where REVSTAMP is either dynamic repo information or a static string inserted by build.pl

-- GeorgeClark - 07 Oct 2011