August 2nd, 2008 by rickbradley 4 comments »
These days it seems that everyone is on the github tip – cloning like a shitty star wars prequel, forking and pulling like a lobster banquet. With github’s success have come a few growing pains – GFS is evidently a rude ho, which means that sometimes you can’t always just take the bus and get there, metaphorically speaking.
Since git doesn’t rely on a centralized repository, github occasionally going pensive like a French actress shouldn’t really cramp one’s style as far as development is concerned. Just commit locally, gitjour with your friends, and when github is back up,
git push github master. Badda bing.
It could conceivably be a bit of a headache if you’re hoping to deploy servers with capistrano or vlad and you’re relying on one single service to be up whenever you want to deploy. Since git is so decentralized, however, it seems only natural to just use that decentralization to our advantage and try to eliminate single points of failure. Redundancy, bitches!
So let’s talk about how OGC rolls.
We were hosting our own internal git repos on our own server before github was publicly available (well, to be fair it was Michael’s server, and we eventually broke the lock off the wallet and rented some slicehost action). Once github came along we hopped on and started pushing our open source projects over there as well. We were pushing to both repositories “by hand” for a while, meaning that we’d
git push origin master and then
git push github master.
First off: ass pain. We could make some aliases or shell scripts, or maybe use git push-all. Meh. Second off, this ain’t DRY – the opportunities for lost or mismatched codez on different servers are too-ubiquitous (“toobiquitous”, in fact) for my liking. Third off, this is clearly an open rabbit hole. Aren’t we obligated to climb into it?
So we made a number of changes. Among them, we started using gitosis to manage our repos (both public and private) on our slicehost. Then we made some configuration changes so that, once a project is setup, pushing to our slicehost repository for a project will automatically force a push to our github repository for the project. Sweet.
So, now, mirroring is easy, though setup on a per-project basis is still a bit cumbersome. We want to document what we’ve done, not because it’s beautiful (yet) but to get it out there and then improve it. Oh, and (to drop the Royal “we”) because ymendel keeps bitching that I need to write this damned blog article. [waves to Yossef.]
Here’s how we’re doing the mirroring… In the discussion that follows there are three machines involved: (1) a local workstation (or laptop, as the case actually is) where I’m doing development work, I’ll refer to it as “local”; (2) a git repository host under our control, which will be running gitosis, to which we push, and which then mirrors to github, called “internal”; and (3) github.com. Suggestions for process improvement are very welcome as there are still a few too many moving parts involved in setting up new projects.
To get things set up initially, get yourself a unixy/linuxy host where you can set up gitosis for hosting. Slicehost, linode, some dude’s box, whatever.
Set up gitosis on this “internal” machine. We have a user named ‘git’ to run gitosis. The repositories are in
~git/repositories. I followed this article on gitosis.
Make sure the internal repo user has an ssh key that can be used on github:
Add the public key (
~/.ssh/id_dsa.pub, for example) to your github account’s key list.
Set up a repository for your project on github.com.
Set up a repository for your project on your internal gitosis server:
local:~$ cd ~/git/gitosis-admin local:~/git/gitosis-admin$ git pull origin master local:~/git/gitosis-admin$ vim gitosis.conf ## add the project local:~/git/gitosis-admin$ git commit -a -v local:~/git/gitosis-admin$ git push origin master
Actually make the internal repository by doing a push to it:
local:~/git/gitosis-admin$ cd ~/git/ local:~/git$ mkdir foo local:~/git$ cd foo local:~/git/foo$ git init local:~/git/foo$ touch README local:~/git/foo$ git commit -a -m "Initial repository push." local:~/git/foo$ git remote add origin git@internal:foo.git local:~/git/foo$ git push origin master:refs/heads/master
UPDATE 2009-04-08: This is the start of the no-longer-necessary section. Check out the new hotness.
Set up a remote called “github” on the internal gitosis repo pointing to github (use your username instead of flogic, duh):
internal:~git/repositories/foo.git$ git remote add github email@example.com:flogic/foo.git
Set up a hook in the internal repository so that when your commits come in to your internal repository they will be pushed out to the corresponding github repository. We used post-update, but post-receive should work as well (discussion of the various hooks git understands). Thanks to Evan Phoenix for the kick-start to get this working:
#!/bin/sh # # An example hook script to prepare a packed repository for use over # dumb transports. # # To enable this hook, make this file executable by "chmod +x post-update". git-update-server-info exec git push --mirror github >> ~/github.log 2>&1
Make sure the new hook is runnable:
internal:~git/repositories/foo.git$ chmod +x hooks/post-update
(One question I have is, is there a way to do this with gitosis that doesn’t involve logging in on the server to add the remote and setup the hook?)
UPDATE 2009-04-08: This is the end of the no-longer-necessary section. Check out the new hotness.
We still set two remotes in our git configs, in case we ever want to manually push straight to github – our server could go down too, y’know (again, substitute your github username for “flogic”):
local$ git remote add origin firstname.lastname@example.org:foo.git local$ git remote add github email@example.com:flogic/foo.git local$ git config --list | grep ^remote firstname.lastname@example.org:foo.git remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* email@example.com:flogic/foo.git remote.github.fetch=+refs/heads/*:refs/remotes/origin/*
Push to the internal server … it should just magically show up on github:
local:~/git/foo$ echo "test" > README local:~/git/foo$ git commit -a -m "testing mirrored push to github" local:~/git/foo$ git push origin master
For troubleshooting, we adopted Evan’s idea and have a log file in the home directory of our internal server’s git user. The log file will show the output of attempting to
git push --mirror to github.
Anyway, happy codes sharings!