Alternative Github : How To Keep Going When Github Is Under Attack

As anyone who depends on code hosted by Github knows, their servers were recently under attack. This meant that some repositories could not be fetched and some bower operations would not succeed. When your continuous integration cannot run unit tests or build rpm packages because Github is down, you realise that your whole build and release process is blocked due to an unforeseen and previously unimaginable single point of failure and you need to find a way of preventing a repeat of this problem. But does a Github Alternative exist ?

What happened?

For quite some time we have been in the habit of always hosting a clone of all the source code of each library, such as gitphp or restler, that we include in the open source ALM software Tuleap. This helps us to eliminate the dependency on the upstream availability; indeed what would happen if a maintainer decided one day to close or we remove one of their repositories? This had always worked for us until we started to include more and more AngularJS stuff into Tuleap.

AngularJS development includes managing dependencies with bower and npm (you can do it with other techniques but that seemed to worked best for us). Each of our AngularJS applications needs to be continuously built with something along the lines of npm install && bower install && grunt. Npm pulls dependencies into a folder named node_modules and bower pulls them into a folder named vendor (originally bower_components). Digging deeper into this process, we can see in more detail just how dependencies are pulled. When you run npm install, npm downloads packages from the registry https://registry.npmjs.org/ . When you run bower install, bower asks the registry https://bower.herokuapp.com/ for the information about the package (such as where it is hosted) and then runs git clone based on this information. The important aspect here is that, most of the time, the git repository in question will be hosted on Github and therein lied our problem.

The solution

In order to break down the dependency on an external entity (Github or official registries) we have two solutions: 1) to include all dependencies along side our source code or 2) to mirror the registries and repositories from Github.

The first solution, would effectively make the Tuleap source code incredibly hard to work with; so much so that it simply cannot be considered. Some front-end designers do advise it but I’lld be surprised if their initial source code was anything near the length and depth of Tuleap. Not only would it clutter the repository, but it will deeply hamper proper code review. Frankly speaking, I don’t want to include node_modules of node_modules of node_modulesinside our own source code (just open node_modules folder and you will see what I’m talking about).

The second solution, to mirror the registries and repositories on Github, seems like a lot more balanced one. Should anyone have any other alternative, please feel free to leave a comment.

Sinopia

Sinopia is a private/ caching npm repository server.

Sinopia keeps its own small database of npm packages. It will fetch from https://npmjs.org any required package and then store it locally. If it realises that a package is no-longer needed then it will stop storing a copy of it. It will also poll for upstream changes and update packages that it holds.

Installation is easy:

 $ npm install -g sinopia $ sinopia 

Then, on each development environment (and on the continuous integration server), set the registry:

 $ npm set registry https://localhost:4873/ 

Note: configure sinopia and replace localhost by a url like https://sinopia.example.com.

Now, each time npm wants to install a package, the package will be cached by sinopia. Sinopia is licensed under the WTFPL license.

Private-bower

Private-bower is really the same thing as sinopia. Properly configured, it will cache both the bower registry and the git repositories.

 $ npm install -g private-bower $ private-bower --config ./myBowerConfig.json 

The myBowerConfig.json is the same as default one (see official documentation) but enable the cache of git repositories:

  {     "repositoryCache": {         "git": {             "enabled": true         }     } }  

In the file .bowerrc, we need to add:

 {   …,   "registry": "https://private-bower.example.com:5678" } 

There is nothing to do on dev environment part since .bowerrc is checked-in.

On each bower install, git repositories will be cached on the server, and every 10 minutes the repository is synced:

 [2015-04-02 13:56:19.947] [INFO] console - [bower]  2/4/15 13:56:19   Refreshing cached public git repositories [2015-04-02 13:56:20.938] [INFO] console - [bower]  2/4/15 13:56:20   Pulled latest for angular-moment [2015-04-02 13:56:21.699] [INFO] console - [bower]  2/4/15 13:56:21   Pulled latest for moment [2015-04-02 13:56:21.907] [INFO] console - [bower]  2/4/15 13:56:21   Pulled latest for angular-ui-router [2015-04-02 13:56:22.022] [INFO] console - [bower]  2/4/15 13:56:22   Pulled latest for angular-gettext [2015-04-02 13:56:22.061] [INFO] console - [bower]  2/4/15 13:56:22   Pulled latest for restangular [2015-04-02 13:56:22.229] [INFO] console - [bower]  2/4/15 13:56:22   Pulled latest for angular-mocks [2015-04-02 13:56:22.234] [INFO] console - [bower]  2/4/15 13:56:22   Pulled latest for angular [2015-04-02 13:56:22.672] [INFO] console - [bower]  2/4/15 13:56:22   Pulled latest for angular-sanitize 

Private-bower is licensed under the MIT license.

Next Step

To put this in place, we are planning to build two docker containers to host a server for each of sinpoia and Private-bower. We will then update our Jenkins software development environments so that they fetch data from these servers instead.

You can also read : Alternative Jira : open source ALM software Tuleap versus Jira.