April Updates

We pushed in some great updates to the Code Spaces’ tool suite that should help improve your day-to-day experiences.

• All Subversion Repositories are running on SVN 1.7 - Earlier in the year, we rolled out our new infrastructure which meant all new SVN repos were running on SVN. We have now transitioned all of our old repositories over to this infrastructure. The change is completely transparent — the only thing you will notice is a better performance
• Delete Git Repos - Previously, CS allowed you the ability to create and update your Git repositories. If you needed a Git repo removed, you had to submit a support request for us to do so. While this limitation wasn’t desired on our part, we felt it more crucial to deliver the Git hosting capabilities as soon as we could, and deploy the ability to remove the repos once the feature was tested and stable. I’d like to announce that you can now delete your Git repositories in the same fashion that you would with SVN.
• Git Repo creation while creating a new project - Previously, the only type of repository you could create while creating a new project was SVN. Not anymore; simply chose SVN or Git from the Repository Type drop down to choose.
• Upload Document to Work Item from any page - Last month, we pushed a handy, new feature that allows you to upload a document directly to a work item, which got some really great feedback. Unfortunately, it was quickly pointed out to use that the feature wasn’t working quite right on certain pages. Long story, short: that nasty bug has been squashed.
• Forgot Your Password - Another feature that many people were asking for was the ability to reset their password when they forget it. That ability is now live, and can be accessed from our login page.

• Backups - As you know we take backups every 5 minutes, right? What you may have not known, is that these backups are stored off-site, and we used to copy all backups back on-site so you could download them whenever you wanted. This, however, was an expensive process: moving 1000′s of backups around every day took an enourmous amount of bandwidth and processing power. We have changed the backups download section to an on-demand model, where you can request that a backup is copied from the off-site storage to your account where they can be downloaded. Check out the backups section of you Code Spaces’ account for more infor.
• No more email bombardment - Many of you find the work item notifications extremely helpful for staying up to date with the progress of your project. The majority of you also found it extremely annoying when the system told you about updates that you were doing. Well, that is no more, and you will now only be notified about things you yourself didn’t do.
• New Notification Option - You can get emails when you make a commit to a SVN repo, as well as the standard emails when somebody else commits, check out your profile page for all the options.

Whats Next?

We are keen to get your feedback on CodeSpaces: if you have any features you’d like to see added, or know of bugs you’d like to see squashed, please let us know.

• Git Notifications – In our quest to bring our Git support inline with our Subversion offerings, we are working on Git Email Notifications for your project members when a Git change is pushed to a Code Spaces repo.
• Deployments - Many people have asked for auto deployments, so when you make a commit to a repo, you can specify deployment rules where your Code Spaces project will push your code to a remote server.
• Web Hooks - When you make a commit to a SVN or GIT repo you can specify remote services to ping, for instance you could fire you own backup server to take a copy, or kickoff an automated build process.
• Much More… We have so much more we are looking forward to sharing with you

At the time of writing our uptime for the last 5 years has averaged at 99.9% however February 2012 is currently (at the time of writing) at 92.061%!! This is clearly a deterioration on our normally excellent levels.

What went wrong?

Code Spaces is growing at an exponential rate, the number of new customers we acquire is going up every week and the amount of data we store is growing too. We saw this coming and made a number of strategic changes towards the end of last year to enable us to support our ever growing number of customers. The new infrastructure we rolled out is amazing; its quicker than anything we have seen before and scales in a way that will allow us to grow like never before. The problem that we face, however, is moving all of our existing customers over to this new infrastructure in a transparent way. That process has been happening for about 2 months now and the issues we are seeing are with those customers, we have completed about 95% of the transition and, while a few edge cases still exist, we are confident that the 99.9% uptime and availability that we are so proud of will be maintained going forward.

Everyone here at Code Spaces and AbleBots LLC are committed to ironing out the remaining issues and exceeding your expectations. We apologize for the inconvenience caused by the recent outages if they have affected you. Please do not hesitate to contact us if you have any questions or concerns.

Subversion 1.7

Code Spaces now has full support for SVN 1.7, allowing you to take advantages of it’s new features. A request we frequently get from CS users is the ability to export your SVN repository on demand. In addition to the realtime backups CS currently offers, 1.7 users can take advantage of the new SVNRDUMP utility, which allows you to export your repo without requiring access to the remote server. Additionally, SVN PATCH was introduced to working along side of SVN DIFF for creating and applying patches.

As far as what 1.7 provides for Code Spaces, two words: “increased performance.” With  this release, the CS architecture was rebuilt from the ground up to push the performance to its limits: the new 1.7 protocol has been streamlined which means less overhead and quicker throughput for faster response times. On top of this our new architecture includes performance gains for repo authentication and fine grained permissions, access to commits, log requests, you name it. Our new architecture provides a very extensible foundation that will enable us to realize some of our more ambitious plans for Code Spaces like IP White/Blacklisting, and many more widely requested features. Keep an eye for these new features in the upcoming months.

SVN 1.7 also makes it easier for users who manage more than one Code Spaces account. Because of the way the SVN clients stored credentials, accessing multiple accounts was a bit of a pain point. SVN 1.7 introduces a slightly different URL scheme for repositories:

https://ACCOUNT_NAME.codespaces.com/svn/REPOSITORY_NAME

This allows your SVN client to properly identify different accounts, ensuring that it always uses the proper authentication.

Work Items

Work items will receive a great detail of attention this year — over this past year, we’ve received all types of great feedback from our customers on ways to make work items even more powerful for their teams. We started with two big requests:

• Assigning owner’s of work items: Previously, if you created a work item, you became the owner. This works the majority of the time, but often you’ll want to create a work item on behalf of someone else. For instance, a project manager assigning a bug item to a developer on behalf of a quality control member. In short, now you can — work items now let you define who is the owner of the work items.
• Upload attachments from work items: While the ability to link previously uploaded documents to work items has always been available, the ability to upload attachments right from the work item was often ask for. We’ve accommodated this request by including an “Upload Attachment” button to the bottom of the new work item form, as well as creating an “Attachment” tab on existing work items, from where you can upload new attachments or link an existing document. Work item attachments go in to a special “Work Item Attachments” folder found in your documents, which contain subfolders whose name matches the Work Item they attachments belong to.

Wiki Updates

The Wiki got some much needed attention as well..

• Better default page: in short, users tend to be a little confused about how to get started  using the wiki for the first time. We provide a quick explanation as the default for every new page, as a means to get you up and running quickly.
• WYSIWYG: We have included a WYSIWYG editor for the wiki to help you create more powerful pages.
• Better default style: You’ll notice the CSS of the wiki has been cleaned up as well. We incorporated a better defined default CSS template that should help to make your content look more mature.
• Customizable CSS: To take the Wiki CSS a step further, you can now edit the stylesheet used by your wiki. Simply click the “Customize CSS” button found in the Wiki toolbar, and you can make real-time updates to the way your Wiki looks.

Honorable Mentions

• Layout theming: We’ve added a more prominent header to the app, including the name of the current project for easy referencing.
• Added invoices to the Payments page.
• Fixed bug where new git repositories were being created twice.

Creating a repository is as simple as clicking the “New Git Repository” button in the admin section toolbar. Once you have selected a name, etc.. you can click save and the repository will be created in Code Spaces (it can take a minute to create the Git repo on our servers).

Permissions

Git access permissions are setup in the repository settings, and you have a few options:

Basic Mode

Basic permissions mode will gave every user who is a member of the repositories linked projects full R/W access to the repository. This is great for quickly getting a project going where all members are going to need full access.

Advanced Mode

Advanced mode is more granular then Basic mode and requires that you explicitly set permissions for each user who will have access to the repository, you can also set up path based permissions in this view, but the most basic advanced permissions is to give “A User”  ”R/W” access to the path “/”.

SSH Keys

No matter what permissions mode you use you wont get access to your Git repository until you set up a SSH key. The following instructions are specific to Mac OSX users but are almost identical to Linux, and Windows instructions are just as easy (here is a great Windows specific write up).

Creating a key for use with Code Spaces

In your favorite terminal go to your .ssh folder and run the ssh-keygen command:

So running the ssh-keygen command will first ask you for a name, i used codespaces_ssh_key in this example but it really doesn’t matter what you call it (its good to use a descriptive name though). Secondly is prompted for a passphrase, i did enter one and simple hit the return key (for a blank passphrase) again this is personal preference but adding a passphrase of your choice here is up to you. (NOTE: this is not your CS password).

And thats it, if you look at the files in your ~/.ssh folder you will now have two new files (codespaces_ssh_key and codespaces_ssh_key.pub in my example).

Uploading your SSH Key to Code Spaces

The file your interested in is the one that ends in .pub so open it up in your editor of choice or just cat it into your terminal output:

Copy the contents of your file as this is what you will be adding to your Code Spaces account to make it all work (pretty geeky ey?)

Open up your Code Spaces account and navigate to your Profile page, you will see a tab names “SSH Keys” and in there you will an area where you can add a key (Key name and key Value). In the Key Value field copy the key from your clipboard and give it a nice name in the Key Name field:

And thats it, once you click the “Add Key” button Code Spaces will associate your key with your account and all the permissions you set up earlier will work.

To clone your repo run the following command:

git clone git@codespaces.com:account_name/repo_name

Troubleshooting

Unable to Clone Repo (Permission Denied)

If you have setup permissions (either basic or advanced) and have your key uploaded but still get an access denied message you can run the following command from the command line to test your configuration:

ssh -vT git@codespaces.com

This will give you a verbose output and will indicate which key was presented to the server and finally Code Spaces will give you a list of repositories it thinks you have access to.

Can’t push in to a new Repository

The first time you try and push code up to Code Spaces (in an empty repository) you might get an error like:

error: src refspec master does not match any. 

fatal: The remote end hung up unexpectedly

This is normal and can me solved my running the following command

git push origin master

The update consists of:

• Subversion 1.7 Support.
• New notification service for Subversion Commits.
• Quicker authentication system.
• IP address based whitelists for Extra Security.
• Recurring Billing.
• Various Bug Fixes.
• News Letters

Subversion 1.7 Support

Once we roll out this update any new repositories you create will be served by version 1.7 of the Subversion Server, any old repositories you have will remain on 1.6. To move a 1.6 repository to 1.7 there will be a “Upgrade Repository” feature that will move your repo from the 1.6 based infrastructure to our new 1.7 based infrastructure. The move will also represent a change in the repository URL so the existing scheme :

https://svn.codespaces.com/[YOUR_ACCOUNT]/[REPO_NAME]

will change to:

https://[YOUR_ACCOUNT].svn.codespaces.com/[REPO_NAME].

Notification Service

The notification service that sites on our Subversion Infrastructure is 5 years old and its absolutly clear that we have outgrown it, currently during busy periods notifications can be delayed by as much as 1 hour which isn’t great when your business and/or development team relies on it to collaborate efficiently. The new notification service is a dedicated out-of-process infrastructure that can scale automatically to meet the notification demands. This end result is consistant, almost instand email notifications of Subversion Commits. This new infrastructre also enables us to offer other post commit notification and processing services (like SMS messaging, Continuous Integration, Deployment) which we will look at early in the new year.

IP Address Whitelists

A number of our enterprise customers like to restrict access to SVN repositories not only by user but also by IP Address, this security feature is a must for many larger corporations who employ strict change control and security procedure, with this update we are bringing this enterprise class feature to Code Spaces with configurable IP Based permissions to compliment our already ground breaking granular permissions system.

Recurring Billing

Our payment system has been upgraded to allow recurring billing, so no more emails with payment reminders

Bug Fixes

We are always fixing bugs and rolling out minor changes, however because the Subversion 1.7 update is so large we are currently on a change freeze so all bug fixes and minor changes will be rolled out as part of this larger release. commencing the new year we will return to our more Agile release process and will be free to rollout bug fixes and minor changes within a week of them being approved for production.

News Letter

From the beginning of the new year we will be sending out a monthly new email to all account admins, this is a simple opt-out from our Code Spaces profile, however it will be only once a month and it will enable us to keep you informed on the upcoming changes to Code Spaces. Please note all important updates will also be posted here on this blog and on Twitter (as normal).

As always please let us know if you have any comments or suggestions?

What you need.

Before you can move your data in to our infrastucture you will need an SVN Dump of your existing repository, If you running your own SVN infrestructure you can do this with the svnadmin dump command as follows:

svnadmin dump repository_path > repository_name.dump

If you repository is hosted with a third party you will need to use the mechinism they have or raise a support ticket to get a SVN Dump of your data.

When you have it.

Once you have your dump file, you are ready to import your repository to Code Spaces, we have 2 methods depending on the size of your repository.

Method A – For smaller dump files

Code Spaces has a repository import function that you can use to upload a dump file and have the system auto import it for you. To use this create a new repository in your Code Spaces admin section and select the import repository option.

This option is ideal if your repository is less than 100MB.

Method B – For larger dump files

If you repository dump file is large (> 100MB) you may find that our manual import process is more efficient, simply raise a support ticket with us and send us a link to the dump file, we will (usually within 24 hours) be able to import of for you and verify the new repository for you.

If you have any problems with Method A or your repository is large, this metod is the best approach.

Our ‘A’ Game

Technical Support is an area where we often trump our competition, we try to respond as soon as possible and to be as personal as possible at all times. If you email us a support request we try to break the common perception that emails to support departments go in to black holes and rarely come back, We try to exceed your expectations and most of the time I think we do.

With that said, managing 24/7 support requires a lot of hard work and commitment, because we want to be fresh and as helpful as we can at all times (yes Sunday evening at 23:00 hrs too) we need to provide our support reps with the tools to enable them to do there job from wherever they are.

Raising the Bar

We recently moved all our support to Collabable.com, and while the platform itself is currently in Beta we couldn’t be happier. Collabable enables us to respond to customer support requests, sales queries and general customer communications quicker and more conveniently than ever before.

Any user email we recieve generates a dedicated discussion in the Collabable system, which we can manage form any location at any time, we can use our mobile devices (ipads, iphones) as if we were tied to our desks. These discussions are recorded for the user to review long after they are closed, we hope this will be a valuable addition to our toolset which will benefit all.

While we hope that you will never need to use our Support Services, should you need anything from us, its now simply a Discussion Away!

It was an interesting day, at around 8:40 (GMT) we started getting notifications from our monitoring instruments that our database cluster was failing over, one node at a time.

Now this alone normally wouldn’t be a problem because we host with Amazon EC2 so new nodes should automatically be firing up and joining the cluster, thus keeping the cluster alive.

But it was no “Normal” day, and new nodes didn’t fire up and the cluster did go down.

About Amazon EC2

Its safe to say that Amazon EC2 is an amazing implementation of Cloud Computing and the mechanisms they have in place to allows companies like ours to scale, and stay up are the best i have seen in the past 12 years of working in and running internet companies.

Amazon allow us to create “instances”, these are VM running our own software stack. They allow us to create an infinite number of them, and we can switch them on and off again at will.

Amazon also have a storage solution called EBS (Elastic Block Storage) which allows us to provision virtual disks that can be attached and detached to instances at will.

For the last four years this system has given us ~99.9% uptime and has allowed us to scale to meet our customer demands.

What happened to Amazon yesterday?

Yesterday amazon had a network failure that triggered a re-mirror of all the EBS volumes in the North Virginia data centre, this re-mirror combined with a large number of customers trying to fire up new instances to recover from the network failure caused Amazon to hit a storage capacity problem.

Basically they didn’t have enough disk space to support the re-mirror and as the thousands upon thousands of customers they have tried to fire up new replacement images the problem compounded its self.

Now its not for me to comment on Amazon running out of disk space as i’m sure there is more to it than that, but those are the details that Amazon have shared with us.

So why did Code Spaces go down?

Take a quick look at the Infrastructure overview below, this is a very high level view of our world, basically each cluster relies on EBS volumes to store data and to perform block level (real time) backups (to the offsite backup devices):

Our DB cluster was in the effected region when amazon had its network failure yesterday, and all our DB volumes were trying to re-mirror and failing due to the capacity issues described above.

Had we known that the outage was going to be as long as it was (Amazon is still experiencing this issues 27 hours later) we would have moved our DB cluster in to a new availability zone and started new volumes from our backup.

Unfortunately Amazon took a long time (8 hours) to let us know that this was a big issue, and for those 8 hours we sat on our hands wondering if it would come back… It didn’t and at around 22:00 GMT we decided we (and you) had waited long enough, and we started our disaster recovery process.

Why did you wait 8 hours?

This is where we made our mistake, we thought (even after 8 hours) that amazon would be only minutes away from solving this, Why? well Amazon didn’t communicate the severity of the issue in fact for the first 8 hours of the outage there network status page indicated that they had “Performance Issues” (See screenshot). As soon as they admitted that this was a Severe Issue we started our disaster recovery process, it took 15 minutes for us to move everything to a new availability zone and to have everything up and running with full data integrity (no data loss).

So you waited 8 hours to do a 15 minute recovery?

In a nut shell, yes! We waited (and waited) when in reality we could have implemented our recovery process and had you guys back up in 15 minutes, for that we are genuinely sorry!

Lesson Learned.

Its still early to fully understand the impact of this, Amazon is still experiencing these issues, and many (many) sites are still down as a result of this (see http://ec2disabled.com for a complete list), However we have certainly learned a lot, and will be implementing the following ASAP:

Communication Channels

We need to be able to let you guys know whats happening in these extreme cases, so we have provisioned a server outside of Amazons infrastructure to hold a service status page, this will also hold a landing page for www.codespaces.com in the event that this happens again.

We will also be setting up a regular email to all customer (you can opt in or out) which will enable us to communicate with you and will give you a way to communicate back with us.

Strict Processes

We have a disaster recovery process and we have the mechanisms in place to implement it in 15 minutes, we have changed our internal processes and added a few more to ensure that on any kind of outage we are ready to push the button and get everything back up and running rather than waiting for our hosting company to do it for us.

The not so lucky ones!

It appears as if there are many sites still down as a result of this, I can only assume that these sites do not employ the level of backup that we do and do not have a well practised and thorough recovery process that we do.

When (or if) these companies will get their data back is still not clear, and I really hope that these guys learn the larger lessons from this.

A long hard look at ourselves

I think this raises an interesting conundrum for anyone who relies on a third party for a core part of their business, be it you guys who use us for hosting your valuable code or companies like us who invest heavily in hosting companies to keep us going. You need to be sure that you can rely on the 3rd party to “do the right thing” and have provision to ensure that even in the worst eventuality your investment is safe.

While i wish we had reacted quicker (and we will learn from this) and started our recovery process earlier I hope we have proved to you that even under the worst eventuality (and it doesn’t get much worse than this, again look at the growing list ec2disabled.com) we have the systems in place and have invested in your best interests in infrastructure that not only scales but provides us with the confidence and security that we can not only keep your data safe but recover from the worst kind of outage.

Here’s to a new day, with new (but slightly less chaotic) challenges!

If you would like to discuss this further please feel free to email me at f[dot]price[at]codespaces.com, I will also be holding a number of Web Chats with people so please contact me if you would like to join one of those.

We have an API that will expose the backup data (file locations, etc…) which will enable you to easilly automate this, and store the backups yourself.

The api is accessable by doing a HTTP GET to:

http://[YOUR_ACCOUNT].codespaces.com/api/backups.xml

or

http://[YOUR_ACCOUNT].codespaces.com/api/backups.json

As you can see the API will expose the backup data in XML or JSON format.

NOTE: you will need to pass your login credentials with the request.