home | sitemap | abstract | introduction | chaos | thinking | checklist | migrating | recovery
pushpull | cost | career | workshop | isconf | list_and_community | papers | references

Version Control

Lack of version control over your infrastructure leads to eventual confusion. We use version control for tracking OS configuration files, OS and application binaries and source code, and tools and administrative scripts. We have managed independent evolution of several infrastructures, and were able to do rollbacks or rebuilds of damaged servers and other components.

It may seem strange to start with version control. Many sysadmins go through their entire careers without it. But infrastructure building is fundamentally a development process, and a great deal of shell, Perl, and other code tends to get generated. We found that once we got good at "doing infrastructures", and started getting more work thrown at us, we had several distinct infrastructures at various stages of development at any given time. These infrastructures were often in different countries, and always varied slightly from each other. Managing code threatened to become a nightmare.

We found that CVS helped immensely in managing many different versions and branches of administrative code trees [cvs] . It took some careful thought and some tool building to be able to cram O/S configuration files and administrative code into a CVS repository and make it come back out okay on all the right machines. It was worth the effort. In later iterations, we began migrating the hundreds of megabytes of vendor-supplied O/S code itself into the CVS repositories, with some success. The latest version of CVS (1.10) has additional features which would have made this much easier, such as managing symbolic links natively.

Since all of our code is mastered from a CVS repository, we can actually destroy entire server farms and rebuild them with relative impunity during the course of development, moves, or disaster recovery. This also makes it much easier to roll back from undesired changes.

In short, based on our experience, we'd strongly advise setting up and using CVS and associated tools as the first step in an infrastructure development program.

We've tried various vendor-supplied version control tools -- everyone has their favorites. While many of these seem to offer better features than CVS, none of them turn out to be flexible, robust, or WAN-centric enough to manage operating system code in-place on live machines scatter all over the world. Because of its Internet heritage and optimized use on far-flung projects [samba] [bsd] , CVS is almost perfect for this. Where it isn't, we've been able to get under the hood and get what we need in a way that we would never have been able to with a proprietary tool.

Checklist

Version Control


Gold Server
Host Install Tools
Ad Hoc Change Tools
Directory Servers
Authentication Servers
Time Synchronization
Network File Servers
File Replication Servers
Client File Access
Client O/S Update
Client Configuration Management
Client Application Management
Mail
Printing
Monitoring
Google
Search WWW Search www.infrastructures.org
Unix System Administration
[ Join Now | Ring Hub | Random | << Prev | Next >> ]
© Copyright 1994-2007 Steve Traugott, Joel Huddleston, Joyce Cao Traugott
In partnership with TerraLuna, LLC and CD International