The Tuenti Release and Development Process: Deploying Configuration

Publicado el 11/11/2013 por Víctor García, DevOps Engineer

Blog post series: part 5
You can read the previous post here.

Normally, website development differentiates between regular code and configuration code.

Having configuration in a separated code allows you to do quick and basic changes without touching the logic. These changes are safer because they should be just an integer value change, a boolean swap, a string substitution, etc. and don’t involve a full release process.

Some good practices have been described recently on the Internet about how to write your code in a flexible way to avoid useless releases, how to do A/B testing, about how to make your database changes backwards compatible, etc. and all of these good practices involved a good configuration system.

Following the DevOps Culture

Here at Tuenti, we are very fond of the DevOps culture and try to apply it as much as possible. We consider it the way to go and the way to do things efficiently, and it’s there is proof that it has helped us to improve quite a lot.

In our company, the configuration deployment is a clear example of DevOps culture. There is no manual intervention and there isn’t anyone who does deployments such as a release manager or operations guy. Every developer pushes his/her configuration changes to production on his/her own. Therefore, Devs are doing Ops tasks.

We use ConfigCop for that.

ConfigCop

ConfigCop is a tool to deploy configuration to preproduction and production servers. Any developer can and must use it to, first, test their changes in a preproduction server and then, deploy them to production.

Deploy to Preproduction

Preproduction deployments logic are fully done on the client side and the basic options a developer can use is a configuration initialization and a configuration update, the latter being the one that uploads any configuration to be tested.

ConfigCop pull the latests code from Mercurial, gathers all configuration files and stages them all applying some overrides necessary to make it work in preproduction servers and also generating some .json files readable by HipHop.

Then, just deploy them using Rsync and the developer is ready to test.

Deploy to Production

A production deployment must be sequential and cannot be done in parallel because conflicts may arise.

Therefore, for this ConfigCop uses its server side. It’s basically a server-client communication over RPC that establishes a locking mechanism to perform sequential deployments. The lock is given to the developer currently deploying a configuration and it can’t be stolen.

Until the developer has finished a configuration change, another one can’t start.

The workflow is pretty simple:

  • Start a configuration change by just typing a command line.
  • ConfigCop checks if it’s unlocked. If it is, it deploys the configuration change to a preproduction testing server that will always have the same code as production.

    • This is important: we are assuring that the configuration change will work properly in production.
  • Notify the developer that s/he can start testing.
  • Once it’s tested, push it to production with a single command.
  • The deployment takes process and it finishes in less than 1 minute.

Results

ConfigCop freed the release managers of doing these operational tasks and now developers with just 2 command lines are able to test and deploy configuration to production in a easy, reliable, and fast way.

Real Examples

  • Due to a network problem, some chat servers are down and the load in those that are alive is increasing very quickly, we will probably suffer an outage.

    • A developer can just do a config change to temporarily disable the chat to 10% of the users.
    • The problem is fixed in less than a minute.
  • A release requires changes in the database schema, the developer coded inserting in both tables to make it backwards compatible. The release is successfully deployed so the old schema can be deprecated.

    • A developer does a config change to make the code to insert to the new tables.
    • The developer does it on his/her own, nobody is bothered with such task.
  • The product team wants to test two different layouts (A/B testing) for the registration form and wants some users using the old one, and others using the new one so they can measure stats and choose the best one.

    • The developer will play with the percentages of users using A version or B version, and when the final decision is taken, set 100% to the chosen one.

Siguenos