The Tuenti Release and Development Process: Conclusion and the Future

Posted on 1/14/2014 by Víctor García, DevOps Engineer

Blog post series: part 6 (the very last one)
You can read the previous post here.

Success

There is nothing I can say other than the fact that all of this has been a great success for the company. The new process and all of the changes we introduced have improved developer throughput, has speed up the development workflow, has nearly achieved continuous integration and delivery, and in the end, we work better. The site and mobile apps have fewer bugs and overall, we’re very proud of what we’ve done:

  • Self service dev environments: disposable, ready to use, less support required
  • All manual processes automated: less time, fewer errors, more accounting
  • Well known processes: used in every project, less support required
  • Automated pull request: ensure testing and “ever green trunk” what in turn improves continuous integration and people integrate more often.

Buzzwords on the Internet and in Talks: Continuous Delivery/Integration, Agility, Automation...


Most of you have probably heard talks or read articles on the Internet about continuous delivery, continuous integration and all of these cool agile methodologies where everything relies on automation.

For the skeptics, here at Tuenti you can see that sooner or later, they pay off.

Not Following Them? Why Not?

Some of you may think that in your company, things are working just fine the way they are and that you don’t need to change them. You may be right, but what you might not know is that things could be even better. Why not always be ambitious and go for the very best?

Maybe you can think that changing your development workflows and the developer culture would take a lot of effort. Ok, it may not be straightforward, but it’s worth to try.

I’m not going to explain why doing smaller but more frequent releases is better, or why doing things manually is not the way to go, or why continuous integration reduces the number of bugs because you probably are tired of hearing the same again and again and because you can easily find that information by googling it.

How to Change

In newer, smaller companies, changing processes is not as hard as it is in bigger companies with old workflows and old people that are used to working with prehistoric methodologies.

Here are some tips that might help you make such changes in these more complicated settings:

  • Sell your idea by demonstrating success stories from other companies, clearly stating why change is better.
  • Start implementing something that makes the developers’ lives a bit easier.
  • Make small changes (Minimum Valuable Product). Bigger changes involve many ruptures in the process and developers won’t like them.
  • If they like your first MVP, you won’t have problems getting an opportunity to continue with those changes.
  • Developers will depend on you, be nice to them. Always be helpful and understanding when it comes to their problems. You can even do polls to find out what’s annoying them the most or what it is they need.
  • Automate everything. It ALWAYS pays off.

Transparency for Everyone

When someone in your company (techies, product managers, CTO, CFO... whoever) needs information are unable to get it, you have a big problem.

If everything is automated, it should be easy to gather information and show it company-wide; the release state, the next code that will be released, what that code does, what bugfixes it will include, who the author is, its risks, how to test it, etc.

Create a dashboard that everyone can see (a good design will help), so you can make them aware of, for example, when a new feature is going to be released. This way, for example, the communications/press team can make an announcement or the marketing people can prepare campaigns in time. Otherwise, they will ask you personally and in most of the cases, this is a waste of either your or their time.

And Now What? Next Steps and Projects

We automate almost everything but there are tons of things left to do and tons of way to improve/optimize the current tasks and operations we’ve automated.

Here is a list of some of the things we would still like to do:

  • Open source flow: Many people have requested this from us, and we are willing to do so but at the moment there are some parts of the code that are far too much coupled with Tuenti-specific things. We would need to do some refactoring before open sourcing it.
  • Speed up the pull request system: By testing more than one pull request at a time. For example, if there are two pull requests, A and B, the idea would be to test A and B together and also A only, if A+B is successful, Flow integrates two instead of one, if A+B fails and A is successful, Flow integrates one, if all of them fail, there is nothing to integrate, but it’s demonstrated that this case is the least frequent because in most of the cases the pull requests are successful.
  • Code risk analysis: Analyze this risk in the pull requests system so we can cherry pick pieces of code according to their risk and better plan releases. For example, we could do low risk releases only on Fridays.
  • Improve the development environment by virtualizing in different ways: Tuenti-in-a-box is not always enough to provide a full development environment because of resources consumption that is too high or because it’s not ready for other kinds of environments like Android or iOS developers. We have been thinking about improving the virtualization system, and instead of using VirtualBox, we would like to try some lightweight virtualization providers like LXC. It’s worth researching because we think there is room for great improvements here.

The Tuenti Release and Development Process: The Development Environment and Workflow

Posted on 7/22/2013 by Víctor García, DevOps Engineer

Blog post series: part 2
You can read the previous post here.

The Development Environment

We maintain a couple of development environments: a shared one and a private self-hosted one, and we encourage developers to move to the private in order to ease and speed up the development workflow, getting rid of constraints and slowness caused by shared resources.

Both infrastructures provide a full “dev” and “test” sub-environments:

  • The dev has its own fake user data to play with while coding.
  • The test environment is basically a full environment with an empty database to be filled in with test fixtures able to run any kind of tests.
  1. Unit and integration tests run in the same machine the code does, just using PHPUnit.
  2. Browser tests run in a Windows virtualbox machine we provide with Webdriver configured to run tests in Firefox, Chrome and Internet Explorer.
These two sub-environments are under two different Nginx virtual hosts, so a developer can use both at the same time.

The Shared Environment
The shared environment are virtual machines in a VMWare ESXi environment. Each development team is assigned to a single virtual machine. The management and creation of them is pretty straightforward as they are fully provisioned with Puppet. They provide full dev and testing environments with its own Nginx server, Memcache  and Beanstalk instances, a HipHop PHP interpreter, MySQL database, etc. All these resources will be shared among all the virtual machine users. The development database, HBase, Sphinx and Jabber servers are hosted in a separated machine, so, these particular resources are shared by all users.

That’s why we’re currently moving everyone to the private environment. We’ve had problems with shared resources, such as when someone executes a heavy script that raises the CPU usage to 100%, makes the memory swaps, or slows the machine down due very high IO usage, sometimes even affecting users sharing the same physical machine.

The Private Environment: Tuenti-in-a-Box
Tuenti-in-a-box is the official name for this private environment. It’s a VirtualBox machine managed with Vagrant that runs on the developer’s personal laptop. The combination of Vagrant and Puppet is a fantastic thing to set this up since it let’s us provision virtual machines with Puppet very easily, every developer can launch as many private virtual machines as he wants just with a simple command. Thus, Tuenti-in-a-box is an autonomous machine that provides the above-mentioned full dev and test environments, ridding us of problems with shared resources. Every resource is within its virtual machine with one exception, the development database that is still being shared among all users.

Right now, project exists that lets every Tuenti-in-a-box user have its own database with a small and obfuscated subset of production data.

The Code Workflow Until Integration

Developers work within their own branches. Each branch can be shared among other developers. These branches must be up-to-date, so developers frequently update them with the integration branch, which is always considered to be safe and with no broken tests (we will see how this is achieved in the next post).

Each team organizes its development in the manner they want, but to integrate the code, some steps must be followed:

  • A code review must be carried out by your teammates and it must pass the review:
  1. In order to keep track of what a branch contains and for the sake of organization, every piece of code must be tied to a Jira ticket.
  2. Doing this lets us use Fisheye and Crucible to create code reviews.
  • The code is good enough to be deployed to an alpha server.
  • The code can’t break any tests, Jenkins will be in charge of executing all of them.
  • A QA team member tests the branch and gives it the thumbs up.
  • After all of that, a pull request is done and then the branch starts the integration process (more details about this in the next post).
Jenkins Takes Part!
At this stage, testing is one of the most important things. The development branches need to be tested. Giving feedback to developers while they are coding is useful for making them aware of the state of their branches.

Those branches can be tested in three different ways:

  • Development Jenkins builds that only run a subset of tests: faster and more frequent.
  1. ~14500 tests in ~10 minutes
  • Nightly Jenkins builds, that run all tests: slower, but nightly, so developers can have all tests feedback the next day.
  1. ~26500 tests in ~60 minutes
  • Preintegration Jenkins builds, that run all tests: slower but it's the final testing before merging to the integration branch.
  1. ~26500 tests in ~60 minutes
The creation of Jenkins builds is automated by scripts that anyone can execute by command line. No manual intervention in the interface is necessary.

Our Jenkins infrastructure has a master server with 22 slave servers. Each slave parallelizes the tests execution in 6 environments, and in order to be faster, a “pipeline mode” takes 6 slaves and make them run together to executes all tests. You might be interested in what the Jenkins architecture is like and how it executes and parallelizes such a large amount of tests in such a short time, but we’ll leave that for another blog post.

You can keep reading the next post here.

Follow us