The Test Advisor

Posted on 9/02/2013 by Rafa de la Torre & Miguel Angel García, Test FW Engineers

The Test Advisor is a tool that lets you know what set of tests you should execute given a set of changes in your source code. It is automatic and does not require you to tag all of your test cases.

A Bit of Background

The story of CI has been a tough one here at Tuenti. At one point, our CI system had to handle a huge load of tests of different types for every branch. We are talking about something around 20K test cases, a 2.4h build time, and Jenkins acting as a bottleneck for the development process. From a developer perspective, we track a measure of their suffering, called FPTR (short of “from push to test results”). It peaked at 8h.

Therefore, something had to be done with the tests. There were multiple options: add more hardware, optimize the tests, optimize the platform, delete the tests... or a combination of these options. When considering solutions, you need to keep in mind that products evolve which could potentially produce more tests that worsen the problem. In the long run, the best solution will always somehow involve test case management and process control, but that’s another story for a future post...

After adding more hardware, implementing several performance improvements, and managing slow and unstable tests, we concluded that we had to improve the situation in a sustainable way.

One solution we thought of was to identify the relevant tests for the changes being tested and execute only those tests, instead of executing the full regression. The idea was not novel and it may sound pretty obvious, but implementing it is not at all.

In short, the idea is to identify the changes in the system, the pieces that depend upon those changes and then execute all of the related tests. Google implemented it on top of their build system. However, we don’t have that great of a build system in which dependencies and tests are explicitly specified, and our starting point was a rather monolithic system.

The first approach we took in that direction was to identify the components of the system and annotate the tests indicating what component they are aiming at testing. Thus the theory was: “If I modify component X, then run all the tests annotated with @group X”. It didn’t work well: the list of components is live and evolves with the system and requires maintenance, tests needed to be annotated manually, maintaining synchronicity required a lot of effort, and there was no obvious way to check the accuracy of the annotations.

A different approach is to gather coverage information and exploit it in order to relate changes in source files and tests covering those changes. Getting coverage globally is not easy with our setup. We still use a custom solution based on phpunit+xdebug. There are still some problems with that approach though, that mostly affect end-to-end and integration tests: it is hard to relate the source files with the files finally deployed to tests servers, partly due to the way our build script works. Yes, it is easier for unit tests, but they are not really a problem since they are really fast. Additionally, we did not want to restrict our solution strictly to php code.

What it Is

The Test Advisor is a service that gathers “pseudo-coverage” information to be used later on, essentially to determine what the relevant test cases for a given changeset are.

When it was proposed, some of the advantages that we envisaged were:

  • Reduced regression times in a sustainable way. It would no longer be limited by the size of the full regression.
  • Improved feedback cycles
  • No need for manual maintenance (as opposed to annotating tests)

Regarding feedback cycles, a couple of benefits we foresaw were related to some pretty common problems in CI setups: robustness and speed of the tests. A common complaint was about robustness and false positives: “My code doesn’t touch anything related to that test but it failed in Jenkins”. If we had the Test Advisor, who would suffer bad quality tests in the first place? The ones modifying code related to those tests and who want to get their changes into production. No more discussion about ownership. No more quick and dirty workarounds for the flakiness of the tests. The same applies to their speed. It would be in your best interest to develop and maintain high quality tests :)

How It Works

Most of our product stack works under debian linux. We decided to use the inotify kernel subsystem to track filesystem events. This way, we can get that pseudo-coverage at a file level, independent from the language and the test framework used, or even the test runner.

We developed it using python as the main language, which is a good language for developing a quick prototype and putting all the pieces together. We also used pynotify.

The Test Advisor is conceptually composed of three main parts:

  • A service for information storage
  • A service for collecting coverage information. It is notified by the test runner when a test starts and ends. It has the responsibility of setting up and retrieving information about the files being accessed to complete the test scenario.
  • A client to retrieve and exploit information from the TestAdvisor.

This is how it looks at a high level:

As you can see from the figure, the TestAdvisor can take advantage of any test type being executed (unit, integration or E2E browser tests). It is also independent from the language used to implement the tests and the SUT.

Problems Along the Way

So we were easily able to get the files covered by a particular test case, but those types files are not actual source files. They go through a rather complex build process in which files are not only moved around, but they are versioned, stripped, compressed, etc. and some of them are even generated dynamically to fulfill http requests in production. We tried to hook into the build process to get a mapping between source and deployed files but it turned out to be too complex. We finally decided to use the “development mode” of the build, which basically links files and directories.

Another problem was the file caching. The hiphop interpreter (among others) caches the php files being processed and decides whether a file has changed based on its access time, and inotify does not provide a way of monitoring stat operations on the files. We researched a few possibilities for getting around this:

  • Override system calls to stat and its variants by means of LD_PRELOAD, not an easy one to get fully working (and also made us feel dirty messing there).
  • Instrument the kernel with systemtap, kprobes or similar. A bit cleaner, but a mistake and your precious machine may freeze. We also got the feeling that we were trying to re-implement inotify.

Finally, we picked the KISS solution: just perform a touch on the files accessed between tests. This way, the regression executed to grab information will be slower, but we don’t care much about the duration of a nightly build, do we? :)

Work that Lies Ahead

Right now, the TestAdvisor is used as a developer tool. We’ve achieved great improvements in the CI by other means (most of them process-related changes). However, we are still eager to integrate the TestAdvisor into our development and release cycle within our CI setup.

Developers will probably use a pipeline for their branches, consisting of 3 distinct phases: Unit Testing , Advised Testing and (optionally) Pre-Integration. An “advised build” in green can be considered sufficiently trustable to qualify for integration into the mainline.

This approach has been applied to the core of our products, the server-side code, and the desktop and mobile browser clients. We expect it to also be applied in the CI of the mobile apps at some point.

The Tuenti Release and Development Process: Development Branch Integration

Posted on 8/26/2013 by Víctor García, DevOps Engineer

Blog post series: part 3
You can read the previous post here.

In the previous blog post, we mentioned that one of the requisites a development branch must fulfill is a pull request. This is the only process in Tuenti to merge code to the integration branch.
We really want to have an evergreen integration branch that is always in a good state and with no tests failing. To accomplish that, we created a pull request system managed by a tool called Flow.

The Flow Pull Request System

Flow is a tool that fully orchestrates the whole integration, release and deployment process and offers dashboards to show everyone the release status, integration branch, pull requests results, etc. A full blog post will explain this, so here we’re focusing on branch integration.

Although Flow has the logic and performs the operations, every action is triggered through Jira.

Following the above diagram, here are the steps performed:

  • The developer creates an “integration” ticket in Jira. This ticket is considered a unique entity that represents a developer’s intention to integrate code, so the ticket must contain some information regarding the code that will be merged:

    • Branch contents
    • Author
    • QA member representative
    • Risks
    • Branch name
    • Mercurial revision to merge
  • Then, the ticket must be transitioned to the “Accepted” status by a QA member, so we can keep track and be assured that at least one of them has reviewed this branch.
  • To integrate the branch, the ticket must be transitioned to the “Pull request” status. This action will launch a pull request in Flow.

    • Flow and Jira are integrated and they “talk” in both directions. In this case, Jira sends a notification to Flow informing of a new pull request.
    • Additionally, in the background, Flow will gather the tickets involved in the branch and will link all of these tickets with the integration ticket using the Jira ticket relationships.
    • This provides a good overview of what it’s included in that branch to be merged.
  • Flow has a pull request queue, so they are processed sequentially.
  • Flow starts processing the pull request:

    • It creates a temporary branch with the merge of the current integration branch and the pull request branch that will be merged.
    • It configures Jenkins and triggers a build for that temporary branch to run all tests.
    • It waits until Jenkins has finished and when it’s done, Jenkins notifies Flow.

      • Jenkins executes all tests in 40 minutes, using a special and more parallelized configuration.
    • Then, it checks the Jenkins results and decides whether or not the branch can actually be merged to the integration branch.

      • If successful, it performs the merge, transitions the Jira ticket to “Integrated,” and starts with the next pull request.
      • Otherwise, the pull request is marked as failed and transitions the Jira ticket to “Rejected”.
    • Every operation Flow sends an email to the branch owner notifying him or her of the status of pull request and adds a Jira comment to the ticket.
  • If the pull request fails, Flow shows the reason for this in its dashboard and in the ticket (failed tests, merge conflicts), so the developer must:

    • Fix the problems
    • Change the Mercurial revision in the Jira ticket because the fix would require a new commit.
    • Transition the ticket to the pull request status again to launch a new pull request.

As you can see, this process requires minimum manual intervention, only clicking some Jira buttons is enough. It’s quite stable and more importantly, it lets the developers work on other projects while Flow is working, so they don’t have to worry about merges anymore. They just launch the pull request in their Jira ticket, and can forget about everything else.

So, this is how we assure the integration branch is always safe and that tests don’t fail. Therefore, every integration branch changeset is a potential release candidate.

It’s been proved that this model has improved the integration workflow, the developers performance throughput has increased so it can be considered as a total success.

You can keep reading the next post here.

JavaScript Continuous Integration

Posted on 8/22/2013 by Miguel Ángel García, Test FW Engineer; Juan Ramírez, Software Engineer & Alberto Gragera, Software Engineer

Here at Tuenti, we believe that Continuous Integration is the way to go in order to release in a fast, reliable fashion, and we apply it to every single level of our stack. Talking about CI is the same as talking about testing, which is critical for the health of any serious project. The bigger the project is, the more important automatic testing becomes. There are three requirements for testing something automatically:

  • You should be able to run your tests in a single step.
  • Results must be collected in a way that is computer-understable (JSON, XML, YAML, it doesn’t matter which).
  • And of course, you need to have tests. Tests are written by programmers, and it is very important to give them an easy way to write them and a functioning environment in which to execute them.

This is how we tackled the problem.

Our Environment

We use the YUI as the main JS library in our frontend, so our JS-CI need to be YUI compliant. There are very few tools that allow a JS-CI, and we went for JSTestDriver at the beginning but quickly found two main problems:

  • There were a lot of problems between the YUI event system and the way that JSTestDriver retrieves the results.
  • The way tests are executed is not extremely human-friendly. We use a browser to work (and several to test our code in), so it would be nice to use the same interface to run JS tests. This discarded other tools like PhantomJS and other smoke runners.

Therefore, we had to build our own framework to execute tests, keeping all of the requirements in mind. Our efforts were focused on keeping it as simple as possible, and we decided to use the YUI testing libraries (and Sinon as mocking framework) because they allowed us to maintain each test properly isolated as well as proper reporting using XML or JSON and human-readable formats at the same time.

The Approach

This consisted in solving the simpler use case first (execute one test in a browser) and then iterate from there. A very simple PHP framework was put together to go through the JS module directories identifying the modules with tests (thanks to a basic naming convention) and execute them. The results were shown in a human-readable way and in an XML in-a-box.
PHP is required because we want to use a lot of the tools that we have already implemented in PHP related to YUI (dependency map generation server side, etc.).

After that, we wrote a client using Selenium/Webdriver so that the computer-readable results could be easily gathered, combined, and shown in report.

At this point in the project:

  • Developers could use the browser to run the test just by going to a predictable URL,
  • CI could use a command-line program to execute them automatically. Indeed, the results are automatically supported by Jenkins so we only needed to properly configure the jobs, telling our test framework to put the results in the correct folder and store them in the Jenkins job.

That was enough to allow us to execute them hundreds of times a day.


Another important aspect of testing is knowing the coverage of your codebase. Due to the nature of our codebase and how we divide everything into modules that work together, each module may have dependencies with others, despite the fact that our tests are meant to be designed per-module.

Most of the dependencies are mocked with SinonJS, but there are also tests that don’t. Since a test in a module can exercise another, we can decide to compute coverage “as integration tests” or with a per-module approach.

The first alternative would instrument all of the code (with YUI Test) before running the tests, so if a module is exercised, it may be due to a test that is meant to exercise another module. An approach like that may encourage developers to create acceptance tests instead of unit tests.

Since we wanted to encourage developers to create unit tests for every module, the decision was finally taken to compute the coverage with a per-module approach. This way, we instrument only the code of the module being tested and ignore the others.

To make this happen, we moved the run operation from the PHP framework to a JS runner because that allowed for some of the required operations to get the coverage as instrument and de-instrument the code before and after running the tests.

Once the test is executed, the final test coverage report is generated, so it would be unnecessary to integrate with our current Jenkins infrastructure and setup all the jobs to generate the report. Coverage generation requires about 50% more time than regular test execution, but it’s absolutely worth it.

To be Done

There is more work that can be done at this point, such as improving the performance by executing tests in parallel or force a minimal coverage for each module, but we haven't done this yet because we think that the coverage is only a measure of untested code and it doesn't ensure the quality of the tests.

It is much more important to have a testing culture than just get the metrics of the code.

The Tuenti Release and Development Process: The Development Environment and Workflow

Posted on 7/22/2013 by Víctor García, DevOps Engineer

Blog post series: part 2
You can read the previous post here.

The Development Environment

We maintain a couple of development environments: a shared one and a private self-hosted one, and we encourage developers to move to the private in order to ease and speed up the development workflow, getting rid of constraints and slowness caused by shared resources.

Both infrastructures provide a full “dev” and “test” sub-environments:

  • The dev has its own fake user data to play with while coding.
  • The test environment is basically a full environment with an empty database to be filled in with test fixtures able to run any kind of tests.
  1. Unit and integration tests run in the same machine the code does, just using PHPUnit.
  2. Browser tests run in a Windows virtualbox machine we provide with Webdriver configured to run tests in Firefox, Chrome and Internet Explorer.
These two sub-environments are under two different Nginx virtual hosts, so a developer can use both at the same time.

The Shared Environment
The shared environment are virtual machines in a VMWare ESXi environment. Each development team is assigned to a single virtual machine. The management and creation of them is pretty straightforward as they are fully provisioned with Puppet. They provide full dev and testing environments with its own Nginx server, Memcache  and Beanstalk instances, a HipHop PHP interpreter, MySQL database, etc. All these resources will be shared among all the virtual machine users. The development database, HBase, Sphinx and Jabber servers are hosted in a separated machine, so, these particular resources are shared by all users.

That’s why we’re currently moving everyone to the private environment. We’ve had problems with shared resources, such as when someone executes a heavy script that raises the CPU usage to 100%, makes the memory swaps, or slows the machine down due very high IO usage, sometimes even affecting users sharing the same physical machine.

The Private Environment: Tuenti-in-a-Box
Tuenti-in-a-box is the official name for this private environment. It’s a VirtualBox machine managed with Vagrant that runs on the developer’s personal laptop. The combination of Vagrant and Puppet is a fantastic thing to set this up since it let’s us provision virtual machines with Puppet very easily, every developer can launch as many private virtual machines as he wants just with a simple command. Thus, Tuenti-in-a-box is an autonomous machine that provides the above-mentioned full dev and test environments, ridding us of problems with shared resources. Every resource is within its virtual machine with one exception, the development database that is still being shared among all users.

Right now, project exists that lets every Tuenti-in-a-box user have its own database with a small and obfuscated subset of production data.

The Code Workflow Until Integration

Developers work within their own branches. Each branch can be shared among other developers. These branches must be up-to-date, so developers frequently update them with the integration branch, which is always considered to be safe and with no broken tests (we will see how this is achieved in the next post).

Each team organizes its development in the manner they want, but to integrate the code, some steps must be followed:

  • A code review must be carried out by your teammates and it must pass the review:
  1. In order to keep track of what a branch contains and for the sake of organization, every piece of code must be tied to a Jira ticket.
  2. Doing this lets us use Fisheye and Crucible to create code reviews.
  • The code is good enough to be deployed to an alpha server.
  • The code can’t break any tests, Jenkins will be in charge of executing all of them.
  • A QA team member tests the branch and gives it the thumbs up.
  • After all of that, a pull request is done and then the branch starts the integration process (more details about this in the next post).
Jenkins Takes Part!
At this stage, testing is one of the most important things. The development branches need to be tested. Giving feedback to developers while they are coding is useful for making them aware of the state of their branches.

Those branches can be tested in three different ways:

  • Development Jenkins builds that only run a subset of tests: faster and more frequent.
  1. ~14500 tests in ~10 minutes
  • Nightly Jenkins builds, that run all tests: slower, but nightly, so developers can have all tests feedback the next day.
  1. ~26500 tests in ~60 minutes
  • Preintegration Jenkins builds, that run all tests: slower but it's the final testing before merging to the integration branch.
  1. ~26500 tests in ~60 minutes
The creation of Jenkins builds is automated by scripts that anyone can execute by command line. No manual intervention in the interface is necessary.

Our Jenkins infrastructure has a master server with 22 slave servers. Each slave parallelizes the tests execution in 6 environments, and in order to be faster, a “pipeline mode” takes 6 slaves and make them run together to executes all tests. You might be interested in what the Jenkins architecture is like and how it executes and parallelizes such a large amount of tests in such a short time, but we’ll leave that for another blog post.

You can keep reading the next post here.

The Tuenti Release and Development Process: Once Upon a Time

Posted on 7/08/2013 by Víctor García, DevOps Engineer

Blog post series: part 1
The release process and development workflow at Tuenti has evolved from a slow, manual and unreliable process to a fast, fully automated and stable one. It’s now evolving and will keep evolving forever.

Some years ago, there was almost no control over the process, and the existing process was manual.

  • Developers coded in branches, those branches were merged to the “release” branch with almost no testing by either automated or manual tests.
  • The testing was only done in that “release branch” by CruiseControl (and later by Hudson) and, at the same time, manually by the QA team.
  • Although there were dozens of broken tests, the release was deployed after someone manually ran those tests and determined that they were brittle tests.
  • Nobody really knew what was going to be pushed.
  • Huge release meetings took place the day before that wasted an hour of approximately 20 engineers’ time.
  • The code building lasted more than an hour and we were doing a bunch of them every release.
  • We didn't trust the process very much. We didn’t feel confident that everything was going to go smoothly, so the release was done very early in the morning to avoid disrupting our users and also because it could take several hours. We even had to wake up 4 hours earlier!
  • A regular release started at 8am and ended at 12pm and required the Engineering Team’s attention the entire time.
  • There were one or two releases per week at most.
  • During the release, there were many code builds and deployments to production because some bugs were found and fixed right there.
  • The bug tracking in the error log was a mess, the stacktraces were mixed, and tons probablu important of erros were ignored.

  • In the end, had a lot of bugs and fixing any one of them was an expensive task that involved a lot of people.

Obviously, this was not the way to go so everything has changed. A lot!!

Everything Has Changed!
Today, everything is automated, and not just the specific release process, but even the development workflow.
Before going into details, some data:

  • We perform about 15 releases per week plus small additional production deployments called hotfixes (urgent and small fixes that are deployed quickly and skip the full process).
  • All branches are tested by Jenkins with more than 25,000 tests run each time.
  • Many tests levels (unit, integration, acceptance, just JavaScript) and types (white/black box).
  • Very few brittle tests.
  • A full release (since the release candidate is chosen until the code is in production) can be done in half an hour.
  • Every integrated changeset is a potential release candidate (“evergreen trunk”).
  • Building the code takes about 20 minutes (taking into account that we also compile the HipHop binaries).
  • The deployment to hundreds of servers takes 1- 2 minutes.
  • Server resources are fully monitored.
  • Development environment is reliable and more similar to production.
  • Every server provisioned by Puppet.
  • We have three deployment phases to reduce risks: alpha, staging, and production.
  • Everything is orchestrated through Jira tickets.
  • No manual intervention at all.

Process overview
Before going further, and to ease understanding, let’s explain how the branches’ tree is configured in our DCVS (Mercurial).

  • There are two permanent branches (live and integration).
  • Every development branch will be eventually merged into integration.
  • From any of the integration changesets, a new release branch will be created.
  • Once the release is finished, the branch is merged into the live branch and then, live back into integration to keep it updated.

Now let’s briefly list what we’ll cover in following blog posts.

2nd post: The Development Environment and Workflow
We’ll talk about:

  • DCVS management
  • The development and testing environment
  • Tuenti-in-a-box: Vagrant+Puppet
  • Webdriver
  • Jenkins and alpha usage
  • Ticket management with Jira
  • Code reviews with Fisheye and Crucible

The development workflow must be as much frictionless as possible for the developer, and the integration process as transparent, fast and safe as possible.

And we’ve achieved that!!

3rd post: Development Branch Integration
We will talk about:

  • Flow pull request system and dashboards
  • Jira orchestration
  • QA management

To make an integrated changeset a potential release candidate to be deployed to live, we wanted to have an “evergreen integration branch”.

And we’ve achieved that!!

4th post: Release Time!
We will talk about:

  • Release branch selection
  • Flow release management
  • Alpha environment
  • Staging environment
  • Deploying to production
  • The deployment: TuentiDeployer
  • Build script
  • HipHop compilation

Our goal is the deployment of any piece of code that is ready, without time constraints or risks. More frequent releases that are smaller is widely known as “continuous delivery”.

And we’ve achieved that!!

5th post: Deploying Configuration
We will talk about:

  • Self-service
  • ConfigCop
  • Safe deployment assurance

The configuration changes are very frequent, and a bad one can create chaos. We didn’t want to waste much time performing and checking these changes, but somehow, we wanted to let everyone deploy their configurations to production on their own.

And we’ve achieved that!!

6h post: Conclusion and the Future
We will talk about:

  • The benefits of these changes.
  • Transparency for everyone in the company.
  • Some tips that demonstrate success.
  • And now what? Next steps and projects.

We want the ideal development workflow and environment.

And we’ll achieve that!

You can keep reading the next post here.


Follow us