The Tuenti Release and Development Process: Conclusion and the Future

Publicado el 14/1/2014 por Víctor García, DevOps Engineer

Blog post series: part 6 (the very last one)
You can read the previous post here.


There is nothing I can say other than the fact that all of this has been a great success for the company. The new process and all of the changes we introduced have improved developer throughput, has speed up the development workflow, has nearly achieved continuous integration and delivery, and in the end, we work better. The site and mobile apps have fewer bugs and overall, we’re very proud of what we’ve done:

  • Self service dev environments: disposable, ready to use, less support required
  • All manual processes automated: less time, fewer errors, more accounting
  • Well known processes: used in every project, less support required
  • Automated pull request: ensure testing and “ever green trunk” what in turn improves continuous integration and people integrate more often.

Buzzwords on the Internet and in Talks: Continuous Delivery/Integration, Agility, Automation...

Most of you have probably heard talks or read articles on the Internet about continuous delivery, continuous integration and all of these cool agile methodologies where everything relies on automation.

For the skeptics, here at Tuenti you can see that sooner or later, they pay off.

Not Following Them? Why Not?

Some of you may think that in your company, things are working just fine the way they are and that you don’t need to change them. You may be right, but what you might not know is that things could be even better. Why not always be ambitious and go for the very best?

Maybe you can think that changing your development workflows and the developer culture would take a lot of effort. Ok, it may not be straightforward, but it’s worth to try.

I’m not going to explain why doing smaller but more frequent releases is better, or why doing things manually is not the way to go, or why continuous integration reduces the number of bugs because you probably are tired of hearing the same again and again and because you can easily find that information by googling it.

How to Change

In newer, smaller companies, changing processes is not as hard as it is in bigger companies with old workflows and old people that are used to working with prehistoric methodologies.

Here are some tips that might help you make such changes in these more complicated settings:

  • Sell your idea by demonstrating success stories from other companies, clearly stating why change is better.
  • Start implementing something that makes the developers’ lives a bit easier.
  • Make small changes (Minimum Valuable Product). Bigger changes involve many ruptures in the process and developers won’t like them.
  • If they like your first MVP, you won’t have problems getting an opportunity to continue with those changes.
  • Developers will depend on you, be nice to them. Always be helpful and understanding when it comes to their problems. You can even do polls to find out what’s annoying them the most or what it is they need.
  • Automate everything. It ALWAYS pays off.

Transparency for Everyone

When someone in your company (techies, product managers, CTO, CFO... whoever) needs information are unable to get it, you have a big problem.

If everything is automated, it should be easy to gather information and show it company-wide; the release state, the next code that will be released, what that code does, what bugfixes it will include, who the author is, its risks, how to test it, etc.

Create a dashboard that everyone can see (a good design will help), so you can make them aware of, for example, when a new feature is going to be released. This way, for example, the communications/press team can make an announcement or the marketing people can prepare campaigns in time. Otherwise, they will ask you personally and in most of the cases, this is a waste of either your or their time.

And Now What? Next Steps and Projects

We automate almost everything but there are tons of things left to do and tons of way to improve/optimize the current tasks and operations we’ve automated.

Here is a list of some of the things we would still like to do:

  • Open source flow: Many people have requested this from us, and we are willing to do so but at the moment there are some parts of the code that are far too much coupled with Tuenti-specific things. We would need to do some refactoring before open sourcing it.
  • Speed up the pull request system: By testing more than one pull request at a time. For example, if there are two pull requests, A and B, the idea would be to test A and B together and also A only, if A+B is successful, Flow integrates two instead of one, if A+B fails and A is successful, Flow integrates one, if all of them fail, there is nothing to integrate, but it’s demonstrated that this case is the least frequent because in most of the cases the pull requests are successful.
  • Code risk analysis: Analyze this risk in the pull requests system so we can cherry pick pieces of code according to their risk and better plan releases. For example, we could do low risk releases only on Fridays.
  • Improve the development environment by virtualizing in different ways: Tuenti-in-a-box is not always enough to provide a full development environment because of resources consumption that is too high or because it’s not ready for other kinds of environments like Android or iOS developers. We have been thinking about improving the virtualization system, and instead of using VirtualBox, we would like to try some lightweight virtualization providers like LXC. It’s worth researching because we think there is room for great improvements here.

The Test Advisor

Publicado el 02/9/2013 por Rafa de la Torre & Miguel Angel García, Test FW Engineers

The Test Advisor is a tool that lets you know what set of tests you should execute given a set of changes in your source code. It is automatic and does not require you to tag all of your test cases.

A Bit of Background

The story of CI has been a tough one here at Tuenti. At one point, our CI system had to handle a huge load of tests of different types for every branch. We are talking about something around 20K test cases, a 2.4h build time, and Jenkins acting as a bottleneck for the development process. From a developer perspective, we track a measure of their suffering, called FPTR (short of “from push to test results”). It peaked at 8h.

Therefore, something had to be done with the tests. There were multiple options: add more hardware, optimize the tests, optimize the platform, delete the tests... or a combination of these options. When considering solutions, you need to keep in mind that products evolve which could potentially produce more tests that worsen the problem. In the long run, the best solution will always somehow involve test case management and process control, but that’s another story for a future post...

After adding more hardware, implementing several performance improvements, and managing slow and unstable tests, we concluded that we had to improve the situation in a sustainable way.

One solution we thought of was to identify the relevant tests for the changes being tested and execute only those tests, instead of executing the full regression. The idea was not novel and it may sound pretty obvious, but implementing it is not at all.

In short, the idea is to identify the changes in the system, the pieces that depend upon those changes and then execute all of the related tests. Google implemented it on top of their build system. However, we don’t have that great of a build system in which dependencies and tests are explicitly specified, and our starting point was a rather monolithic system.

The first approach we took in that direction was to identify the components of the system and annotate the tests indicating what component they are aiming at testing. Thus the theory was: “If I modify component X, then run all the tests annotated with @group X”. It didn’t work well: the list of components is live and evolves with the system and requires maintenance, tests needed to be annotated manually, maintaining synchronicity required a lot of effort, and there was no obvious way to check the accuracy of the annotations.

A different approach is to gather coverage information and exploit it in order to relate changes in source files and tests covering those changes. Getting coverage globally is not easy with our setup. We still use a custom solution based on phpunit+xdebug. There are still some problems with that approach though, that mostly affect end-to-end and integration tests: it is hard to relate the source files with the files finally deployed to tests servers, partly due to the way our build script works. Yes, it is easier for unit tests, but they are not really a problem since they are really fast. Additionally, we did not want to restrict our solution strictly to php code.

What it Is

The Test Advisor is a service that gathers “pseudo-coverage” information to be used later on, essentially to determine what the relevant test cases for a given changeset are.

When it was proposed, some of the advantages that we envisaged were:

  • Reduced regression times in a sustainable way. It would no longer be limited by the size of the full regression.
  • Improved feedback cycles
  • No need for manual maintenance (as opposed to annotating tests)

Regarding feedback cycles, a couple of benefits we foresaw were related to some pretty common problems in CI setups: robustness and speed of the tests. A common complaint was about robustness and false positives: “My code doesn’t touch anything related to that test but it failed in Jenkins”. If we had the Test Advisor, who would suffer bad quality tests in the first place? The ones modifying code related to those tests and who want to get their changes into production. No more discussion about ownership. No more quick and dirty workarounds for the flakiness of the tests. The same applies to their speed. It would be in your best interest to develop and maintain high quality tests :)

How It Works

Most of our product stack works under debian linux. We decided to use the inotify kernel subsystem to track filesystem events. This way, we can get that pseudo-coverage at a file level, independent from the language and the test framework used, or even the test runner.

We developed it using python as the main language, which is a good language for developing a quick prototype and putting all the pieces together. We also used pynotify.

The Test Advisor is conceptually composed of three main parts:

  • A service for information storage
  • A service for collecting coverage information. It is notified by the test runner when a test starts and ends. It has the responsibility of setting up and retrieving information about the files being accessed to complete the test scenario.
  • A client to retrieve and exploit information from the TestAdvisor.

This is how it looks at a high level:

As you can see from the figure, the TestAdvisor can take advantage of any test type being executed (unit, integration or E2E browser tests). It is also independent from the language used to implement the tests and the SUT.

Problems Along the Way

So we were easily able to get the files covered by a particular test case, but those types files are not actual source files. They go through a rather complex build process in which files are not only moved around, but they are versioned, stripped, compressed, etc. and some of them are even generated dynamically to fulfill http requests in production. We tried to hook into the build process to get a mapping between source and deployed files but it turned out to be too complex. We finally decided to use the “development mode” of the build, which basically links files and directories.

Another problem was the file caching. The hiphop interpreter (among others) caches the php files being processed and decides whether a file has changed based on its access time, and inotify does not provide a way of monitoring stat operations on the files. We researched a few possibilities for getting around this:

  • Override system calls to stat and its variants by means of LD_PRELOAD, not an easy one to get fully working (and also made us feel dirty messing there).
  • Instrument the kernel with systemtap, kprobes or similar. A bit cleaner, but a mistake and your precious machine may freeze. We also got the feeling that we were trying to re-implement inotify.

Finally, we picked the KISS solution: just perform a touch on the files accessed between tests. This way, the regression executed to grab information will be slower, but we don’t care much about the duration of a nightly build, do we? :)

Work that Lies Ahead

Right now, the TestAdvisor is used as a developer tool. We’ve achieved great improvements in the CI by other means (most of them process-related changes). However, we are still eager to integrate the TestAdvisor into our development and release cycle within our CI setup.

Developers will probably use a pipeline for their branches, consisting of 3 distinct phases: Unit Testing , Advised Testing and (optionally) Pre-Integration. An “advised build” in green can be considered sufficiently trustable to qualify for integration into the mainline.

This approach has been applied to the core of our products, the server-side code, and the desktop and mobile browser clients. We expect it to also be applied in the CI of the mobile apps at some point.

JavaScript Continuous Integration

Publicado el 22/8/2013 por Miguel Ángel García, Test FW Engineer; Juan Ramírez, Software Engineer & Alberto Gragera, Software Engineer

Here at Tuenti, we believe that Continuous Integration is the way to go in order to release in a fast, reliable fashion, and we apply it to every single level of our stack. Talking about CI is the same as talking about testing, which is critical for the health of any serious project. The bigger the project is, the more important automatic testing becomes. There are three requirements for testing something automatically:

  • You should be able to run your tests in a single step.
  • Results must be collected in a way that is computer-understable (JSON, XML, YAML, it doesn’t matter which).
  • And of course, you need to have tests. Tests are written by programmers, and it is very important to give them an easy way to write them and a functioning environment in which to execute them.

This is how we tackled the problem.

Our Environment

We use the YUI as the main JS library in our frontend, so our JS-CI need to be YUI compliant. There are very few tools that allow a JS-CI, and we went for JSTestDriver at the beginning but quickly found two main problems:

  • There were a lot of problems between the YUI event system and the way that JSTestDriver retrieves the results.
  • The way tests are executed is not extremely human-friendly. We use a browser to work (and several to test our code in), so it would be nice to use the same interface to run JS tests. This discarded other tools like PhantomJS and other smoke runners.

Therefore, we had to build our own framework to execute tests, keeping all of the requirements in mind. Our efforts were focused on keeping it as simple as possible, and we decided to use the YUI testing libraries (and Sinon as mocking framework) because they allowed us to maintain each test properly isolated as well as proper reporting using XML or JSON and human-readable formats at the same time.

The Approach

This consisted in solving the simpler use case first (execute one test in a browser) and then iterate from there. A very simple PHP framework was put together to go through the JS module directories identifying the modules with tests (thanks to a basic naming convention) and execute them. The results were shown in a human-readable way and in an XML in-a-box.
PHP is required because we want to use a lot of the tools that we have already implemented in PHP related to YUI (dependency map generation server side, etc.).

After that, we wrote a client using Selenium/Webdriver so that the computer-readable results could be easily gathered, combined, and shown in report.

At this point in the project:

  • Developers could use the browser to run the test just by going to a predictable URL,
  • CI could use a command-line program to execute them automatically. Indeed, the results are automatically supported by Jenkins so we only needed to properly configure the jobs, telling our test framework to put the results in the correct folder and store them in the Jenkins job.

That was enough to allow us to execute them hundreds of times a day.


Another important aspect of testing is knowing the coverage of your codebase. Due to the nature of our codebase and how we divide everything into modules that work together, each module may have dependencies with others, despite the fact that our tests are meant to be designed per-module.

Most of the dependencies are mocked with SinonJS, but there are also tests that don’t. Since a test in a module can exercise another, we can decide to compute coverage “as integration tests” or with a per-module approach.

The first alternative would instrument all of the code (with YUI Test) before running the tests, so if a module is exercised, it may be due to a test that is meant to exercise another module. An approach like that may encourage developers to create acceptance tests instead of unit tests.

Since we wanted to encourage developers to create unit tests for every module, the decision was finally taken to compute the coverage with a per-module approach. This way, we instrument only the code of the module being tested and ignore the others.

To make this happen, we moved the run operation from the PHP framework to a JS runner because that allowed for some of the required operations to get the coverage as instrument and de-instrument the code before and after running the tests.

Once the test is executed, the final test coverage report is generated, so it would be unnecessary to integrate with our current Jenkins infrastructure and setup all the jobs to generate the report. Coverage generation requires about 50% more time than regular test execution, but it’s absolutely worth it.

To be Done

There is more work that can be done at this point, such as improving the performance by executing tests in parallel or force a minimal coverage for each module, but we haven't done this yet because we think that the coverage is only a measure of untested code and it doesn't ensure the quality of the tests.

It is much more important to have a testing culture than just get the metrics of the code.

The Tuenti Release and Development Process: Once Upon a Time

Publicado el 08/7/2013 por Víctor García, DevOps Engineer

Blog post series: part 1
The release process and development workflow at Tuenti has evolved from a slow, manual and unreliable process to a fast, fully automated and stable one. It’s now evolving and will keep evolving forever.

Some years ago, there was almost no control over the process, and the existing process was manual.

  • Developers coded in branches, those branches were merged to the “release” branch with almost no testing by either automated or manual tests.
  • The testing was only done in that “release branch” by CruiseControl (and later by Hudson) and, at the same time, manually by the QA team.
  • Although there were dozens of broken tests, the release was deployed after someone manually ran those tests and determined that they were brittle tests.
  • Nobody really knew what was going to be pushed.
  • Huge release meetings took place the day before that wasted an hour of approximately 20 engineers’ time.
  • The code building lasted more than an hour and we were doing a bunch of them every release.
  • We didn't trust the process very much. We didn’t feel confident that everything was going to go smoothly, so the release was done very early in the morning to avoid disrupting our users and also because it could take several hours. We even had to wake up 4 hours earlier!
  • A regular release started at 8am and ended at 12pm and required the Engineering Team’s attention the entire time.
  • There were one or two releases per week at most.
  • During the release, there were many code builds and deployments to production because some bugs were found and fixed right there.
  • The bug tracking in the error log was a mess, the stacktraces were mixed, and tons probablu important of erros were ignored.

  • In the end, had a lot of bugs and fixing any one of them was an expensive task that involved a lot of people.

Obviously, this was not the way to go so everything has changed. A lot!!

Everything Has Changed!
Today, everything is automated, and not just the specific release process, but even the development workflow.
Before going into details, some data:

  • We perform about 15 releases per week plus small additional production deployments called hotfixes (urgent and small fixes that are deployed quickly and skip the full process).
  • All branches are tested by Jenkins with more than 25,000 tests run each time.
  • Many tests levels (unit, integration, acceptance, just JavaScript) and types (white/black box).
  • Very few brittle tests.
  • A full release (since the release candidate is chosen until the code is in production) can be done in half an hour.
  • Every integrated changeset is a potential release candidate (“evergreen trunk”).
  • Building the code takes about 20 minutes (taking into account that we also compile the HipHop binaries).
  • The deployment to hundreds of servers takes 1- 2 minutes.
  • Server resources are fully monitored.
  • Development environment is reliable and more similar to production.
  • Every server provisioned by Puppet.
  • We have three deployment phases to reduce risks: alpha, staging, and production.
  • Everything is orchestrated through Jira tickets.
  • No manual intervention at all.

Process overview
Before going further, and to ease understanding, let’s explain how the branches’ tree is configured in our DCVS (Mercurial).

  • There are two permanent branches (live and integration).
  • Every development branch will be eventually merged into integration.
  • From any of the integration changesets, a new release branch will be created.
  • Once the release is finished, the branch is merged into the live branch and then, live back into integration to keep it updated.

Now let’s briefly list what we’ll cover in following blog posts.

2nd post: The Development Environment and Workflow
We’ll talk about:

  • DCVS management
  • The development and testing environment
  • Tuenti-in-a-box: Vagrant+Puppet
  • Webdriver
  • Jenkins and alpha usage
  • Ticket management with Jira
  • Code reviews with Fisheye and Crucible

The development workflow must be as much frictionless as possible for the developer, and the integration process as transparent, fast and safe as possible.

And we’ve achieved that!!

3rd post: Development Branch Integration
We will talk about:

  • Flow pull request system and dashboards
  • Jira orchestration
  • QA management

To make an integrated changeset a potential release candidate to be deployed to live, we wanted to have an “evergreen integration branch”.

And we’ve achieved that!!

4th post: Release Time!
We will talk about:

  • Release branch selection
  • Flow release management
  • Alpha environment
  • Staging environment
  • Deploying to production
  • The deployment: TuentiDeployer
  • Build script
  • HipHop compilation

Our goal is the deployment of any piece of code that is ready, without time constraints or risks. More frequent releases that are smaller is widely known as “continuous delivery”.

And we’ve achieved that!!

5th post: Deploying Configuration
We will talk about:

  • Self-service
  • ConfigCop
  • Safe deployment assurance

The configuration changes are very frequent, and a bad one can create chaos. We didn’t want to waste much time performing and checking these changes, but somehow, we wanted to let everyone deploy their configurations to production on their own.

And we’ve achieved that!!

6h post: Conclusion and the Future
We will talk about:

  • The benefits of these changes.
  • Transparency for everyone in the company.
  • Some tips that demonstrate success.
  • And now what? Next steps and projects.

We want the ideal development workflow and environment.

And we’ll achieve that!

You can keep reading the next post here.

Continuous Integration at Tuenti

Publicado el 13/8/2012 por Jose Mucientes, Software Test Engineer


Tuenti is a fast paced growing company and website. Our site is constantly growing with more and more users getting registered, new features being added and new issues being addressed to ensure a safe and steady growth of our system. It is therefore, within our company methodology to develop and release code as quick as possible with certain quality guarantees.
At the time of writing we have code releases on a tri-weekly basis, and we are aiming to be able to safely release code on a daily basis, minimizing the QA team efforts. In the next few lines I’ll describe what our test framework engineers have been working on to achieve the current system and what steps are being taken to get to the next level.

The Release Process

Typically the workflow for anyone to check code into Tuenti is the following:

So the cycle begins with a feature request and its implementation. That is a complex process on its own, but we will not get into that now. When a development team coding a feature starts the actual coding, the team will fork a branch from the last stable version of the code and he will include its project in our Continuous Integration system: Jenkins. Jenkins is an open source tool that among many other things, it can trigger code builds when code is checked into the repositories. During the development process Jenkins will give feedback to the developer so that he knows whether he broke the build or any of the tests.

Typically the developer will write its own tests to unit test the classes he is modifying or coding, some integration tests, and at last, the QA engineers assigned to the team will write some UI browser tests to ensure the feature works and feels right on the UI. Developers have test sandboxes at their environments, which can be very useful for them to quickly assert that their code is working, nonetheless they can only run a reduced set of tests, most likely the ones related to the feature being developed. This is where Jenkins comes into play.

Jenkins CI environment will automatically run for the developer the whole test suite on his project so that he can be sure that his feature is not breaking any already existing features nor causing any disruption on the site.
Once Jenkins gives the developer the thumbs up (represented by a blue icon), the developer can automatically merge his code into the integration branch. Each time a branch is merged, a build is triggered in the integration branch, so we ensure that what was already integrated by other developers, works with the code which has just been merged.

At some point the release operations team selects an error-free changeset which will be the one going live. The set of error-free changesets will be moved to a different branch, called release, so that developers can continue integrating code in the integration branch without affecting the code release. The set of changes in the release branch, is then manually tested by the QA team, and bugs are corrected as soon as they are reported. On the meantime automatic regressions keep happening to ensure that the fixes don’t introduce new bugs into the code.

Once the branch is stable and all the bugs are under control, the code will be rolled out to live in off-peak traffic hours to avoid disrupting our users. Our QA team will ensure that everything works as expected and report any bugs found which will be fixed. When the devops (Development Operations)  team considers that the site is stable the release is closed and the release branch is considered to be safe and stable. At that point the code is merged to the stable branch, thus stable branch will have the last version of the code in live.

If bugs are spotted after the release is closed, depending on how important and compromising they are, they will be fixed immediately and committed to the Hotfix repository, where the fix will be tested and later pushed into live outside the planned workflow. If the bugs were not urgent to fix, the fixed would be pushed to live in the next scheduled code release.

Continuous Integration Framework

Now that we gave you an insight to our release workflow, let’s take a look at how our test framework is implemented and what is it capable of.

Tuenti Test framework is always at continuous development. Is a pretty complex tool which at the same time provides a lot of value to the development teams. We constantly face new challenges to solve.  We keep developing features to help developers to get feedback as soon as possible and in the clearest way and we keep adding tweaks to the system to maximize speed and throughput.

Currently we have a test suite with over 10.000 tests. Among this tests some are slower than others, for example browser UI tests take an average of 16 seconds each, integration tests 2 seconds and unit tests just a few milliseconds. The number of tests is constantly growing, at a pace of around 400 tests per month, making the full regressions slower and slower. As if that wasn’t enough, the rapid growth of the development team skyrocketed the amount of code to be tested.
It currently takes our CI system around 110 minutes to run all those tests, and about 25 minutes when we use the pipeline strategy. For some it might be good enough, but there are certain scenarios in which we need very quick feedback to react ASAP, plus developers are always happy to get feedback as soon as possible.

How do we achieve all this? What challenges did we face and what challenges arose?

To achieve this we are using 21 fairly powerful machines (8 Cores @2.50GHz, with 12 and 24 GB of RAM). Each job is run a in a single machine, except for pipe-lined jobs for extra quick feedback. In each machine we execute tests in parallel in 6 isolated environments (as you can see in the figure below) which are distributed by a test queue. Each environment has its own DB with the required fixtures, and for the browser tests different VNC environments, as we had some problems with some tests failing when they lost browser focus by other browsers running tests in parallel. 

Optimizing the code of the browser tests managed to decrease the test regression time by about 20 minutes. We speed up builds by removing unconditional sleeps which aren’t strictly necessary, in some parts of the test framework.

On the issues side, we have a few non-deterministic tests. This tests produce a different outcome each time they are ran, that seems to not depend on code changes. If you have worked in test automation, you probably know what I’m talking about: those tests that seem to fail from time to time, making your build as failed for no apparent reason and at last, the test seems to pass when you run it again. Unstable tests are indeed a big issue. Some of the core benefits and goals of automation are lost when your suite has unstable tests: your suite is not self checking nor fully automatic (requires manual debugging) and the test results are not repeatable. Unstable tests waste a lot of test engineer's’ time in debugging to figure out what went wrong, plus it makes the suite look as flaky and unreliable, and you don’t want that happening. If your test regression becomes unreliable, developers will get angry and will always blame the framework when tests fail, no matter whether it was their fault or not.

To fight off this enemy, we took two strategies:
● Short-term strategy: we keep track of all the tests which have failed, and after the complete suite is finished, if the number of failed tests is below a given threshold, then the build process will automatically retry to run this tests in isolation without parallel execution. If the tests pass, we modify the logs and mark them as passed. We implemented this approach to filter false positives and save time analysing reports This approach is pretty effective and does a good job cleaning the reports but it has some drawbacks. The retry task adds some extra 15 minutes to the whole build. The strategy doesn’t cope with the problem root as the number of unstable tests will keep increasing and at last, some tests do fail even after being retried, requiring manual debugging.
● Long-term strategy: when a test fails and then passes is added into an unstable tests report produced by Jenkins after every build. After the reports are produced, the Quality Assurance or the DevOps Team  will temporarily remove the test from the regression suite and will examine the test to try to determine the root cause of non determinism when running the test. Once the test is fixed and stable is brought back to the suite. Thanks to this approach the number of unstable tests has drop by 80%, making our regressions reliable and completely automatic in most cases.

When it comes to providing feedback ASAP, we can’t parallelize limitlessly as we don’t have infinite machines and we need to find the optimal number to provide quick feedback minimizing the potential loss of total throughput of builds that might be affected as trade off if we use more than one machine per job execution. Also running tests in more environments, mean more setUp time (the built code shall be deployed in that environment, the data bases should be prepared for the environment, etc). So far we  have applied the following strategies:
● Early feedback: we provide results per test on real time as they are executed and we run the tests which failed during the last build first, so that developers can check ASAP whether their tests have been fixed.
● Execution of selected tests: developers can customize their regressions to customize which tests they want to run. They can choose to run only unit and integration or only browser tests. In early development stages we advise to run unit and integration, and at last acceptance upon merging to integration branch. Developers can also specify the tests to run by using phpunit @group tag annotation so they can have a fine grained selection of the tests they need.
● Build Pipeline: for special jobs in which is critical to get very quick feedback or we need reports of nearly all the changesets. For the pipeline we use 6 testing nodes. The pipeline is divided in 3 steps mainly: build and unit tests, main test regression and aggregating the test reports. We trigger 5 jobs to run the main regression  in parallel.  The pipeline produces results in about 25 -35 minutes compared to 110’ - 120 minutes  for normal builds. The drawback is that build pipelines require many machines and reduce the total build throughput per hour at peak times.
● On demand cloud resources: we used to use Amazon’s web services to get  testing resources in the pas as a proof of conceptt, but we decided to invest in our own infrastructure as it proved to be more convenient economy wise. Nowadays we are thinking of going back to on demand virtualization as we need to produce quicker feedback for full regression. Quicker feedback of complete regressions with a constantly growing amount of tests, can only be translated into an increment of machines in our testing farm. However, as  most builds are needed during peak hours, it would make sense to have on demand resources rather than acquiring new machines which would be under-used at off-peak hours.