HipHop and the HHVM at the 2nd PHPMad Meetup at Tuenti

Posted on 11/21/2013 by Eng Outreach Committee

Yesterday, we had the people from PHPMad user group back in our Madrid offices to hold the 2nd monthly meetup. This time, the speaker was our backend engineer Daniel Pañeda, and he talked about his experience migrating to HipHop, the HipHop virtual machine, and the general state of the project. You can read the epic tale of the migration process at Tuenti in this post. The talk wasn’t recorded this time, but you can see the slides here and if you are interested, you can also browse our patches to HipHop in our Github account.

You can find upcoming events related to this group on their meetup page.

The Tuenti Release and Development Process: Release Time!

Posted on 9/16/2013 by Víctor García, DevOps Engineer

Blog post series: part 4
You can read the previous post here.

Release Candidate Selection

So, we have made sure that the integration branch is good enough and every changeset is a potential release candidate. Therefore, the release branch selection is trivial, just pick the latest changeset.

The release manager is in charge of this. How does s/he do it? Using Flow and Jira.

Like the pull request system we talked about in the previous post, Jira orchestrates the release workflow, and Flow takes care of the logic and operations. So:

  1. The release manager creates a Jira “release” type ticket.
  2. To start the release, just transition the ticket to “Start”

    1. When this operation begins, Jira notifies Flow and the release start process begins.
  • This is what Flow does in the background:

    1. It creates a new branch from the latest integration branch changeset.
    2. Analyzes the release contents and gather the involved tickets, to link them to the release ticket (Jira supports tickets linking)
    3. It configures Jenkins to test the new branch.
    4. It adds everyone involved in the release as watchers (a Jira watcher will be notified in every ticket change) in the release ticket, so that all of them are aware of anything related.
    5. Sends an email notification with the release content to everyone in the company’s tech side.
    6. This process takes ~1 minute.

Building, Compiling and Testing the Release Branch

Once the release branch is selected, it is time to start testing it because there might be a last minute bug that escaped all of the previous tests and eluded our awesome QA team’s eyes.

Flow detects new commits done in release (this commits almost never occur) and builds, compiles and updates an alpha server dedicated for the release branch.

Build

Why do we build the code? PHP is an interpreted language that doesn’t need to be built! Yes, it’s true, but we need to build other things:

  1. JavaScript and HTML code minimization
  2. Fetch libraries
  3. Static files versioning
  4. Generate translation binaries
  5. Build YUI libraries
  6. etc.

Compilation

We also use HipHop, so we do have to compile PHP code.
The HipHop compilation for a huge code base like ours is a quite heavy operation. We get the full code compiled in about 5 minutes using a farm built with 6 servers. This farm is dedicated just to this purpose and the full compilation only takes about 5 - 6 minutes.

Testing

The code built and compiled is deployed to an alpha server for the release branch, and QA tests the code there. The testing is fast and not extensive. It’s basically a sanity test over the main features since Jenkins and the previous testings assure its quality. Furthermore, the error log is checked just in case anything fails silently but leaves an error trace.
This testing phase usually takes a few minutes and bugs are rarely found.
Furthermore, Jenkins also runs all of the automated tests, so we have even more assurance that no tests have been broken.

Staging Phase, the Users Test for Us

Staging is the last step step before the final production deployment. It consists of a handful of dedicated servers where the release branch code is deployed and thousands of real users transparently “test” it. We just need to keep an eye on the error log, the performance stats, and the servers monitors to see if any issue arises.

This step is quite important. New bugs are almost never found here, but the ones that are found are very hard to detect, so anything found here is is more than welcome, especially because those bugs are usually caused by a big amount of users browsing the site, a case that we can’t easily reproduce in an alpha or development environment.

Updating Website!

We are now sure that the release branch code is correct and bugs free. We are ready to deploy the release code to hundreds of frontends servers. The same built code we used for deploying to that alpha for the release branch will be used for production.

The Deployment: TuentiDeployer

The deployment is performed with a tool called TuentiDeployer. Every time we’ve mentioned a “deploy” within these blog posts, that deploy was done using this tool.

It is used across the entire company and for any type of deployment for any service or to any server uses it. It’s basically a smart wrapper over Rsync and WebDav that parallelizes and supports multiple and flexible configurations letting you deploy almost anything you want wherever you want.

The production deployment, of course, is also done with TuentiDeployer, and pushing code to hundreds of servers only takes 1 - 2 minutes (mostly depending on the network latency).

It performs different types of deployments:

  1. PHP code to some servers that does not support HipHop yet.
  2. Static files to the static servers.
  3. An alpha deployment to keep at least one alpha server with live code.
  4. HipHop code to most of the servers:

    1. Not  fully true, we can’t push a huge binary file to hundreds of servers.
    2. Instead, it only deploys a text file with the new HipHop binary version.
    3. The frontend servers have a daemon that detect this file has changed.
    4. If it changed, all servers get the binary file from the artifact server.

      • This file is there as a previous step, pushed after its compilation.
    5. Obviously, hundreds of servers getting a big file from an artifact server will shut it down, so there are a bunch of cache artifacts servers to fetch from and relieve the master one.

Finishing the Release

Release done! It can be safely merged into the live branch so that every developer can get it to work with the latest code. This is when Jira and Flow takes part again. The Jira ticket triggers all the automated process with just the click of a button.

The Jira release ticket is transitioned to the “Deployed” status and a process in Flow starts. This process:

  1. Merges the release branch to the live branch.
  2. Closes the release branch in Mercurial.
  3. Disables the Jenkins job that tested the release branch to avoid wasting resources.
  4. Notifies everyone by email that the release is already in production.
  5. Updates the Flow dashboards as the release is over.

Oh No!! We Have to Revert the Release!!

After deploying the release to production, we might detect there is something important really broken and we have no other choice but to revert the release because that feature cannot be disabled by config and the fix seems to be complex and won’t be ready in the short term.

No problem!! We store the code built compression of the last releases, so we just need to decompress it and do the production deployment.
The revert process takes only about 4 minutes and almost never takes place.

You can keep reading the next post here.

The Epic Tale of Moving to HipHop

Posted on 3/05/2013 by By: Daniel Paneda (Senior Engineer) and Jaime Medrano (Principal Engineer)

This article will not explain what HipHop itself is (if you are not familiar with this software, you can read about it here). What we’d like to share are the main changes we had to make within our codebase to get things working in HipHop and evaluate if –performance wise– it was worth it.

TL;DR: it was very much worth it. Performance improvements have been significant. It was not easy to do the migration, and it took us about a year to make our full code base HipHop compliant. We had to make many changes on our side so about a million of lines of code would be compiled on HipHop but that wasn’t it- we also needed to make several changes to HipHop itself...but back to our epic tale!

While the driving motivator to move to HipHop was performance (we expected a much lower usage of CPU and memory and somewhat lower latencies), there were also added benefits to having our code base compiled on HipHop. An important one is that HipHop can be used to do static analysis.

 

Changes to PHP code

If your codebase is on the bigger side, be prepared to invest a lot of effort in getting something that can run on HipHop. The more extensions you use, the greater the chances are that you’ll have to invest in some major refactoring. Unit and integration tests are your friends. Our browser tests were almost useless in helping us find issues mostly because, when compared to unit and integration tests, they had a stronger non-deterministic behaviour. However, without unit tests, our refactoring efforts would have taken a lot longer.

The most important changes we had to make to the code were the following:

  • Get rid of all PHP > 5.3 functionality
  • Remove all uses of eval and dynamic defines.
  • A lot of work regarding having files physically on disk. When possible, we tried to remove instances of those files from PHP code. When that wasn’t an option, we had to get creative. Since HipHop keeps an internal cache of files in order to resolve PHP requires, we hooked up to this functionality to resolve our own files that, in some cases, were not PHP files.
  • Moved configuration from PHP code to json files.
  • Refactor some non-supported features:

    • Removed any uses of ArrayObject since it is not implemented on HipHop (ArrayObject is not your friend, don’t use it).
    • Remove ftp_* functions using curl extension.
    • The only stream wrappers supported are http, zlib, and some php ones (memory, temp, input, output and stderr).

HipHop improvements

Sometimes, refactoring the PHP code to make it HipHop compliant  was not an option or even the best option. In some cases, it was very difficult to remove the dependency on some PHP extensions we were using. In others, we discovered a difference in behaviour between PHP and HipHop that we needed to fix. The code quality of HipHop is pretty good, so fixing things is not a big problem. The problem is finding the exact behaviour difference that exists between PHP and HipHop that produces any given bug.  Again, unit tests to the rescue here.

The most important changes that we had to make to HipHop code were the following:

  • Migration to libevent2 (the stock version is quite old and has some memory leaks)
  • Create a proper build script on top of CMake for debian packages
  • Many, many modifications to match PHP behavior.
  • Some stability fixes in HipHop code regarding non-thread safe PHP libraries we were using. For example, we moved the locale implementation from libc  to boost (thread-safe implementation)
  • Addition of new extensions:

    • MemcachePool fully compatible with PHP one (also involves large changes on libmemcache to fully support UDP).
    • Configuration handling
    • Filter extension
    • Geoip extension
    • GMagick extension
    • Some internal extensions
  • Summary of total changes, excluding autogenerated files
666 files changed, 61990 insertions(+), 35065 deletions(-)
So, in the end, there are major differences between the vanilla HipHop and the one we are using. Of the changes we made, the ones we thought everyone could use are now pull requests to HipHop.
 

Environment changes

Moving over to HipHop wasn’t all about code changes. There were big changes needed in our development environment.  For example, we run HipHop on interpreted mode in development and we only fully compile in later testing phases. Changes to the development environment and the code affected a lot of teams- from devops, to backend engineers to QA. Our main concern was to make the migration as transparent as possible when it came to the development environment. 

Main changes made in this area were the following:

  • Moving development machines to use HipHop instead of php-fpm.
  • Migrate the base debian distro we are using on live and in development.
  • Create a compilation farm with distcc to be able to compile our source code faster.
  • Create new deployment tools to distribute and run HipHop binary balls.

Wrap Up

Moving to HipHop took a huge effort. In our case, it took two people working full time for over a year to complete the migration. On the way, not only did we improve the quality of our codebase but also our tools for building and deployment.

While it was a very long development effort, the reward was excellent to say the least. Since we migrated our fleet to HipHop, we’ve dropped our response times (measured from server side) a whooping 45%.  We use 2-3x less CPU and 10x less memory.

Follow us