BcnDevCon 2011: Security, Mobile and our Release Process

Publicado el 30/11/2011 por Diego MuñozSenior Fronted Engineer

Two weeks ago we were platinum sponsors of the Barcelona Developers Conference 2011, spending three days assisting to developer and entrepeneur talks but also speaking on three different tracks.

The 17th I spoke about how Tuenti release workflow works, both from the development perspective and from a broader devops and testing one. I detailed some of the opensource tools we use, the reasons why and some bottlenecks or problems we’ve run into.

On the 18th our Backend Framework and Security Lead gave a talk about security: Common web attack vectors, how they work, how to prevent them and some specific countermeasures we apply.

And finally, on the 19th our Mobile Core Tech Lead gave an interesting talk about the evolution of Tuenti’s Mobile site, challenges you face when you build for such a broad and varied device segment and how we try to give the best experience on each of them.

More or less at the same time, Pedro Álvarez and Diana López from the Apps Platform team, gave a presentation at the Universidad de Alcalá de Henares, during the first national RITSI event on Agile Methodologies and Cloud Computing. Their talk: Tuenti, from idea to the web, was a great success.

FICOD 2011: Tuenti, the Mobile Web, and TU

Publicado el 28/11/2011 por Erik Schultink CTO

Last week, I had the chance to speak at FICOD - a fantastic event located right here in Madrid. I spoke about the history of Tuenti, the lessons we’ve learned, the opportunity we see in the Mobile/Web ecosystem, and our vision for TU to address that opportunity.

You can find the links to the video and slides below - but there are a few key messages about TU worth highlighting:

  • social networking is becoming less about connecting with people, and more about communicating with them.
  • TU takes this communication to the next level; keeping you always connected with your friends
  • TU’s combines content/experience (namely, Tuenti’s social content and apps) with an operator - and explore what what innovation a social web company can bring to what’s a largely undifferentiated product offering in the Telco market today

TU leverages social context in a couple unique ways:

  • the bill (and billing model) is social; through the web, you see who you’re calling and how much it costs; and with most plans, friends on TU are free.
  • voicemail arrives instantly to the web interface, where you can see the sender’s profile and play, delete, and reply.

Check out the video at FICOD TV.

Find the deck on Slideshare.

You can check out TU itself at:

I’d like to thank the organizers of FICOD for the invitation to speak - and our Communications and TU teams for helping me prepare the presentation.

Finally, the mind map FICOD made during the talk: FICOD mind map

Because SSH and "for" loop doesn't cut it

Ricardo Bartolomé Senior Systems Engineer

Publicado el 15/11/2011 por Here at Tuenti we were managing our servers in a old-fashioned way for a long time. We used "for" loops in order to replicate changes across different servers, but as we’re grew that became more complex over time and with new people joining the team. Fortunately, now we have an automated deployment system that deploys a basic image to a server based in its server role. Once the node is up we use Puppet for transforming the node to the desired state, ensuring specific versions of the packages are installed, configuration files are in place and services are running.

We have chosen Puppet because it's easy to understand, it's straightforward, it’s extensible and it has a very good community around it.

  • Easy to understand: Puppet doesn’t have obscure components that are hard to understand. For example, the communication between client and server is made using a REST API over a SSL channel using client certificates issued by the integrated Puppet CA.
  • Straightforward: Moving from manual configuration to configuration management system requires a change of mind about how you manage your platform. Nevertheless, you'll be able to deploy it and write your automated configurations in minutes. People in the team was able to configure new components and modify existing ones after a brief explanation.
  • Extensible: At the moment we just created custom facters, but we'll need to extend puppet with custom types for easily managing our queue system configurations in a more flexible way. Some people have started to share some custom types and modules in the Puppet Module Forge.
  • Community: I had the pleasure of attending last PuppetCamp Europe 2011 held in Amsterdam on April 28th. I have met and shared experiences with a lot of developers, people working in systems operations, and even the PuppetLabs engineers. In the open conferences, the strong technical background of the attendees was apparent and you could see how PuppetLabs engineers listened to us about which features we would like to have in Puppet or how to improve them.

When we configure the amount of PHP processes that we spawn in our frontends we base this decision on the amount of CPU power of the server. This has a direct relation with the amount of memory, given that more CPU means more concurrent requests and also more memory usage. For automating the amount of PHP children that should be spawned we use a custom facter. A custom facter is a piece of Ruby code that is executed to return a fact for a node. The default facts are hostname, ip address, netmask, amount of memory, processors, kernel version, etc, and they're very useful, but sometimes you need to extend it. In the example displayed below we create a custom fact called "tuenti_phpchildren" that will take decisions about the amount of PHP children spawned per server. I use an existing custom facter called "tuenti_memorysize" that returns the amount of memory installed in a server as an integer.

Facter.add("tuenti_phpchildren") do setcode do memtotal = Facter.value('tuenti_memorysize') if memtotal > 20000000 phpchildren = "300" elsif memtotal > 9000000 phpchildren = "200" elsif memtotal > 5000000 phpchildren = "100" else phpchildren = "50" end phpchildren end end

Then we use the custom facter inside a template, to ensure the file contains the proper configuration for the daemon.

max_children = <%= tuenti_phpchildren %>

Automating your whole platform using Puppet is not something you'll accomplish in three days (if you have a large fleet of servers with different roles as we have), but it's a really worth the effort. Some advantages we got from moving our configuration management to Puppet are:

  • No configuration drifts: Puppet is checking in the background that everything is fine and will revert to the desired state if something has changed (accidentally or on purpose).
  • Become more agile: We are a fast-moving company that requires changes every day and we can't continuously spend our time on repetitive tasks. Now we deploy changes to production within minutes having more control than we use to have in the past.
  • Save time: We automate once and we could be able to easily bootstrap hundreds of nodes in minutes.
  • Comply with industry standards: Some law enforcements like Spanish LOPD or PCI Compliance require a list of policies that must be met in all the servers from your company, like access controls, encryption ciphers used in SSL transactions, password management and more.
  • Document itself: Puppet won't excuse you from documenting your platform, but once someone reads a manifest they’ll already understand what that piece of code is doing. No more tedious explanations about what to do or what to execute for configuring a service.
  • Share knowledge: Now we're able to share our Puppet modules with other teams like QA or Dev Tools in order to align our development, testing, and production environments. Also, using different Puppet environments we can have different versions of a module in different stages like development, staging and production, while having a reasonable workflow: First you develop it, then you try it across some servers, and finally you deploy it to the whole platform.

Our current Puppet architecture is based in three Puppet Masters that bring configuration management to more than 1k nodes. Scaling this platform is as easy for us as adding extra nodes and adding them to the load balancing pool. Puppet also offers a reporting feature that can be used with the Puppet Dashboard in order to have a view at a glance of your platform. Given that all the reporting data is stored in a database you can integrate it with your internal tools or dashboard in order to detect nodes that are not applying configuration properly or are suffering unexpected changes.

To make the development of manifests easier, a project named Geppetto was born some time ago. Besides the tool itself, it offers a module for Eclipse that will make your life much easier. For hardcore people, you also have Vim syntax highlighting provided by PuppetLabs.

In summary, even if you only have one server you need to be able to reproduce its configuration. So if you have the opportunity, try some configuration management like Puppet and move out from sysadmin hell to operational bliss.

EmTech Spain

Publicado el 31/10/2011 por Erik Schultink CTO

Last week, I had the chance to speak at EmTech Spain as part of a panel discussion on “The Future of the Internet”. Our challenge as panelists was to envision what the web would look like in 10 years. I talked mostly about mobile and its implications, which, given the launch of Tu, our MVNO, have occupied a lot of my thinking during the past year.

The biggest change in the mobile web over the last 2-3 years has been the emergence of Apps, driven by the success of the iPhone and its application platform. The signature user experience that the iPhone introduced, with its mass consumer appeal, has earned Apple the market power to control not just the OS and its UX, but to control the App distribution platform as well. This is without precedent; Apple, a device manufacturer, is in a position to gate the individual apps for UX and functionality, as well as control the payments coming from users to download and transact within those apps. Other device manufacturers have followed suit - Android, Blackberry, with Microsoft/Nokia and Amazon soon to follow. This is the biggest shift we’ve seen producing software products shifted from building native applications to building web applications. Now we must build device applications - again native - but with different UX considerations (touch v keyboard, screen resolution, tablet v phone) on each. Cross-platform development is no longer just a technical challenge of trying to build the same product experience on Mac + Windows or on IE + Netscape/Firefox. It’s now a product and design challenge to reframe that experience within the unique bounds of each platform/device - as the signature UX of the device carries through into the product experience.

Consider our case at Tuenti. Tuenti today builds two HTML clients - and - and mobile clients on 4 platforms (iPhone/iOS, Android, Blackberry, J2ME). Already we have two distributions of the J2ME, built to target lower- and higher-end devices. It’s foreseeable a similar need will emerge for Android, and potentially as the tablet markets for iOS and Android grow, we’ll need to target those with variants targeted towards that specific form factor. That’s 6 different UXes currently (and more coming) built to let users do the same fundamental things: chat, view friend’s content, share photos, etc.

The above is without mentioning an entire class of platforms we don’t build for now, but I expect to be an increasingly relevant way for people to access the Web: video game consoles. Of course, these consoles are increasingly used for a lot more than video games: casual web browsing, video on-demand, etc. It’s something that Tuenti - and anyone planning to deliver products that people use multiple times per day - will need to address.

How this ecosystem evolves in the next few years will dictate what products are brought to market and what types of companies bring them to market (how can start-ups build 6 apps?). I expect some convergence, driven probably as much by the people building products (eg us) as the dynamics of the consumer markets. Apps are a major aspect of the user experience of the phone; consumers will be drawn to the platform that has the apps they want to use. And App makers, particularly start-ups, will need to make tough choices about which platforms they build for. The next generation of innovative apps won’t be available on every device platform. I would also guess that the most innovative apps will be built for the most-open platforms - with the fewest limits on UX and what device functionality the developer can access.

Finally, I want to thank the organizers (MIT Tech Review Spain, Club Malaga Valley) and sponsors of EmTech Spain for the invitation; Ariel Poler for moderating the panel; and the fellow panelists - Geoff Ralston (Co-Founder, Imagine K-12) and Othman Laraki (Director of Growth, Twitter).

On November 24, I’ll be speaking more about the evolution of the mobile web and how Tu fits into that at FICOD.

Summer Reading

Publicado el 05/8/2011 por Erik Schultink CTO

It’s that time of year again - your family/significant other is dragging you to the beach for a week. It’s a sunny, sandy place that won’t do your laptop any favors. How to make the best of it? My suggestion is to bring along some good books - and I’ve devoted this entry to listing some specific recommendations with relevance to our work.

Dreaming in Code, by Scott Rosenburg, chronicles the Chandler project - an effort led by Mitch Kapor to build a personal organization application that seems to do just about everything (which should tip you off to how that ends up working out). The account reveals a lot of the challenges faced by product dev teams - most of which I’m sure you’ll find familiar. If your team’s ever struggled in the face of shifting requirements and attempts to integrate new technologies, you’ll be happy to learn that you definitely are not alone. (link)

Finding Flow, by Mihaly Csikszentmihalyi, introduces the concept of flow - optimal engagement in the task at hand - and discusses how to structure your experiences to create it. I think you’ll find that most of us are attracted to engineering precisely because it lends itself to engaging us in this way. Most people find such engagement fulfilling (Csikszentmihalyi equates it with happiness and general life satisfaction) - but in an engineering context it’s also quite productive. How can you craft your time, work, and environment to lend themselves to periods of flow? (note - he has several newer books, which I have yet to read)

Talent is Overrated, by Geoff Colvin, debunks a common myth that exceptional performers are blessed with some type of exceptional innate talent for their discipline. It then advances the concept of “deliberate practice” as an alternative explanation - arguing that the capability to perform at an exceptional level is acquired only with lots of hard, structured work - ideally under the guidance of expert teachers. Such work is not fun, not easy, and requires intense concentration and personal drive to pursue. (while the first half of this book is quite interesting, the second diverges into some anecdotes about leadership at GE which I don’t think are very relevant to our field).

The Art of Scalability by Martin L. Abbot and Michael T. Fisher. This book is unbelievably relevant to actual engineering management - it is the closest thing I’ve ever found to guide for doing my job. While it’s focused more on the systems ops/scalability side of things and is occasionally very process heavy, it is a very useful book. (link)

The Innovator’s Dilemma, by Clayton M. Christensen. A classic work analyzing how some companies are able to innovate - and others aren’t. Prevailing wisdom in tech is usually that the bigger a company is, the worse it is at innovating. Christensen illustrates that the size of the company isn’t as important you might think - the type of technological change and the organizational history are also critical determinants of whether a company can successfully innovate and commercialize the technology. Big companies do some things very well; start-ups do other things very well; the question is whether your company is in the right position for what it's trying to do.

Switch, by Chip and Dan Heath. All this summer reading is only valuable if you can leverage this knowledge to change yourself/your team for the better. So the last book on this list is focused on that: motivating yourself or others to make a change, especially in the face of prevailing attitudes, habits, or organizational inertia.

After reading these books, I hope you’ll return from summer not only well-rested, but also with some new ideas on how to do your work better.