Migrating a running service (Mollom) to AWS without service interruptions and reduce costs

Mollom is a high traffic service that used to require high maintenance by operations and engineering. It was running in a classic datacenter setup and the migration to AWS had many positive effects such as reduction of maintenance costs, hardware costs and improvement of engineering velocity by using best practices in distributed cloud systems. By embracing the DevOps culture we also reduce costs by making sure the operations team is only utilized for tasks that are not automated yet. Mollom strives towards the goal of having fully automated processes and failure recovery.

Mollom is a sophisticated anti-spam technology and it is popular in the Drupal-sphere. 61,925 Drupal sites mention they use Mollom and every day Mollom processes over 24 million content evaluations to determine whether they are more “spammy” or more “hammy”. Let’s not take Mollom offline during the migration as that would increase the amount of spam in many many Drupal sites.

We realized that, as the service grew, we needed to make it much more robust when it comes to handling failures. The operations team should not be tasked with restarting services but by building automation. Testing end-to-end, including repeatable infrastructure, is part of the solution.

Mollom was running on physical hardware and scaling hardware was a challenge. Mollom was having over 60 to 80 alerts, that required human interaction, per month. The service also grew every day, and the physical hardware we were using wasn’t as scalable as it needed to be. We’ll show you how we migrated without downtime and support this with background information on how to build a distributed system and how this maps to “the cloud”.

With 24 million content checks a day, a lot of logs are generated. We’ll show you how we handle that volume of logs and how we search and analyze them.

In the presentation we’ll show you in-depth information how Mollom is built. Eg.: Cassandra &  DynamoDB, MySQL & RDS, Glassfish & Java, AWS AutoScaling, RabbitMQ, Puppet, CloudWatch & Munin, Jenkins, StatsD, ...