Performance Optimizing a Massive Feed Import

The feeds module does a great job of importing an RSS feed or two into Drupal. But does it stand up when you have hundreds of feeds to preprocess, import and index? For DrupalSun, we developed a way to quickly import over seven hundred feeds, even though we're doing extensive processing on each one.

Some techniques we'll discuss:

• Doing background processing in node.js, instead of Drupal cron
  • Using promises for asynchronous network and database requests
  • Communicating efficiently between node.js and the feeds module
• Aggressive network caching
• Testing
  • Using curl_multi to test custom fetchers
  • Testing limited subsets of data safely