Moving from Lambda+DynamoDB to SQS

This is a follow-up to my previous post, where I was trying to offload the email tracking capabilities from our web server. Even with the more bulletproof Lambda function, I was losing some events because evidently, SNS tries 4 times and then gives up. Lambda was automatically scaling, but my writes to a DynamoDB table were not. So back to the drawing board!

I next moved from Lambda + DynamoDB to SQS, which seems to handle anything thrown at it, so that's awesome! Then I wrote a cronjob script to pull SQS messages, process them, and then delete them from the queue.

So now I can send a batch of emails through SES (using SendBulkTemplatedEmail), 50 at a time, which are set to use a configuration set to send email events (delivery, bounce, complaint, click, open, etc) to a SNS topic that is set to add it to a SQS queue. Once that's done, all events go to the queue and then every hour, I poll the queue to gather all messages and process/delete the relevant ones. I was able to send 7k emails in a few minutes and the load on my server was unnoticeable!