A typical day in the office

Hey there.
As a software developer you write code. Some days you'll write more of it and on some days you'll write less. Today was a day for me, where I wrote quite few LOC (lines of code). Today was a "debugging" day. You see, the last few weeks I hammered out feature after feature in a timeframe that would have rather been months instead. If you work in such a hurried pace, trying to reach the deadline and finish your features, you inevitably will introduce bugs to the code.

The problem

Today it was about sending an email where the attached images were corrupt: They couldn't be opened and had file sizes around a few bytes, while it should have been kilo- or megabytes.

The files were uploaded with Paperclip into the Rails 4 app. If you want to get the url from a file to link or display you can do it like this:

<%= image_tag(@my_model.avatar.url(:thumb)) %>  
// :thumb specifies the size of the attachment,
// leaving it will give you the original file 

Now this works great in a view, where you include your image. It did not work in my mailers.

Here's the original code, that did not work (don't copy it!):

attachments[@my_model.avatar_file_name] = "http://example.com/#{@my_model.avatar.url}"  

Here's a quote from the Rails docs:

Pass the file name and content and Action Mailer and the Mail gem will automatically guess the mime_type, set the encoding and create the attachment.

(Highlight done by me)

The thing is, I did not pass the content, only the url to the content. D'oh!

Funnily, everything worked in development and I had some tests, that made sure the attachment was present:

it "includes attachments" do  
  attached_files = mail.attachments.map(&:filename)
  expect(attached_files).to include(my_model.avatar_file_name)

But this test does not test the contents of the attachment. I test my mails locally in dev with Mailcatcher. This is a website, that displays the sent/received mails. My guess is, that the files/urls to the attached image were present on my local machine, and that's why the link to the attachment from Mailcatcher worked. But it did not work, once I sent the mails to real receivers.

The Debugging

Because it "worked for me" I decided to log into the staging server and reproduce the problem there.

  • I changed the mailer to send all mails to my address, so nobody would receive my test mails.
  • I edited the source code using vim directly on the staging server (don't do this at home!)
  • I used the rails console to trigger the email sending with: MyMailer.send_notification(my_model).deliver_later

This uses Sidekiq to send the emails via a background job.

Now I realised pretty quickly that I did not pass the content, but an url. So I changed the code to actually grab the file contents:

file = File.read(File.join(Rails.root, "public", @my_model.avatar.url))  
attachments[@my_model.avatar_file_name] = file  

This did not work.

I always got an error, that the file wasn't present. It took me a few, to see that Paperclip puts a digest at the end of the url, so it looks like this: /path/to/the/image.jpg?156396840

I googled around but couldn't find a way to make Paperclip omit this digest param. So I changed the code again:

file = File.read(File.join(Rails.root, "public", @my_model.avatar.url.split("?").first))  
attachments[@my_model.avatar_file_name] = file  

Now I manually omit the digest and the file was found.

I triggered the email send again.

I did not reveice a mail and became nervous. I wanted to check, that it worked! So I decided to ignore the Sidekiq and background jobs and delivered my email immediately:
MyMailer.send_notification(my_model).deliver_now # notice the deliver_now instead of deliver_later

I received an email! With images! Hooray!

Not so fast cowboy: A few seconds later, I received the sidekiq-generated email: No images. Still corrupt.


I spent the next ten minutes trying to figure out how in the name of Sidekiq could corrupt my images.

I sent test emails again, with special flags inside to make sure only the sidekiq-emails were faulty. It was confirmed.

The next thing to do if you're facing a dilemma like that? Obvious: Write a rant in your team's chat. Sure enough I did.

The solution

Yeah, what can I say. I wrote the real solution way up there. It worked. My emails confirmed it. The piece of the puzzle I was missing? I somehow forgot, that Sidekiq in production caches your code. It's not reloaded. YOU ARE ON A SERVER, STUPID!.

Alright. I just had to commit the code, push and deploy (which triggers a Sidekiq restart with the new code). And everything worked like a charm.

That was fun.

How was your day? Let me know on Twitter or via email.