Optimize Your Rails Fragment Cache By Lazily Loading your Objects

I was looking into the performance of our my portfolio page the other day which was supposedly cached.  Here's what I noticed within the controller.





And here was our view:

This being a page that show properties and spaces, we are attempting to cache the entire page based on some keys that get updated whenever any of the spaces or properties within the user's portfolio is updated.  But here's the problem, our queries are doing some eager loading from the database.  Even though we were fragment caching the entire page, each cached hit was still hitting the database, creating a bunch of active record objects and then never actually using them. Even on a cached hit, this is what I saw in the logs:

To fix this, we moved all of the query logic out of the controller show method and into a our query object where it is safely memoized

Now instead of accessing instance variables created within the controller show method, our view accesses helper methods lazily from within the cache fragment.

Our cached my portfolio page no longer needlessly loads dozens of active record objects.

Testing SSL under capybara

If your website runs under ssl than your integration tests should too.  This post outlines how to set up testing over https with capybara 2.1 and a few popular capybara drivers.  The majority of this info was borrowed from this capybara github issue.

In order to run your capybara test server under ssl, you will need to generate a self signed certificate.
You'll want to check the certificate and private key into your project.  I added them to my spec/support directory.

Now that the certificate is all set, start capybara with ssl enabled using your certificates.  Hint: If you are using rspec, the following snippets can be pasted into your spec_helper.rb.



Next tell capybara, to make requests using https:


Now you'll need to instruct your web driver to automatically accepted your self signed certificate even though you are running from localhost.



Finally, add the following monkey patch capybara to make http with ssl enabled:



I am not proud of the last step, and jnicklas indicated in this closed issue that this should not be needed, but not sure how else to get it done.  Plan on following up with him and updating the post.

Scaling Heavy Web Requests with Sidekiq and Pusher

Long Running Web Requests

At View The Space, many of our pages deliver real time data over extended time periods.  The database queries producing this data can take significant time to run and since the reporting is meant to be real time, caching is not an option.  Our application servers consist of a bunch of single threaded unicorn processes sitting on top of Heroku.  Tying up our web requests with long running heavy queries would be a bad idea as all other web requests would queue up causing bad performance for our entire app.

Push it in the Background with Sidekiq

Instead of performing long queries during the course of the rails web request, we push that heavy lifting to the background with Sidekiq.  Sidekiq is a multi-threaded background job queueing library built on top of redis and by default each Sidekiq process leverages 25 threads.  When a request is made to our building traffic dashboard, a simple html page is quickly rendered to the user followed by an ajax json request.  That second ajax json request simply pushes a critical job onto a sidekiq queue:


That first job pushes another critical query job for each query to be performed for the dashboard.  Since sidekiq is multi-threaded, each query should be performed in parallel.



Once the query is complete, the job pushes the result to the browser using Pusher and the pusher-gem. As a side note, Pusher is a service that makes using websockets easy and it supports all major browsers including Internet Explorer.

Pusher publishes the result to a unique user channel based on the user's database id and rails session id.  The query result should be some simple raw data within a hash or an array that can easily be converted into json.

Rendering the Data within the Browser

Now that the data has been pushed from the server side sidekiq job, we need some client side pusher javascript code listening for the results from the browser. The following coffee script initializes pusher with the unique channel we published to on the server side and listens for each event. Once we know the pusher subscription has succeeded, we submit the ajax form which sets the whole sidekiq to pusher process into motion.

Summary

Heavy lifting such as real time data reporting is not something you want to execute during your rails web requests.  Instead perform that expensive data crunching in the background with Sidekiq and push the results to your users' browsers with Pusher.

What's Missing from Heroku Add-ons

Today I got burned by my redis service for the last time.  I am not going to name names, but they are listed within the heroku add-ons.  Everybody needs a redis instance so I had to pick a new service to replace my existing, but there are currently 5 redis providers listed as add-ons.

I had no way of deciding which one to pick.   Yeah yeah, i could have done some research, but my existing redis instance was just flat out not working and parts of our app was down.  I was in a hurry.  In the end, like  the seasoned engineer I am, I picked the one with the prettiest logo.  My point is, why doesn't Heroku have some type of transparency within their add ons section?  It should look like the itunes store.  When choosing between heroku add-ons, I should be able to see how many other projects are using the add-on and some reviews would also be nice.   Can anyone think of a reason why it doesn't work this way today?

Setting the Rails Timezone to the User's Browser Timezone

Timezones confuse me.

At VTS we have lots of reports and charts that group data by the day. These reports should look different based on the user's configured timezone. I did not want to force all of our users to configure their timezone within our app when they already configured it within their operating system and hence their browser. The only problem... the browser does not pass the timezone to the server by default. You have to take care of it yourself.

The solution is pretty straight forward:

1. Set each users timezone as a cookie
2. Each request sets the Rails timezone to the value of that cookie

The above coffee script code uses jsTimezoneDetect to set the browsers timezone within a cookie that will live for the next 365 days. All that is left is setting the timezone during each request.


The above code would go into your ApplicationController. Simple around filter for each request that sets the timezone if it's present. Notice that it sets the Time.zone back to the original default timezone after each request and, yes, the Time.zone method is actually thread safe.

Slight Caveat

The timezone is set after each page request on document ready. That means the very first request will not have the cookie set for the server side around_filter and will not set the timezone. I have seen some people out there getting fancy with ajax redirects but this is probably a bit overkill for most use cases.  I start to have nightmares involving infinite redirects.

Since I extracted all of the above into a Rails Engine, you can ignore all of my rambling and just install the browser-timezone-rails gem.