View The Space is Hiring

Tuesday, July 9, 2013

Scaling Heavy Web Requests with Sidekiq and Pusher

Long Running Web Requests

At View The Space, many of our pages deliver real time data over extended time periods.  The database queries producing this data can take significant time to run and since the reporting is meant to be real time, caching is not an option.  Our application servers consist of a bunch of single threaded unicorn processes sitting on top of Heroku.  Tying up our web requests with long running heavy queries would be a bad idea as all other web requests would queue up causing bad performance for our entire app.

Push it in the Background with Sidekiq

Instead of performing long queries during the course of the rails web request, we push that heavy lifting to the background with Sidekiq.  Sidekiq is a multi-threaded background job queueing library built on top of redis and by default each Sidekiq process leverages 25 threads.  When a request is made to our building traffic dashboard, a simple html page is quickly rendered to the user followed by an ajax json request.  That second ajax json request simply pushes a critical job onto a sidekiq queue:

That first job pushes another critical query job for each query to be performed for the dashboard.  Since sidekiq is multi-threaded, each query should be performed in parallel.

Once the query is complete, the job pushes the result to the browser using Pusher and the pusher-gem. As a side note, Pusher is a service that makes using websockets easy and it supports all major browsers including Internet Explorer.

Pusher publishes the result to a unique user channel based on the user's database id and rails session id.  The query result should be some simple raw data within a hash or an array that can easily be converted into json.

Rendering the Data within the Browser

Now that the data has been pushed from the server side sidekiq job, we need some client side pusher javascript code listening for the results from the browser. The following coffee script initializes pusher with the unique channel we published to on the server side and listens for each event. Once we know the pusher subscription has succeeded, we submit the ajax form which sets the whole sidekiq to pusher process into motion.


Heavy lifting such as real time data reporting is not something you want to execute during your rails web requests.  Instead perform that expensive data crunching in the background with Sidekiq and push the results to your users' browsers with Pusher.