View The Space is Hiring

Wednesday, November 30, 2011

How not to delete your heroku production environment and all of your database backups in one command

So you're never going to guess what I did today. Ok, I'll give you a hint. It has something to do with the title of this post.

Wait.. before you go judging me, I want to explain how I deleted my entire heroku production environment and what I learned as a result of this slight mishap.

My intentions were good. I already had a production environment called myapp and I wanted to create an additional staging environment. I harnessed all of my creativity and came up with myapp-staging. To create the new environment, I executed:

heroku create myapp-staging

Whoops, that creates a bamboo app and I need cedar to match my production environment. No problem, I'll just delete that and create a new one.

heroku apps:destroy myapp-staging

! WARNING: Potentially Destructive Action
! This command will affect the app: myapp
! To proceed, type "myapp" or re-run this command with --confirm myapp
> myapp
Destroying myapp (including all add-ons)... done

Wait.. what did I just do? Ctrl-c... too late. My app == gone. I forgot the "--app" and heroku just ignored myapp-staging. Being the cowboy that I am, I just entered "myapp" without thinking.

Ok, on one hand.. that was really dumb. Heroku gave me an out with the confirm, but I still entered "myapp". On the other hand, my heroku app:destroy command was completely invalid and instead of throwing an error, the heroku client just silently went on. Probably less than ideal.

Either way, since we weren't yet officially launched and we did not lose any valuable data , making that mistake helped me to realize something extremely important. Heroku database backups go bye-bye when you delete your app. In hindsight, this seems obvious since heroku:pgbackups is just an add on and all add ons are deleted, but it's a little scary how in just one command, your entire app including your data and all backups are gone. Seems like that backup strategy isn't exactly full proof. At that moment, I decided it would be a good idea, to have regular dumps of our database exported out of heroku to Amazon S3.

I looked around and there were a few libraries and examples of people uploading their database dumps to S3, but I decided to go with the heroku_cloud_backups gem. I followed all of the instructions on the github readme and the whole process was extremely seamless. The only slight difference is that heroku now wants people to use the scheduler add-on instead of cron.

So lesson learned. If you are only using heroku pgbackups, you are one idiot away from losing all of your data.

1 comment:

  1. Ah yes, had the same exact problem. This is a major oversight. Someone could just heroku delete your app from your computer while you're out for coffee and that's it, you're finished.

    ReplyDelete