Sidekiq : Production :running on older deployment codes

I have been deploying for may days; there were latest release directories in the production/staging environment. But my background jobs were failing unexpectedly. However, it was running fine in development environment.

I tailed the log file

$ tail log/sidekiq.log -f -n 400

and saw

/home/deployer/www/staging/releases/20160217062712/app/workers/notification_async.rb:40:in `perform'

# but there was no such directory
~/www/staging/releases $ ls
20160327044951 20160327050311 20160327050746 20160327051344 20160327052411

the `20160217062712` was outdated one.


I had the following configuration in my monit config file.

Start program ‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiq –config /home/deployer/www/staging/current/config/sidekiq.yml –index 0 –pidfile /home/deployer/www/staging/shared/tmp/pids/ –environment staging –logfile /home/deployer/www/staging/shared/log/sidekiq.log -d’ timeout 30 second(s)
Stop program ‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiqctl stop /home/deployer/www/staging/shared/tmp/pids/’ timeout 110 second(s)

means; monit was pointed to false pid file, so monit was not able to stop the old running process of sidekiq.

therefore my background processes were sometimes working fine; because the newly created processes started with new code base might have processed the tasks.