I have been deploying for may days; there were latest release directories in the production/staging environment. But my background jobs were failing unexpectedly. However, it was running fine in development environment.
I tailed the log file
$ tail log/sidekiq.log -f -n 400
/home/deployer/www/staging/releases/20160217062712/app/workers/notification_async.rb:40:in `perform' / # but there was no such directory ~/www/staging/releases $ ls 20160327044951 20160327050311 20160327050746 20160327051344 20160327052411
the `20160217062712` was outdated one.
I had the following configuration in my monit config file.
|Start program||‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiq –config /home/deployer/www/staging/current/config/sidekiq.yml –index 0 –pidfile /home/deployer/www/staging/shared/tmp/pids/sidekiq-0.pid –environment staging –logfile /home/deployer/www/staging/shared/log/sidekiq.log -d’ timeout 30 second(s)|
|Stop program||‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiqctl stop /home/deployer/www/staging/shared/tmp/pids/sidekiq-0.pid’ timeout 110 second(s)|
means; monit was pointed to false pid file, so monit was not able to stop the old running process of sidekiq.
therefore my background processes were sometimes working fine; because the newly created processes started with new code base might have processed the tasks.