Sidekiq : Production :running on older deployment codes

I have been deploying for may days; there were latest release directories in the production/staging environment. But my background jobs were failing unexpectedly. However, it was running fine in development environment.

I tailed the log file

$ tail log/sidekiq.log -f -n 400

and saw

/home/deployer/www/staging/releases/20160217062712/app/workers/notification_async.rb:40:in `perform'
/

# but there was no such directory
~/www/staging/releases $ ls
20160327044951 20160327050311 20160327050746 20160327051344 20160327052411

the `20160217062712` was outdated one.

Reason:

I had the following configuration in my monit config file.

Start program ‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiq –config /home/deployer/www/staging/current/config/sidekiq.yml –index 0 –pidfile /home/deployer/www/staging/shared/tmp/pids/sidekiq-0.pid –environment staging –logfile /home/deployer/www/staging/shared/log/sidekiq.log -d’ timeout 30 second(s)
Stop program ‘/bin/su – deployer -c cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiqctl stop /home/deployer/www/staging/shared/tmp/pids/sidekiq-0.pid’ timeout 110 second(s)

means; monit was pointed to false pid file, so monit was not able to stop the old running process of sidekiq.

therefore my background processes were sometimes working fine; because the newly created processes started with new code base might have processed the tasks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s