Install and Configure Monit web interface

Install Monit

Monit is easiest to install through apt-get:

$ sudo apt-get install monit

Once monit downloads, you can add programs and processes to the configuration file:

$ sudo nano /etc/monit/monitrc

Monit can be started up with a command that then keeps it running in the background

$ monit

Typing monit status displays monit’s details:

The Monit daemon 5.3.2 uptime: 1h 25m 

System 'myhost.mydomain.tld'
  status                            Running
  monitoring status                 Monitored
  load average                      [0.03] [0.14] [0.20]
  cpu                               3.5%us 5.9%sy 0.0%wa
  memory usage                      26100 kB [10.4%]
  swap usage                        0 kB [0.0%]
  data collected                    Thu, 30 Aug 2012 18:35:00

Configure Monit

Monit is very easy to use nearly out of the box. By default, it is set up to check that services are running every 2 minutes and stores its log file in “/var/log/monit.log”.

These settings can be altered at the beginning of the configuration file in the set daemon and set logfile lines respectively.

Web Service

Monit comes with it’s own web server running on port 2812. To configure the web interface, find and uncomment the section that begins with set httpd port 2812. Once the section is uncommented, write in your server’s IP or domain name as the address, allow anyone to connect, and then create a monit user and password

    set httpd port 2812
    use address 12.34.56.789  # only accept connection from localhost
    allow 0.0.0.0/0.0.0.0     # allow localhost to connect to the server and
    allow admin:monit         # require user 'admin' with password 'monit'

Once this is configured, monit should reload and reread the configuration file, and the web interface will be available:

$ monit reload

When you are done, try

http://12.34.56.789:2812

You will see a dialog box appear in the browser.

General Monit Scripts

Sidekiq

  # /etc/monit/conf-available/sidekiq
  check process sidekiq_thepact_staging0 with pidfile "/home/deployer/www/staging/shared/tmp/pids/sidekiq.pid"  
  start program = "/bin/su - deployer -c 'cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiq --config /home/deployer/www/staging/current/config/sidekiq.yml --index 0 -e staging -d'" with timeout 30 seconds  
  stop program  = "/bin/su - deployer -c 'cd /home/deployer/www/staging/current && /usr/local/rvm/bin/rvm default do bundle exec sidekiqctl stop /home/deployer/www/staging/shared/tmp/pids/sidekiq.pid'" with timeout 110 seconds  group thepact-sidekiq-0

PostGreSQL

 # /etc/monit/conf-available/pg
 check process postgres with pidfile /var/run/postgresql/9.5-main.pid
 group database
 start program = "/etc/init.d/postgresql start"
 stop program  = "/etc/init.d/postgresql stop"
 if failed unixsocket /var/run/postgresql/.s.PGSQL.5432 protocol pgsql
 then restart

Puma

 # /etc/monit/conf-available/puma
 check process puma with pidfile /var/www/myapp/production/shared/tmp/pids/puma.pid
 group www

 start program   = "/bin/su - john -c '~/.rvm/bin/rvm default do bundle exec puma -C /var/www/myapp/production/shared/puma.rb --daemon'"
 stop program    = "/bin/su - john -c 'pumactl stop -P /var/www/myapp/production/shared/tmp/pids/puma.pid'"
 restart program = "/bin/su - john -c 'pumactl restart -P /var/www/myapp/production/shared/tmp/pids/puma.pid'"

or

 # /etc/monit/conf-available/puma
 check process puma with pidfile /var/www/myapp/production/shared/tmp/pids/puma.pid
 group www
 start program   = "/bin/su - john -c '~/.rvm/bin/rvm default do bundle exec puma -C /var/www/myapp/production/shared/puma.rb --daemon'"
 stop program    = "/bin/su - john -c '~/.rvm/bin/rvm default do bundle exec pumactl -S /var/www/myapp/production/shared/tmp/pids/puma.state stop'"
 restart program = "/bin/su - john -c '~/.rvm/bin/rvm default do bundle exec pumactl -S /var/www/myapp/production/shared/tmp/pids/puma.state restart'"

Using the scripts

By default Monit does not load conf-scripts in conf-available. You have to copy the scripts to conf-enabled to have them executed by Monit. So, we are going to link those files that folder instead of copying/duplicating them. Its a good practice.

$ sudo ln /etc/monit/conf-available/puma /etc/monit/conf-enabled/puma

This will link the file there, same file will act as if it is there too.

Sources

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-monit

Advertisements

Issue tracking in production via Gitlab Ruby API

It’s so easy for you to track production issues/exceptions with Gitlab. It has super sexy issue board and Issue management UI along with RESTful API. Thankfully we need not write REST API, however Ruby API is sufficient. There is an awesome Ruby Gem written by `@NARKOZ` (https://github.com/NARKOZ/gitlab).

Configuration example:

gem 'gitlab'
# in config/initializers/gitlab_config.rb
Gitlab.configure do |config|
  config.endpoint       = 'https://example.net/api/v3' # API endpoint URL, default: ENV['GITLAB_API_ENDPOINT']
  config.private_token  = 'qEsq1pt6HJPaNciie3MG'       # user's private token or OAuth2 access token, default: ENV['GITLAB_API_PRIVATE_TOKEN']
  # Optional
  # config.user_agent   = 'Custom User Agent'          # user agent, default: 'Gitlab Ruby Gem [version]'
  # config.sudo         = 'user'                       # username for sudo mode, default: nil
end

You can find API for issues here

Example Library I wrote

This should handler the rest. Make sure you include this module in ApplicationController

# in controller/concerns/exception_handler.rb
module ExceptionHandler
  extend ActiveSupport::Concern
  class InvalidRequest < StandardError;  end

  included do
    unless Rails.application.config.consider_all_requests_local
      rescue_from InvalidRequest,
                  NoMethodError,
                  ActiveRecord::RecordInvalid,
                  NameError,
                  ArgumentError, with: :handle_unauthorized_requests

    end
  end

  private
  def handle_unauthorized_requests(error)
    create_issue_in_gitlab(error)
    render 'errors/unauthorized', status: 400, layout: 'static', locals: {error: error}
  end

  def create_issue_in_gitlab(error)
    title = "#{error.class} : #{error.message}"

    description = "## Details\n"
    description << "**DateTime**: #{DateTime.now}  \n"
    description << "**TimeZone**: #{Time.zone}  \n"
    description << "**Environment**: #{Rails.env}  \n"
    description << "**UserID**: #{current_user && current_user.id}  \n"
    description << "**Requested From**: #{request.ip}  \n"
    description << "**Request Referrer URL**: #{request.referrer}  \n"
    description << "**Requested URL**: #{request.url}  \n"

    description << "## Request\n"
    description << "```ruby\n"
    description << "parameters: #{request.filtered_parameters}\n"
    description << "```\n"

    description << "## Session Dump\n"
    description << "```ruby\n"
    description << "#{session_dump}\n"
    description << "```\n"

    description << "## Backtrace\n"
    description << "```ruby\n"
    description << "#{error.backtrace[0..25].join("\n")}\n"
    description << "```\n"

    # TODO Move this task to ActiveJob
    # project_id is integer you get from gitlab for your project
    #  I figured out using
    #    > Gitlab.projects  # in rails console after configuring the gem
    #    this gives array of projects, you will get ID for project
    Gitlab.create_issue(ENV['gitlab_issifix_project_id'], title,
                        {description: description, labels: 'System Generated, Bug, Production'})
  end

  def session_dump
    keys = ['_csrf_token', 'session_id', 'warden.user.user.key', 'warden.user.user.session']
    keys.map { |key| "#{key}: #{session[key]}" }.join("\n")
  end
end

 

Then you should be seeing issues created in your project.

Screenshot of an example Issue

Screenshot from 2016-09-04 22:44:56

Developer’s checklist: Production deployment

One has to be very careful for shipping new changes/features to the production server.

When feature is said to be done

Assumption: Proper development processes have been followed

  1. Smoke Test in local machine
  2. Production Database backups
    • Generate snapshots or database-dumps
    • Store the backups in safe places
    • In case of Heroku, you will be provided with backup-tools
  3. Codebase versioning in Github / Diffusion
    • Create tags with proper naming conventions followed
    • eg V 0.1.1 or V 1.0.0 pre
  4. Deployment to production
    • Deploy the changes to the server
    • Make sure you run rake db:migrate is any migration pending
  5. Do data migrations if required
    • Already existing Production data might not co-op with your new code changes
      • reasons: there might be some data-fields manipulations or
        • new cols might have been added and data need to be filled in; data to be gathered from existing fields
    • Document the migration code you wrote for data-migrations [remember I am not taking about schema migrations]
    • eg
TransactionHistory.recurring_rewards.each do |tx|
   charge = Stripe::Charge.retrieve(tx.charge_id)
   invoice = Stripe::Invoice.retrieve(charge.invoice)
   subscription_id = invoice.subscription
   metadata = tx.transaction_additional_detail.metadata
   tx.transaction_additional_detail.update metadata: metadata.deep_merge(
                                           {
                                               subscription_id: subscription_id,
                                               invoice_id: invoice.id
                                           })
end
  1. Smoke test in production

Rails : Heroku : Production ActionController :: InvalidAuthenticityToken

 

So I’m not sure if this is 100% causing it but I was able to replicate the error

To reproduce:
1) Open two browser windows of the same browser type (ie. 2 chrome windows)
2) Go to the login page in both windows
3) Login on one of the windows and then logout
4) Login on the other browser window and you’ll see the error

Logging out updates the csrf token but if the other login page isn’t refreshed it doesn’t get the updated token. Shouldn’t devise be handling this situation gracefully?

Continue reading

Heroku : PostGreSQL : Dump production / staging database to localhost

$ heroku pg:backups capture -a app_name

this will be echoed in your screen

curl -oUse Ctrl-C at any time to stop monitoring progress; the backup
will continue running. Use heroku pg:backups info to check progress.
Stop a running backup with heroku pg:backups cancel.
DATABASE ---backup---> b001
Backup completed

 

then you need to download the dump using curl

$ curl -o latest.dump `heroku pg:backups public-url -a app_name`

 

then create a database in PG and make the dump import to that database

$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U [username] -d [database name] latest.dump
Example
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U john -d new_dump_database_name latest.dump
$ pg_restore --verbose --clean --no-acl --no-owner -h localhost -U john -d production-dump prod

now config Rails’s Database.yml

development:
  <<: *default
  #  database: rails-app_development
  database: new_dump_database_name

 

Engineyard : Ruby On Rails : Setting environment variables at once

I was also using Heroku previously now I moved to Engineyard. This is how I get my ENvironemnt variables in Heroku I added gem figaro. This gem basically needs file application.yml in app/config directory. When Rails app is initialized, it gets executed and loads the key value pair set in YAML format into memory. In Heroku Figaro has option to set the content of application.yml. Continue reading