Golygon

Stuck with Rails debugging? Here are some useful Commands that you should know and will help you in diagnose your system. First few commands are in general for any production environment. Last few ones are specific to ruby on rails. This is basic diagnose and my next post will elaborate more on the various tools you can use to fix things up.

• last – Checkout out who all last logged in on the machine. This helps a little to get an idea whether the issue was self triggered or due to someone’s mischief.
• who – Who all are logged in right now? Is someone else experimenting on the machine?
• pstree -ap -u <user> – Let us dive into more serious debugging. This command will give you various processes running under a user’s name. It shows the processes in a tree format so that you know which process spawned subsequent children processes. By default you will observe that there are x number of ruby processes spawned by init. This x is the configuration specified to nginx for how many processes to spawn in parallel. Here is the output that I caught on a stable production m/c:

pstree -ap -u ubuntu
redis-server,2725 redis.conf

ruby,12368
—{ruby},12370

ruby,12376
—{ruby},26025

searchd,5346 –pidfile –config /home/ubuntu/github/trips3m/config/production.sphinx.conf

sshd,27650
—bash,27651
——pstree,27804 -ap -u ubuntu

It is easy to note that the two processes of ruby were started by nginx and they will serve to the requests made by the client. They work as pool threads and nginx keeps them up even if there is no current request to be served.

• ps -ef – This command will give you long list of processes, their parent processes and the commands with which they were invoked. We can do a grep to find out the details about a particular process. Here is the output of the process we are interested in (pid: 12368 – as mentioned by pstree output).
  $ ps -ef | grep 12368
  ubuntu 12368 1 0 Apr05 ? 00:01:18 Rack: /home/ubuntu/github/trips3m
  ubuntu 27806 27651 0 11:04 pts/0 00:00:00 grep –color=auto 12368

  Usually, the parent process of this ruby child is 1 (init). The reason is that any process spawned by a demon (here nginx/passenger) will get its parent id as that of init, 1. This is to save the process by getting killed in case its parent process was killed. init lives in your system untill you reboot. Another observation is that is shows the time when it was spawned (here Apr 05). This is the date when you restarted your server!
  Finally the command with which the process was spawned: Rack. This command is provided by rails so that 3rd party servers or scripts can invoke rails and serve the query to browser. So this is a very generic command.

• top - Ok, so we now know about the processes running on our system and their utility. How do we debug if we are not getting a timely response or nothing is showing up in logs. Well, if for a web request you are seeing that browser is just waiting for data from your server, first check if any entry is getting populated in:
  • logs/production.log, or
  • /opt/nginx/logs/error.log

  The entries in these files are populated when the request gets completed by the web server (rails). If nothing is getting populated, it means that rails is still stuck while resolving the response for your request. There are high chances that your web request is still running on the server due to one of many possible issues. (may be in infinite loop, or is hanging somewhere, etc.). Check out this command (top) to know what exactly is happening on system now. This will give you leads on CPU usage, memory usage and which process is eating how much resources. See for following symptoms:

  • High CPU consumption – This means one of your process is in infinite loop or running some complex algorithms which you need to fix. Normally when a web request comes to your server, the ruby processes consumes 90% cpu for couple of seconds and then chills down. If this is eating high CPU for long, you need to improve your code. Check out the controller and other ruby functions you invoked while serving for this particular action.
  • High Memory consumption - Memory Leaks! You are doomed. Resort to one of the memory debugger tools or take a careful look at your code again.
• lsof – It talks about the files that are being handled by a running process. Not sure what you can make out of it, but at times it may be useful.
• /proc/$pid/ – Contains all the information about any running process. All the commands derive their data from this virtual filesystem. So hack it down if you are searching for any answers related to a process.

System information: I am using nginx + Passenger on Rails 3 on Ubuntu.