Run a shell command in a separate thread, terminate it after a time limit, return its output

timeout.rb

# Runs a specified shell command in a separate thread.
# If it exceeds the given timeout in seconds, kills it.
# Returns any output produced by the command (stdout or stderr) as a String.
# Uses Kernel.select to wait up to the tick length (in seconds) between 
# checks on the command's status
#
# If you've got a cleaner way of doing this, I'd be interested to see it.
# If you think you can do it with Ruby's Timeout module, think again.
def run_with_timeout(command, timeout, tick)
  output = ''
  begin
    # Start task in another thread, which spawns a process
    stdin, stderrout, thread = Open3.popen2e(command)
    # Get the pid of the spawned process
    pid = thread[:pid]
    start = Time.now

    while (Time.now - start) < timeout and thread.alive?
      # Wait up to `tick` seconds for output/error data
      Kernel.select([stderrout], nil, nil, tick)
      # Try to read the data
      begin
        output << stderrout.read_nonblock(BUFFER_SIZE)
      rescue IO::WaitReadable
        # A read would block, so loop around for another select
      rescue EOFError
        # Command has completed, not really an error...
        break
      end
    end
    # Give Ruby time to clean up the other thread
    sleep 1

    if thread.alive?
      # We need to kill the process, because killing the thread leaves
      # the process alive but detached, annoyingly enough.
      Process.kill("TERM", pid)
    end
  ensure
    stdin.close if stdin
    stderrout.close if stderrout
  end
  return output
end

Almost any value for BUFFER_SIZE > 1000 should be fine, since if there is data to be read in excess of your BUFFER_SIZE, you read a chunk of it then immediately loop back and read more data, so it doesn't seem like this will significantly affect the time spent vis-a-vis the timeout. But still pick a value that will work best for the external application you're calling. If you just want a number, try 4096 (4KiB).

P.S. lpar, thanks for this!

You can prepend something like "timeout #{timeout}" to the command itself.

The below actually hangs forever (ruby 2.2.3 on linux):

Timeout::timeout(5) {
  stdout, exit_status = Open3.capture2e("/usr/bin/time sleep 1000")  
}

So I conclude Timeout is not an option. That means to avoid process leak, one has to do process kill after a timeout.

I find Process.kill(pid) unreliable though because it would easily miss sub processes. In the above example there are two processes - the time process and the sleep process. The latter is a sub-process. What I found reasonably well working on linux is launch process with :pgroup => true then doing Process.kill(-pid) e.g. sending signal to process group. That kills all procs (unless they spawned children with new group.
My issue is that I'm not sure that works on windows. I see :new_pgroup option in there but no time to spawn a windows machine somewhere and test kill(-pid). I'm not even sure open3 works there..

update: FYI ended up going with Process.spawn to have access to all options. kill -pid if good on linux. For windows as far as I'm reading best approach is to use sys-proctable to traverse process tree and kill individually. Have not tried that yet though.

@lpar do you see anything wrong with this code? It doesn't use Kernel.select but I think it's just as solid.

Just want to be sure I'm not missing anything by not using Kernel.select.

Note:

• requires ActiveSupport for #suppress method
• requires MAX_CMD_OUTPUT_LENGTH constant #remembermemory
• requires ALLOWED_CMDS whitelist #security

  # timeout is in seconds
  def with_timeout(cmd, *cmd_args, timeout)
    raise "Invalid command!" unless ALLOWED_CMDS.include?(cmd)

    # popen2e combines stdout and stderr into one IO object
    i, oe, t = Open3.popen2e(cmd, *cmd_args) # stdin, stdout+stderr, thread
    t[:timed_out] = false
    i.close

    # Purposefully NOT using Timeout.rb because in general it is a dangerous API!
    # http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html
    Thread.new do
      sleep timeout
      if t.alive?
        suppress(Errno::ESRCH) do # 'No such process' (possible race condition)
          # NOTE: we are assuming the command will create ONE process (not handling subprocs / proc groups)
          Process.kill('TERM', t.pid)
          t[:timed_out] = true
        end
      end
    end

    t.value # wait for process to finish, one way or the other
    out = oe.read(MAX_CMD_OUTPUT_LENGTH).strip
    oe.close

    if t[:timed_out]
      out << "\n" unless out.blank?
      out << "*** Process failed to complete after #{timeout} seconds ***"
    end

    out
  end

To anyone stumbling upon this: I wrote a tried and tested gem for this purpose (and others!): https://rubygems.org/gems/command_runner_ng or https://github.com/kamstrup/command_runner_ng. The API is simple yet powerful and there are no threads involved.

@kamstrup I wrote one too: https://github.com/thewoolleyman/process_helper

Who hasn't really 😂? (although I think mine has a few more features than yours 😉).

And yet Ruby still doesn't support this in the standard API...