Environment aware robots.txt for Rails

README.md

Make /robots.txt aware of the Rails environment

You probably don't want Google crawling your development staging app. Here's how to fix that.

$ mv public/robots.txt config/robots.production.txt
$ cp config/robots.production.txt config/robots.development.txt

Now edit config/routes.rb to add a route for /robots.txt, and add the controller code.

home_controller.rb

def robots                                                                                                                                      
  robots = File.read(Rails.root.join("config", "robots.#{Rails.env}.txt"))
  render :text => robots, :layout => false, :content_type => "text/plain"
end

robots.development.rb

# (moved from public/robots.txt)
#
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-Agent: *
Disallow: /

robots.production.txt

# (moved from public/robots.txt)
#
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

routes.rb

get '/robots.txt' => 'home#robots'

If we want to disable the search engine crawlers to scrap the website, then there is a simple alternative:

<% if !Rails.env.production? %>
  <meta name="robots" content="noindex,nofollow">
<% end %>