grunt-robots-txt

Grunt plugin which generates a robots.txt file

npm install grunt-robots-txt
5 downloads in the last week
55 downloads in the last month

grunt-robots-txt

Grunt plugin for generating robots.txt.

NPM version Dependency Status Build Status

Installation

Install npm package:

npm install grunt-robots-txt

Add this line to your project's Gruntfile.js:

grunt.loadNpmTasks('grunt-robots-txt');

Configuration

robotstxt is a multitask, so you can use robotstxt:dist and register some other tasks than "dist":

    grunt.initConfig({

        robotstxt: {
            dist: {
                dest: 'src/',
                policy: [
                    {
                        ua: 'googlebot',
                        disallow: '/private/'
                    },
                    {
                        ua: 'googlebot-news',
                        disallow: ['/other/', '/another/'],
                        allow: '/new/'
                    },
                    {
                        ua: 'googlebot-third',
                        allow: ['/new1/', 'new2']
                    },
                    {
                        sitemap: ['http://example.com/sitemap.xml', 'http://alernate.org/sitemap.xml']
                    },
                    {
                        crawldelay: 100
                    },
                    {
                        host: 'www.example.org'
                    }
                ]
            }
        }

    });

Options

  • dest: String (./ by default)

    Path where robots.txt will be created

  • policy: Object (`` by default)

    Hash of your policy. Note that you can give an array of values for disallow in order to create multiple Disallow lines for the concerned User-agent. Same is possible for Allow and Sitemap lines.

    There can be only 1 Crawl-delay option, and only 1 Host option. If you supply more, only th last one will be added to the robots.txt file.

    As Sitemap, Crawl-delay and Host are non-standard, if you don't supply any of these in your config, your robots.txt will not include these options by default.

Build from sources

There is a build process (which involves Grunt) for this Grunt task. Basically, the process will check the code with JSHint and JSCS, and execute some tests.

You can launch this process with the grunt command:

grunt

Or if you just want to test (and not check code):

grunt test

devDependency Status Built with Grunt

License

This plugin for Grunt is released under the MIT License.

npm loves you