Search results
121 packages found
Generate a robots.txt for a web project (uses the robots.txt from HTML5 Boilerplate). This can be run from the command line when Generate is installed globally, or used as a plugin in another generator.
- boilerplate
- build
- cli
- cli-app
- command-line
- create
- dev
- development
- framework
- front
- frontend
- generate
- generate-generator
- generate-plugin
- View more
Web Builder Module - Create robots.txt file
Generate a robots.txt file for your Astro site
A web crawler frontier implementation in TypeScript backed by Redis. MicroFrontier is a scalable and distributed frontier implemented through Redis Queues.
A middleware for the @neobeach/core to serve a robots.txt file.
Registers /sitemap.txt, /sitemap.xml and /robots.txt, can be added to Google Search Console.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Higher order Next.js configuration for generating sitemap.xml and robots.txt
Generate Robots.txt file with your custom configs
Basic robots.txt generator plugin for Hexo
Basic robots.txt generator plugin for Hexo with Host options
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Site mapper utilizing puppeteer to generate a xml sitemap
An autonomous webcrawler for indexing robots.txt files.
robots.txt generation for webpack 3
Ensures polite outgoing HTTP requests that respects robots.txt and aren't made too close to each other
Install robots.txt files on every site on a staging server, taking into account both PHP and node/stagecoach sites
Sitemap generator for TypeScript projects
A web crawler. Automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Kerouac robots exclusion components for Bixby.js.