Search results
117 packages found
robots.txt
exact matchrobots.txt middleware for express/connect to server up your robots.txt
A specification compliant robots.txt parser with wildcard (*) matching support.
Express middleware for generating your robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
This library is designed to help generate and maintain dynamic sitemap.xml and robots.txt for their SvelteKit apps.
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
`robots-linter` implements parser and matcher `robots.txt` which is specified at [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309).
Middleware for Express to add a robots.txt
Generate a robots.txt for Astro
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A metalsmith plugin for generating a robots.txt file
A proper robots.txt parser and combinator that works with eulalie
Adds a middleware that exposes a robots.txt file for search engines
Vite plugin for generating robots.txt for production and development builds