Search results
65 packages found
A specification compliant robots.txt parser with wildcard (*) matching support.
Parser for robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input
Middleware for Express to add a robots.txt
Generate a robots.txt for Astro
A TypeScript-based project for easy serving of static files with support for live reloading, compression, and typed requests.
- TypeScript
- static file serving
- live reload
- compression
- typed requests
- HTTP server
- SSL
- cors
- express middleware
- proxy
- sitemap
- feeds
- robots.txt
- compression (gzip, deflate, brotli)
A proper robots.txt parser and combinator that works with eulalie
Vite plugin for generating robots.txt for production and development builds
It uses the user-agents.org xml file for detecting bots.
Sitemap generator for TypeScript projects
The grunt task for html-snapshots
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Express middleware for generating your robots.txt
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Deney all robots express middleware
An Eleventy plugin to generate a robots.txt file for your static site