Search results

117 packages found

robots.txt

exact match

robots.txt middleware for express/connect to server up your robots.txt

published 1.1.0 9 years ago
M
Q
P

A specification compliant robots.txt parser with wildcard (*) matching support.

published 3.0.1 a year ago
M
Q
P

Express middleware for generating your robots.txt

published 1.0.0 4 years ago
M
Q
P

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

published 3.2.10 10 months ago
M
Q
P

This library is designed to help generate and maintain dynamic sitemap.xml and robots.txt for their SvelteKit apps.

published 1.0.21 a year ago
M
Q
P

Validate urls against robots.txt rules.

published 1.0.2 3 years ago
M
Q
P

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

published 2.0.3 2 years ago
M
Q
P

Streaming parser for robots.txt files

published 2.0.1 3 years ago
M
Q
P

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

published 3.2.9 2 years ago
M
Q
P

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

published 1.1.5 2 months ago
M
Q
P

`robots-linter` implements parser and matcher `robots.txt` which is specified at [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309).

published 0.6.1 3 months ago
M
Q
P

Middleware for Express to add a robots.txt

published 1.0.1 8 years ago
M
Q
P

Generate a robots.txt for Astro

published 1.0.0 7 months ago
M
Q
P

A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, robots.txt, or arbitrary input

published 3.6.1 2 days ago
M
Q
P

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

published 3.2.8 4 months ago
M
Q
P

A metalsmith plugin for generating a robots.txt file

published 1.1.0 6 years ago
M
Q
P

A proper robots.txt parser and combinator that works with eulalie

published 1.1.0 8 years ago
M
Q
P

Adds a middleware that exposes a robots.txt file for search engines

published 2.5.4 2 months ago
M
Q
P

Hapi plugin to serve up robots.txt

published 3.2.2 3 years ago
M
Q
P

Vite plugin for generating robots.txt for production and development builds

published 1.0.5 8 days ago
M
Q
P