Search results

astro-robots-txt

Generate a robots.txt for Astro

• astro-integration
• astro
• astro-component
• seo
• robots.txt
• robotstxt
• robots-txt

alextim

published 1.0.0 • 8 months agopublished 1.0.0 8 months ago

astro-robots

The lightweight robots.txt generator makes it simple to create compliant robots.txt files specifically for Astro integrations, with zero-config support, an intuitive JSDoc API, and always up-to-date verified bots user-agents.

• withastro
• astro-integration
• robots.txt generator
• seo assistant
• verified bots
• verified robots
• robots
• robots-txt
• astro-robots

jakichen

published 2.0.1 • 2 months agopublished 2.0.1 2 months ago

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

• robots.txt
• robots
• robotstxt
• generate
• robots-txt
• user-agent
• webpack
• webpack-plugin
• plugin

evilebottnawi

published 8.0.0 • 2 years agopublished 8.0.0 2 years ago

generate-robotstxt

Awesome generator robots.txt

• robotstxt
• robots.txt
• generate
• robots-txt
• user-agent
• police
• allow
• disallow
• crawl-delay
• robot
• robots

evilebottnawi

published 8.0.3 • 4 years agopublished 8.0.3 4 years ago

parse-robots

Promise based parser for robots.txt files.

• crawl
• web
• robots.txt
• seo
• robotstxt
• robots
• robots-txt
• sitemap

jaygoodfellow

published 3.2.0 • 4 years agopublished 3.2.0 4 years ago

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

• robots.txt
• robots
• robots-parser
• robots-exclusion-standard
• javascript
• js
• node-js
• robots-txt
• crawler
• crawlers

ibragim64

published 0.0.5-dev • 2 years agopublished 0.0.5-dev 2 years ago

@itsmatteomanf/astro-robots-txt

Generate a robots.txt file for your Astro site

• astro-integration
• astro-component
• withastro
• astro
• robots
• robots.txt
• robots-txt

itsmatteomanf

published 0.1.1 • a day agopublished 0.1.1 a day ago