Search results
121 packages found
Express middleware for generating your robots.txt
Rules related to checking for any SEO issues on the page given
Collection of SEO utilities like sitemap, robots.txt, etc. for a Remix application. A fork of https://github.com/balavishnuvj/remix-seo with some added bug fixes and features.
manual: favicon.ico显示在浏览器的地址栏或收藏夹中的网站icon 可以把其放在根目录中或在html中使用link来调用 robots.txt--爬虫协议文件,用于规定哪些区域可以被抓取
A simple, RFC-compliant robots.txt parser
A webpack plugin to output a robots.txt file
## About
robots.txt middleware for Kerouac.
A plugin to output a robots.txt file with options per environment
A plugin to output a robots.txt file with options per environment
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
Express middleware to serve and generate robots.txt
highlight.js syntax definition for Google's robot.txt format
- [x] Add all element types supported in Obsidian to markdown - [x] Add rich link previews - [x] Add RSS - [x] Add robots.txt - [x] AI - [x] Search - [x] Image - [x] Typing - [x] Add sitemap - [x] Make taxonomies linkable in frontmatter (if a corres
Sitemap generator for TypeScript projects
Gridsome plugin that automatically creates robots.txt for your site
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.