Search results
121 packages found
Express middleware for generating your robots.txt
Introducing Scrappey, your comprehensive website scraping solution provided by Scrappey.com. With Scrappey's powerful and user-friendly API, you can effortlessly retrieve data from websites, including those protected by Cloudflare. Join Scrappey today and
- cloudflare anti bot bypass
- cloudflare solver
- scraper
- scraping
- cloudflare scraper
- cloudflare turnstile solver
- turnstile solver
- data extraction
- web scraping
- website scraping
- data scraping
- scraping tool
- API scraping
- scraping solution
- View more
It uses the user-agents.org xml file for detecting bots.
The grunt task for html-snapshots
The lightweight robots.txt generator makes it simple to create compliant robots.txt files specifically for Astro integrations, with zero-config support, an intuitive JSDoc API, and always up-to-date verified bots user-agents.
- withastro
- astro-integration
- robots.txt generator
- seo assistant
- verified bots
- verified robots
- robots
- robots-txt
- astro-robots
A TypeScript-based project for easy serving of static files with support for live reloading, compression, and typed requests.
- TypeScript
- static file serving
- live reload
- compression
- typed requests
- HTTP server
- SSL
- cors
- express middleware
- proxy
- sitemap
- feeds
- robots.txt
- compression (gzip, deflate, brotli)
A Nuxt.js module that injects a middleware to generate a robots.txt file
Parser for XML Sitemaps to be used with Robots.txt and web crawlers
Deney all robots express middleware
A Nuxt.js module that injects a middleware to generate a robots.txt file
RFC 5234 spec compliant robots.txt builder and parser. 🦾
Robots.txt generator for Node.js
Sitemap generator for next.js. Generate sitemap(s) and robots.txt for all static/pre-rendered/dynamic/server-side pages.
Gatsby plugin that automatically creates robots.txt for your site
Check user agents against robots.txt files
An Eleventy plugin to generate a robots.txt file for your static site
An Assetgraph plugin that adds support for robots.txt and sitemaps