Robots.txt generator

Use it locally

Create crawler guidance from user-agent groups and Allow/Disallow rules.

Change settings to clear the old result. Reset clears inputs and downloads; cancellation terminates the task worker. Clearing references does not promise secure physical memory erasure.

Know the limits

Up to 100 crawler groups, 100 agents per group, 1,000 rules per group and 100 sitemap URLs.