SEO → Robots & Crawl has four tabs.
robots.txt
Add your own rules to WordPress’s virtual robots.txt. Click View your robots.txt to check the result. The sitemap address is added automatically.
Ask AI companies not to train on this site blocks the known AI training crawlers (GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent and a few others). Search engines and AI search answers are not affected. Crawlers can choose to ignore robots.txt.
If a physical robots.txt file exists in your site folder, the page warns you that it takes precedence.
robots.txt is not noindex
robots.txt tells crawlers what to skip. It doesn’t remove pages from search results; use noindex for that.
Clean-up & feeds
- Head clean-up: remove tags most sites don’t need (generator tag, RSD link, shortlink, REST API and oEmbed discovery links, emoji script, extra feed links).
- RSS feeds: keep feeds out of search results, and add text before or after each feed item (for example a link back to the original post).
IndexNow
To tell Bing, Yandex and other participating search engines about changes immediately, check Turn on IndexNow and save. The plugin creates your key and key file, then submits pages when you publish, update or delete them.
- Use Submit URLs now to send addresses by hand.
- Recent submissions shows what was sent and the response.
- Nothing is sent from local or staging sites.
llms.txt
Turn on Serve /llms.txt to publish a plain-text index of your key pages for AI assistants (see llmstxt.org). Choose what to include, write an introduction, and check the preview. It is a proposal: Google doesn’t use it, and few AI tools read it yet.