A public service page helps new customers see what you offer, while a paid research archive is something you sell. Cloudflare lets you set different rules for AI crawlers and other bots, so you will want to know which kind of page each rule covers before anyone on your team changes a setting.
You may dislike an AI company collecting your work without sending customers back and still want ChatGPT to read your service page when someone asks for a plumber nearby, and one broad block would shut out both. So before you block anything, decide which pages you want AI assistants to read and which you would rather keep to yourself.
You can leave the wider debate about AI to others. For your own site, write your rule down in a sentence or two, then check that the people and tools you want can still open your public pages.
Work out what each visitor is doing
Cloudflare’s July 1, 2026 update introduced controls organized around Search, Agent and Training behavior. Search means collecting pages for later answers or search results, Agent means a tool acting on a person’s behalf, and Training means collecting content to build AI models.
The three groups use your content in different ways. Agent traffic can come from a real customer, for example someone asking an assistant to compare your opening hours with a competitor’s before deciding where to go. A large training crawl, on the other hand, is a company collecting data, so look for signs of buyer interest in your inquiries instead.
Even so, judge each crawler on its own, because one group can hold crawlers of very different value. One bot visit might feed a useful answer to a customer while another has nothing to do with buyers, so treat the group label as a starting point and check it against what you want the site to do.
Cloudflare’s AI Crawl Control overview explains how to see crawler activity and set controls for each crawler. Look at that activity before you change any access, and write down which public pages the crawlers ask for and whether those requests go through.
Separate the pages that sell from the content you protect
A short spreadsheet with one row per group of pages is enough to start, and you can add every address later. For a hypothetical service business, it might look like this:
| Page group | Purpose | Initial policy question |
|---|---|---|
| Service descriptions and coverage areas | Help potential customers decide whether to inquire | Which search tools and customer assistants should reach these pages? |
| Public educational articles | Explain a problem and show how the business approaches it | Does broad reading support the reason we published these pages? |
| Paid reports or member material | Give paying customers their content | Does a login control access? |
| Customer records | Let a person manage their own information | Does every request check the person’s permissions? |
| Old offers | Keep a record or send people to the current offer | Should the page stay up, be corrected or redirect? |
The last two rows can turn up work that sits outside the bot settings. An old offer can mislead a person reading it too, and whatever a visitor says it is, a private customer page needs a login check.
A login is what keeps information private, so if a report should open only after payment, sign out and try to open its address directly. Bot settings can sit on top of that login, and the login still needs a test of its own.
Check the settings in your own account
Product names and defaults change over time. Cloudflare’s current bot-policy documentation describes policies under Security Settings, where you can block everywhere, block on pages with ads or allow a behavior, and it also covers changes scheduled for September 15, 2026 that include mixed-purpose Search and Training crawlers.
Accounts can have different choices and exceptions, and an old tutorial may show options you do not have, so we would open the live settings in your own account and take a dated screenshot of them before changing anything.
If your site depends on search traffic, take extra care here. Cloudflare’s announcement about the new controls warns that blocking training can also affect crawlers that do more than one job, so after you block anything labeled “training”, check that search tools can still reach your pages.
Then ask your developer, or whoever makes the change, to show you which requests the new rule would cover and which other security rules could still block a request later on. When you allow a group in one place, check what the site sends back too, because another setting can still block or change the response.
Make one change at a time
Say you decide your public service information should stay available to discovery tools, while one particular crawler should stop collecting a separate public archive. Write down the business reason before you touch the settings, in a short change note with:
- The content affected and the visitors affected.
- The current setting and the one you plan to use.
- Why you are changing it.
- Who made the change, and the date.
- The checks that show it worked.
- The old settings, so you can put them back if something breaks.
With a note like this, two people are less likely to change related settings at the same time, and if the site acts differently afterward, you know which change to look at.
Make the change when someone is around to check the result. Open the homepage and one service page, then look at recent requests from the crawlers you changed, and test as a crawler as well as in a normal browser, because the two can get different responses.
If a request gets blocked when you did not expect it, find the rule that blocked it and change that one rule while your broad protections stay on.
Read the result without inventing a sales story
A change in bot requests shows that your policy had an effect, and your inquiries are where you count the customers gained or lost.
Keep crawler activity, search visibility and inquiries side by side in separate columns. If you check what ChatGPT or Gemini says about you, save the question and the date, and when your business shows up in a later answer, note what else changed around that time before you give the access setting the credit.
A stretch with no requests may mean the service has not come back yet, so give it a few weeks before you change the rule or decide the page is invisible.
When you are deciding which findings to fix first, our readiness review guide helps turn what you see on your site into decisions, and our Cloudflare readiness score guide explains how to read each technical check.
Put a review date on the policy
A good first decision can still need changes later. An archive can become more or less valuable, or a crawler can end up in a different group, and a new assistant your customers start using is another reason to look again.
Look at the policy again when you launch a new section, change how the site makes money or see an unexpected drop in successful requests. Keep your page list with your change notes, so whoever looks next can see why each rule exists.
Moving to a new website counts too. The new site may sit in a different account or start with different defaults, so check each access setting again once the move is done.
Keep the final say yourself by writing a one-page policy, covering what is public, which AI uses you allow and who checks the settings, and let your engineer set up the site from it.
If you want AI answers to find and use your service pages, our Get found page explains how we help.