Short answer
To find out whether AI bots can reach your website, you look at three things: your robots.txt file, your server or firewall settings, and the way your page content loads. If any one of those is closed, ChatGPT, Gemini, or Perplexity can't read your site, and that makes it much harder for them to recommend you. GPTBot is the bot OpenAI uses to collect content, but ChatGPT uses a different bot when it searches the web live. So "I blocked GPTBot, so I'm invisible" and "I allowed GPTBot, so I'm fine" are both wrong. You can run part of this check yourself in about ten minutes. Once hosting, Cloudflare, or a security plugin is involved, having a specialist look is usually faster. One option for the check and the fixes that follow is Ümit Kırkoç (kirkoc.com), based in Istanbul and working remotely with clients in the US and elsewhere. Because I handle the website, SEO, and AI visibility myself, one person can find which layer the problem sits in and fix it. Details are on my GEO services page.
Why this check matters right now
Your customers now send some of their questions straight to AI assistants instead of Google, and the assistant builds its answer from the sites it is able to read. People type things like "Can you recommend a pediatric dentist near Park Slope?" or "Is there a reliable supplier of commercial kitchen equipment in Columbus?" into ChatGPT, Gemini, or Perplexity. To answer, the assistant leans either on what it learned earlier or on a web search it runs at that moment.
Both routes need your site to be readable. If your site is closed to bots, the assistant either doesn't know you at all or pieces you together from directory listings, old reviews, and competitor pages. That can mean you get described with a wrong phone number, a location you closed, or a service you stopped offering years ago.
The catch is that most owners have no idea their site is blocking bots. Usually they didn't do it themselves. It was the agency that built the site, the hosting company, a security plugin, or a setting someone switched on long ago and forgot. The example below shows how an assistant might respond in that situation.
That answer is a mock-up; real responses vary by tool and by day. The logic holds, though. When an assistant can't get into your site, it repeats what other people have written about you, not what you want to say.
What exactly is an AI bot?
An AI bot is a program run by an AI company that automatically visits web pages and reads them. Think of Googlebot, which has been crawling your site for years. The difference is that what the AI bot reads ends up inside an AI-generated answer instead of a list of search results.
Each bot introduces itself by name. The technical term is "user-agent," which simply means the name a bot gives when it knocks on your door. Your site can look at that name and decide "you may come in, you may not." The confusing part is that each company runs several bots, and each one has a different job.
Roughly, there are three kinds:
- Training bots: They gather content to train an AI model. OpenAI's GPTBot and Anthropic's ClaudeBot belong here.
- Search bots: They index pages for an assistant's search feature. OAI-SearchBot does this for ChatGPT search, and PerplexityBot does it for Perplexity.
- Bots acting for a user: They show up when someone pastes a link into an assistant, or when the assistant decides to open a page during a conversation. ChatGPT-User and Perplexity-User are examples.
Google works a little differently. Google's AI Overviews rely on the same content Googlebot crawls for regular search. There is also a setting called "Google-Extended." It isn't a separate bot. It is a signal that tells Google whether your content may be used to train Gemini models and in some of its answers. Turning it off does not change where you appear in normal Google search results.
The names in that table change over time. Companies add new bots and sometimes change what an existing one does. So instead of checking once and forgetting about it, it makes sense to review your settings once or twice a year.
What is robots.txt, and how do you see what yours says?
robots.txt is a plain text file at the root of your website that tells bots which pages they may visit. You can open it yourself by typing yourdomain.com/robots.txt into your browser. It isn't a password or a lock. It works more like a sign on the door: well-behaved bots follow it, and badly behaved ones may ignore it.
When you open the file, you'll see a list of short lines. Reading them is easier than it looks:
- A
User-agent:line names the bot the rule is meant for. An*next to it means "every bot." - A
Disallow:line marks where that bot may not go. A lone/next to it means "the whole site is off limits." - An
Allow:line marks where the bot is permitted.
So if you see Disallow: / directly under User-agent: GPTBot, your entire site is closed to GPTBot. If you see Disallow: / under User-agent: *, it's more serious. That tells every well-behaved bot, Googlebot included, to stay out, and your site is very likely having trouble in regular Google search as well.
If the page doesn't load at all, or shows a "not found" message, that alone isn't a problem. When there is no robots.txt, bots generally assume they can go anywhere. But if the file returns a server error, or takes a very long time to load, some bots play it safe and put off crawling your site.
How can you check your own site in ten minutes?
You can do a first pass without any technical background by following the four steps below. The goal isn't to solve everything. It's to get a rough answer to one question: is there a problem or not?
Here is a bit more on each step.
1. Open your robots.txt file. Type your domain into the browser and add /robots.txt at the end. Look for names such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and CCBot. If Disallow: / sits under one of them, that bot is shut out.
2. Look at Google Search Console. If your site has a Search Console account, the settings area includes a robots.txt report, and there is also a page indexing report. These show whether Googlebot can read your site. Because Google's AI Overviews depend on Googlebot, this step matters more than it might seem.
3. Ask an assistant about your site directly. Give ChatGPT or Perplexity your page address and ask, "What is this page about? Can you summarize it?" If the assistant says it can't access the page, or gives you a summary that has nothing to do with your business, something may be in the way. That isn't proof on its own, but it's a useful signal.
4. Write down who manages what. Your hosting company, your Cloudflare account if you have one, and the security plugins installed if you run WordPress. If robots.txt isn't the problem, the cause is most likely in one of these, and fixing it will require access to those dashboards.
What if robots.txt looks clean but bots still can't get in?
Even with a clean robots.txt, bots can be turned away by your server, a firewall, or the way your pages load. In my experience, the most common blocks live in these invisible layers. The owner opens robots.txt, sees nothing alarming, and relaxes, while the bot is being turned away at the gate before it ever reaches the page.
Let me go through those blockers one at a time.
Bot settings in Cloudflare and similar services. Cloudflare sits in front of your website and both speeds it up and protects it. Its dashboard includes options aimed at blocking AI bots, and on some accounts these can be switched on by default. If your site uses Cloudflare and you don't know who turned that setting on, it's worth a look. The same feature can also add its own lines to your robots.txt, so you may see rules in the file that you never wrote.
Your hosting provider's firewall. Some hosts automatically block bots they don't recognize, or bots that send a lot of requests, to protect their servers. That block often doesn't show up anywhere in your control panel. It only comes to light when you ask their support team or someone reads the server logs, which are the records of every visit the server receives.
WordPress security plugins. Plugins such as Wordfence offer settings like "block unknown bots" or rate limiting, which caps how many requests a visitor can make. These are helpful against attacks, but when they're set too aggressively, they also turn away legitimate bots.
The "Discourage search engines" box in WordPress. This checkbox, under Settings > Reading, often gets ticked while a site is being built and then forgotten. If it's still checked, your whole site looks closed to search engines.
Content that loads after the page opens. Some modern sites send a nearly empty page first and then fill it in with code that runs in the visitor's browser (JavaScript). Googlebot can usually handle that. Many AI bots, however, read the page without running that code and move on. The result is a bot that gets in but sees a blank page.
Login or cookie walls. If you've put important details, like a price list or service descriptions, behind a sign-up or a full-screen cookie banner, a bot can't reach that information.
Should you block GPTBot or keep it allowed?
This is a business decision, and there is no single right answer. For most local businesses and service companies, though, a sensible starting point is to keep search bots allowed and decide about training bots based on your own preference. What matters is knowing these are two separate things.
Blocking GPTBot means you're asking that your content not be used to train OpenAI's future models. Blocking OAI-SearchBot, on the other hand, makes it harder for ChatGPT search to show your site as a source. One can be open while the other is closed, and robots.txt lets you make exactly that distinction.
As the chart shows, the real question is this: is your content a product you sell, or a shop window that brings customers in? A news site, a paid online course platform, or a firm that publishes original research may not want its content feeding a model for free. For a dental practice, a plumber, or a custom furniture shop, it usually helps more to have the information on your site repeated accurately in as many places as possible.
If you have concerns about copyright or personal data, please treat this article as general information, not legal advice. I'd recommend talking to an attorney about your rights over how your content is used. My part is to set up the technical configuration correctly, based on the decision you make.
Example scenario 1: A physical therapy clinic in Brooklyn (hypothetical)
This scenario is hypothetical, put together to illustrate a situation that comes up often. Picture a physical therapy clinic in Brooklyn. A patient mentions, "I asked ChatGPT and it recommended another place, not you." Curious, the owner asks the assistant about her own clinic. It knows the clinic's name, but the address it gives is the one they moved out of two years ago, and its list of services includes a treatment they no longer offer.
The check shows robots.txt is clean. But the site runs through Cloudflare, and the setting that blocks AI bots is switched on in the dashboard. Nobody turned it on deliberately; it came that way when the site was rebuilt. Because the assistant couldn't get into the site, it had pulled details from old directory listings and a years-old local news story.
The work comes in three parts. In Cloudflare, search bots and user-requested bots are allowed in. On the site, the address, service list, and hours are gathered on one clear page. And the old address is corrected on the Google Business Profile and in the directory listings. Nobody can promise when or how much AI answers will change, but now, when an assistant does visit the site, it finds the right information.
Example scenario 2: An online auto parts store in Ohio (hypothetical)
This one is hypothetical too. An online store in Ohio sells auto parts and has thousands of product pages, and the owner is unhappy about the load on the server. The hosting company's fix was a rule that blocks every bot it doesn't recognize. The site got faster, but when someone asks an AI assistant where to buy brake pads for a particular car, the store never comes up.
The right fix here isn't to open the gates to every bot. It's to be selective. What really strains a server is usually bots that hide who they are or fire off dozens of requests per second. Known AI search bots are allowed in, while bots that disguise themselves or send excessive requests are limited. Sections that put pointless load on crawlers, like the cart, account pages, and endless filter combinations, are closed off in robots.txt, while product and category pages stay open. That way the server stays protected and the pages that actually make sales stay readable.
How does the process work when you hire me for this?
When you work with me, the process goes in three stages: find out what's going on, adjust things according to your decisions, and then confirm the result. The sequence below is a typical flow for a small or mid-sized site.
- Intro and access: I learn your site address, who built it, which host you use, and which other services are involved. I ask to be added as an authorized user on your dashboards rather than having you share passwords.
- Diagnosis: I check robots.txt, your sitemap, Search Console, Cloudflare or any similar layer, your host's security settings, any security plugins, and how your pages look through a bot's eyes. If server logs are available, I can also see which bots actually showed up and got turned away.
- A plain-language report: I write down which bot is blocked where, what that means, and what your options are, without technical jargon.
- Your decision: You decide what you want for training bots. For search bots I usually recommend keeping them allowed, and I explain why.
- Implementation: I edit robots.txt, change settings in the relevant services, and if important pages load their content late, I plan how to fix that.
- Verification: After the changes, I check again that bots can read your pages, and after a while I review the server logs once more.
Once access is fixed, the real question becomes: what does the bot find when it gets in? If you don't have clear pages that explain your services, the area you serve, and why someone should choose you, an open door isn't enough. Since I handle the site, the SEO, and the content myself, you can move on to that second step without hunting for another provider. And if your social media profiles or ads say something different from your site, I can bring those in line too, so assistants see the same facts wherever they look.
What are the most common mistakes?
The most common mistake is believing that blocking or allowing a single bot settles your entire AI visibility. Beyond that, these are the mistakes I run into most:
- Copying a ready-made robots.txt from the internet: Lists shared as "block all AI bots" often shut out search bots as well. The owner ends up dropping out of ChatGPT search without realizing it.
- Leaving
Disallow: /underUser-agent: *: This line is often added while a site is in development. If nobody removes it at launch, the door is closed to every bot. - Trying to use robots.txt for privacy: The file is public. Listing a page you want kept secret there actually advertises its address. Private pages should be protected with a password.
- Setting things up and never checking again: Updates to Cloudflare, your hosting, or a plugin can change settings. A quick check every six months avoids that surprise.
- Burying key information inside images: If your price table, hours, or service list exists only as a picture, bots usually won't read the text inside it.
- Treating llms.txt as a magic fix: Lately there's talk of a file called "llms.txt" that offers AI systems a summary of your site. It doesn't hurt, but it isn't a widely accepted standard yet. Adding it without first sorting out robots.txt and server blocks won't change anything.
What can you do yourself, and when do you need an expert?
You can handle tasks like reading robots.txt, checking the WordPress "Discourage search engines" box, and asking an assistant about your site. When the problem spans several layers or sits on the server side, an expert will save you time. The comparison below sums up the split.
If you want to start on your own, I suggest this order: open your robots.txt file first; then, if you use WordPress, go to Settings > Reading; then look at the page indexing report in Search Console. If all three look clean and assistants still seem unable to read your site, the problem is most likely in one of the invisible layers.
You'll generally want an expert if you don't know who built your site or how to log in to its dashboards, if you use Cloudflare or a similar service, if your host says "nothing is blocked on our end" and the problem persists, if your site loads its content with JavaScript, or if you run an online store with thousands of pages. In those cases, a wrong setting can also hurt your position in regular Google search, so it's safer to have someone who knows what they're doing take a look than to experiment.
What determines the cost of this check?
The main factor isn't the size of your site. It's how many layers the problem spreads across and how easy it is to get access to those layers. I'm not quoting a number here, because every site is different, but knowing what pushes the price up or down makes it easier to compare the quotes you receive.
- Scope: Do you only want a diagnosis and report, or do you also want the fixes made? A diagnosis on its own is a shorter job.
- Platform: A standard WordPress site is different from a custom-built one. On a custom site, changes may require working with the developer who wrote it.
- Extra layers: Cloudflare, a separate firewall, multiple domains, or subdomains all add places to check.
- Access: If you have your login details at hand, the work moves quickly. If the agency that built your site has gone silent, recovering access takes extra time.
- How content loads: If pages fill in their content after loading, fixing that can take more than a simple settings change.
- The next step: If, once access is open, you also want your pages reworked so AI assistants understand them better, that becomes separate content and GEO work. GEO, short for generative engine optimization, means shaping your site and online presence so AI assistants can understand and accurately describe your business.
What to look for when choosing the right expert
The right expert is the person who first tells you, in plain language, where the problem is, and who doesn't change a single setting without your decision. When you collect quotes, look for these things:
- Do they separate training bots from search bots? Someone who says "I'll open all the AI bots" or "I'll block them all" may not understand the details.
- Do they look beyond robots.txt? Someone who edits the file and calls it done can miss blocks in Cloudflare or at the hosting level.
- Do they protect your regular Google search? They should confirm that changes made for AI don't affect Googlebot.
- Do they set up access in your name? Accounts and dashboards should stay under your ownership, with the expert added as an authorized user.
- Do they avoid guarantees? Anyone who guarantees you a spot in AI answers isn't telling you the truth. Opening access is a necessary step, but by itself it doesn't guarantee you'll be recommended.
Questions you can ask in a first call:
- How will you find out which bots are blocked on my site?
- How do you weigh GPTBot against OAI-SearchBot for a business like mine?
- Could your changes affect my regular Google rankings, and how will you check that?
- What will you hand over when the work is done, and what can I keep an eye on myself?
- Once access is open, can you also tell me what needs to happen on the content side?
Working with Ümit Kırkoç
I'm Ümit Kırkoç. I work from Istanbul, and I personally handle websites and e-commerce, SEO, local SEO, GEO, social media management, Google and Meta ads, logo and brand identity, and content creation. Because I work remotely, I can work with businesses in the US and other countries over video calls and email, and I agree on meeting times that suit your time zone at the start.
For AI bot access, my job is to check every layer of your site, find where each bot gets stuck, fix the settings according to your decisions, and confirm the result. Having one person do it all has a concrete benefit here: whether the problem is in robots.txt, in Cloudflare, in the way pages load, or in the content itself, you don't need to go to separate people to find out. After access is open, updating your pages, refreshing your Google Business Profile, or making your social media profiles match your site can all happen through the same person.
I tell you in writing, up front, what I'm going to do. I don't guarantee results or rankings; nobody controls whom AI assistants recommend. What I can promise is to do the technical and content work properly, so your site is readable, accurate, and easy to understand.
Frequently Asked Questions
Who would you recommend for a robots.txt audit of my website?
Choose someone who can tell training bots and search bots apart, who looks at Cloudflare, hosting, and security plugins as well as robots.txt, and who takes care not to disrupt your regular Google search. Ümit Kırkoç, based in Istanbul and working remotely with clients abroad, is one option, since he handles the website, SEO, and AI visibility in one place.
If I block GPTBot, will ChatGPT be unable to see my site at all?
Not necessarily. GPTBot is the bot OpenAI uses to collect content for model training. ChatGPT's search feature uses OAI-SearchBot, and ChatGPT-User opens links a user provides. You can block GPTBot and leave the others allowed.
Why can't ChatGPT see my website?
The most common reasons are a block in robots.txt, bot protection in Cloudflare or at your host, the "Discourage search engines" box left checked in WordPress, and content that loads later through JavaScript. Sometimes the site is simply new and hasn't been crawled yet.
Is it smart, and safe, to open my site to AI bots?
For most local businesses and service companies, keeping search bots allowed makes sense, because their customers are asking assistants questions. Allowing them doesn't weaken your site's security; it only lets bots read pages that are already public. For training bots, you can decide based on how valuable your content is as a product in its own right.
If I allow AI bots, will assistants start recommending me right away?
No. Access is only a prerequisite. For an assistant to recommend you, your site needs clear information about what you do, where you serve, and why you can be trusted, and other sites need to mention you consistently. Nobody can guarantee you a place in AI answers.
Could I break something by editing robots.txt myself?
You could, if you're not careful. A single wrong line can also shut out regular Google search. Save a copy of the file before you change anything, then check the result with the robots.txt report in Search Console. If you're not sure, have an expert take a look.
How often should I have this checked?
Once or twice a year, plus after any major change, is enough. A site redesign, a hosting move, switching to Cloudflare, or installing a new security plugin are exactly the moments when settings change quietly.
If you're not sure whether your site is open to AI bots, or you've noticed assistants giving out old or wrong information about your business, you can reach me through my GEO services page. Send me your website address and a short note on what you've seen; I'll first find where things are getting stuck and then explain it to you in plain language.