You had a new website built. You wrote up your services, added your photos, and launched. Months later, you search for your own business on Google and your inner pages are nowhere to be found. Maybe only the home page shows up, with no "Services" or "Contact" page. And the phone is not ringing as much as you hoped.
Behind this, there are often two small files: a sitemap and a robots.txt file. The names sound technical, but the jobs are simple. One tells Google "these are the pages I have," and the other tells it "please stay out of these areas." If they are set up wrongly, your entire site can be hidden from Google. In this article I explain what is a sitemap, why it matters, and what you can check yourself.
What is a sitemap? A page list you hand to Google
Picture a guest arriving at a hotel for the first time. If reception hands over a list of room numbers and floors, the guest finds their way at once. Without the list, they wander the corridors and may never see some rooms.
A sitemap is that list. When Google's robots (automated visitors that read websites) crawl your site, they look at this list to learn which pages exist and which changed recently. It is created automatically, visitors never see it, and it is meant only for search engines.
On a small, tidy site, Google often finds most pages even without the list. But if you add pages regularly, if some pages are not linked from your menu, or if your site is new, the list makes a real difference.
What is robots.txt? A note on the door
In the same hotel, some doors carry a sign: "Staff only." The laundry room, the boiler room, the storage area. Robots.txt is a sign like this hanging on the front door of your website. When Google's robots arrive, they read it first and skip the areas they are asked not to enter.
There are legitimate uses. You may not want your admin area, your shopping cart page, or a test section showing up on Google. The trouble starts when the sign is written wrongly. Write "Nobody may enter" on the main door and you have closed the whole hotel, not one room. That is the next section.
How one wrong setting can hide your whole site
I see this regularly. While a site is being built, the developer blocks the robots completely so Google does not see an unfinished site. That makes sense. But if the block is not removed at launch, your site looks like it is working for years and never appears on Google.
A similar thing happens in some website admin panels, where a checkbox such as "Discourage search engines from indexing this site" exists. If it stays ticked, the result is the same.
This is not visible from outside, and it costs real inquiries. A patient looking for an appointment searches for a dentist in your area. Even if everything on your site is written correctly, Google does not list your pages, so the patient finds your competitor and calls them. You never find out, because the person you lost never called. "My site opens, so it must be fine" is misleading. A site that opens and a site that Google can see are two different things.
A quick overview:
| Situation | Result |
|---|---|
| No sitemap | Google may find some pages late or never |
| Outdated sitemap | Deleted pages stay listed, new pages are missing |
| The door sign closes the whole site | Your site does not appear on Google |
| Set up correctly | Google finds your key pages and skips the unnecessary ones |
What you can check yourself
Three checks need no technical knowledge.
The "site:" search on Google
Type site: into the Google search box, followed immediately by your domain name with no space, for example site:yourcompany.com. Google lists the pages it knows from your site.
- No results at all means Google does not see your site yet. This is urgent.
- Far fewer results than the pages you actually have means some pages are missing.
- Old, unfamiliar, or irrelevant pages showing up means a cleanup is needed.
The page count in Search Console
Search Console is Google's free tool. Once you have confirmed you own the site, the "Pages" section shows how many of your pages are known to Google and why others are not. The "Sitemaps" section shows whether your list was sent to Google and read.
One warning sign: if you see a message like "Blocked by robots.txt" for a page that should be bringing in customers, show it to your specialist straight away.
The checkbox in your admin panel
If you manage your site through a WordPress-style panel, look in the settings for an option that discourages search engines. If you are unsure, do not change anything. Take a screenshot and send it to your specialist.
What the specialist does
This is the technical part, and you do not need to do it yourself:
- Create the sitemap and make sure it updates automatically as the site grows
- Submit it to Search Console
- Review the door sign and close off only what really needs to stay hidden
- Remove pages that should not be in the list, such as thank-you pages, duplicates, and deleted pages
- Make sure old pages do not stay in the list when the site is redesigned or moved
- Check everything again after any major change
What to ask your specialist
- "Do we have a sitemap, and has it been submitted to Search Console?"
- "Which areas did you close in robots.txt, and why?"
- "How did you confirm the robot block was removed at launch?"
- "When we add a page, does the sitemap update on its own?"
- "How many pages does Search Console show for us, and why are the missing ones missing?"
Someone who cannot answer these comfortably may never have checked.
How long does it take?
For most small and medium sites, setting up and checking is quick. The real wait is for Google to visit your pages again. That can take a few days or a few weeks, and nobody can name an exact time. A sitemap speeds things up but does not guarantee a result.
Frequently asked questions
If I have no sitemap, will my site not appear on Google?
It can still appear. Google also finds pages by following links from other pages. But having the list helps pages get found more reliably and quickly, especially on new or large sites.
Can I edit robots.txt myself?
I would not recommend it. One wrong line can remove the whole site from Google. You can look at it, but get expert advice before changing anything.
Will a sitemap improve my ranking?
Not directly. It helps Google know your pages exist. Ranking depends on the quality of your content and on your competitors.
I suspect my site is not showing on Google. What should I do first?
Do the site: search first, then look at Search Console. If the results look bad, pass what you found to your specialist.
Conclusion
To appear on Google, Google first has to be able to open every door of your site. A small check can uncover a block that went unnoticed for months. If you would like to review your site together, you can reach me through my SEO services page.