Setup Robots TXT Singapore: The Complete SME Owner Guide

Quick answer: To setup robots txt Singapore site owners should create a plain text file at the root of the domain, allow everything by default, block only admin, cart and internal search paths, declare the sitemap, and test in Search Console before going live. Blocking is not the same as deindexing.

The robots.txt file is the smallest file on your website and the one with the greatest capacity to destroy your search visibility in a single line. It is usually under ten lines long. It takes four seconds to break. We have been called in by Singapore businesses whose organic traffic collapsed over a fortnight for no reason anyone could identify, and on more than one occasion the cause was a two-line robots.txt file left over from a staging environment, telling every search engine in the world to stay away from the entire site. Nobody had touched the content, nobody had lost rankings to a competitor, and nobody had looked at the file, because it is not something most owners know exists. This guide explains what the file does, the one conceptual distinction that causes almost all the damage, how to write a sensible one for a Singapore SME site, and how to test it before it goes anywhere near production. If you want a second pair of eyes on yours, it is one of the first things covered in our SEO audit and consulting work.

What Robots.txt Does, in Plain English

Robots.txt is a plain text file that sits at the root of your domain, meaning it is always reachable at your domain followed by /robots.txt and nowhere else. Put it in a subfolder and it does nothing at all.

Its purpose is to tell automated crawlers which parts of your site they may request. When Google’s crawler arrives at your site, the first thing it does is fetch this file and read the instructions. If a path is disallowed, a well-behaved crawler will not request it.

The word to underline in that sentence is well-behaved. Robots.txt is a request, not a barrier. It is published publicly, anyone can read yours right now, and it is honoured voluntarily. Google, Bing and every reputable crawler respect it. Scrapers, content thieves and malicious bots read it as a helpfully annotated map of the directories you would prefer nobody visited. Never use robots.txt to hide anything sensitive. If a page must not be accessed, protect it with a password or server-level authentication. Listing your admin directory in robots.txt does not secure it; it advertises it.

It also controls crawl efficiency, which is the practical reason larger sites care. Every request Google makes to your site consumes a share of the crawling it is willing to do. Directing it away from thousands of pointless URLs, such as internal search result pages or faceted filter combinations, leaves more of that capacity for the pages that earn revenue. On a 30 page site this is irrelevant. On a 5,000 product catalogue it is significant.

The Distinction That Causes Almost All the Damage

Here is the single most misunderstood point in technical SEO, and it costs Singapore businesses real traffic every month.

Disallowing a page in robots.txt stops Google crawling it. It does not remove it from Google’s index.

Those are different operations. Crawling means fetching and reading the page. Indexing means including it in search results. A page can be indexed without ever being crawled, because Google can learn of its existence from links pointing at it elsewhere.

The consequence catches people out constantly. A business decides a page should not appear in Google, adds a Disallow line for it, and the page continues to appear in search results, now displayed as a bare URL with a note explaining that no description is available because the site’s robots.txt prevented access. The page is still listed. It just looks broken.

Worse, the two methods actively conflict. The correct way to remove a page from search results is a noindex tag in the page’s code. But Google can only read that tag by crawling the page. If you disallow the page in robots.txt, Google never fetches it, never sees the noindex tag, and the page stays indexed indefinitely. To deindex a page, you must allow crawling and apply noindex. Blocking it in robots.txt at the same time guarantees failure.

We find this error most often on sites handling sensitive or regulated content, where somebody has tried to suppress an old promotional page or a superseded pricing page. In clinical and financial contexts the stakes are higher than a stray URL, which is why we check for it specifically during medical SEO onboarding and in our finance SEO engagements.

The Syntax, Line by Line

The file uses a small set of directives and the syntax is unforgiving about slashes and case.

Several details matter more than they look. Paths are case sensitive, so Disallow: /Cart/ does not block /cart/. A trailing slash limits the rule to that directory, while omitting it also matches anything beginning with those characters, so Disallow: /car blocks /cars/, /career/ and /car-dealer-promotions/ alike. That particular mistake has real consequences for a dealership site where the whole commercial inventory sits under a path beginning with those three letters, a scenario we have had to untangle in car dealer SEO work.

Disallow with nothing after it means allow everything, which is the opposite of what most people assume when they see the word. Disallow: / with a single slash blocks the entire site. Those two lines differ by one character and by everything else.

Rules are matched by specificity, not by order. Google applies the most specific matching rule, so an Allow for a particular file beats a broader Disallow covering its directory.

A Sensible Starting File for a Singapore SME

The correct default posture for almost every SME site is to block almost nothing. Search engines are good at ignoring what does not matter, and every line you add is a line that can go wrong.

For a typical WordPress business site, a sound file allows all crawlers, blocks the admin area while permitting the one administrative file that front-end features depend on, blocks internal search result pages, and declares the sitemap. That is four or five lines in total and it covers the genuine needs of the overwhelming majority of Singapore SME sites.

For an e-commerce site, add blocks for the cart, checkout and account paths, since those pages are unique per session and have no business being crawled. Be careful with filter and sort parameters: blocking them in robots.txt prevents crawling but, as covered above, does not deindex any that are already in the index. Canonical tags are usually the better tool for that job.

For a site with a staging or development environment, the staging site should have its own robots.txt blocking everything, and the live site should not. The single most common catastrophe in this entire topic is the staging file being copied to production during a launch. Make checking robots.txt the first item on your post-launch checklist, before you check anything else.

For a booking-driven service business, block the confirmation and thank-you paths so that low-value, duplicate confirmation pages do not accumulate in the index. Appointment-led sites generate these quickly, and we see the pattern regularly across beauty SEO accounts where every booking flow produces its own URL.

How to Edit It on Each Platform

On WordPress, if no physical robots.txt file exists in the root directory, WordPress serves a virtual one. Most SEO plugins include a robots.txt editor in their tools section, which is the simplest route. Alternatively you can upload a real file via your hosting file manager or file transfer access, which then takes precedence over the virtual version.

On Shopify, the file is generated automatically with sensible defaults and, since the platform introduced template editing, you can customise it through a robots.txt.liquid template in your theme. Be conservative here. Shopify’s defaults are reasonable and most custom edits we review have introduced a problem rather than solved one.

On Wix and Squarespace, a robots.txt editor is available in the SEO settings on current plans, with the platform managing the defaults. The editing surface is limited by design, which for most owners is a mercy.

On a custom build, the file is either a static file in the web root or generated by the application. If your developer generates it dynamically based on an environment variable, confirm which value production uses. This is exactly the arrangement that produces a blocked live site when an environment variable is set incorrectly during a deployment.

On any platform with a content delivery network or a security layer in front, check that the file you see in your editor is the file being served publicly. Caching can serve a stale version for hours after you change it.

Testing Before and After You Publish

Test before deploying, always. Search Console includes a robots.txt report showing the file Google last fetched, when it fetched it, and any parsing errors. Use it to confirm Google is reading what you think it is reading.

Test individual URLs. The URL Inspection tool in Search Console will tell you whether a specific page is blocked from crawling. Check your homepage, your most important service page, and one deep page after any change.

Load the file in a private browsing window rather than relying on your editor’s preview, because caching layers lie.

Watch Search Console for a week afterwards. A spike in Blocked by robots.txt entries in the Indexing report is your early warning that a rule is broader than intended. The first signal of a serious error usually appears within days, well before traffic moves.

Set a recurring reminder to check the file quarterly. It costs a minute. Files drift: a plugin update writes a new one, a developer adds a line, a migration restores an old version. We have seen a site’s robots.txt silently reverted to a months-old version during a hosting migration, with nobody aware until rankings moved. A simple quarterly glance would have caught it in a fortnight rather than a quarter, and that kind of discipline protects gains like those in our interior design SEO results work, where 9 style and segment keywords reached page 1 within five months.

The Question Every Owner Now Asks: Should I Block AI Crawlers?

This is the live debate, and it deserves an honest answer rather than a confident one. A growing number of crawlers now gather content for training and for answering questions inside AI assistants, and many of them declare their own user-agent names that can be addressed in robots.txt. Blocking them is technically straightforward.

Whether you should is a commercial judgement, not a technical one, and it splits along a clear line. If your business model depends on people arriving at your website to read your content, blocking makes some sense. If your business model depends on people finding out that your Singapore firm exists and then contacting you, being absent from AI-generated answers removes a discovery channel that is growing, not shrinking.

Here is where we part company with common advice. Most agencies are currently telling clients to block these crawlers as a defensive default, and for a typical Singapore service business we think that is the wrong call. Your competitors’ content will be cited in those answers instead of yours. For a clinic, a law firm, an insurance broker or a contractor, the citation is the value and the click is a bonus. We would rather a prospective patient learn your clinic’s name from an AI answer than have your name absent from it. Publishers with paywalls and proprietary research are a genuine exception. Most SMEs are not publishers.

Our aesthetic clinic SEO results show how factual, compliant educational content earns search visibility in regulated categories, with the clinic’s patient FAQ content appearing in People Also Ask by Months 5-6.

Our insurance SEO results point the same way, with MAS-compliant product education pages as the first phase of a programme that grew monthly organic leads from 6 to 19.

Field notes: In our ecommerce case study, the WooCommerce store’s robots.txt had no rules blocking faceted navigation parameters, so three years of filter combinations had produced thousands of duplicate URLs and only 34% of product pages were indexed. Rewriting robots.txt to block the 14 parameter combinations generating duplicate content, alongside a clean XML sitemap and redirect chain fixes, helped move product indexation to 79% by the end of Month 2. Robots.txt problems cut both ways: a missing rule can hurt as much as a broken one, and nothing in a normal marketing dashboard surfaces either, which is why a quick quarterly check of the file is worth building into your routine.

Our Take

Robots.txt rewards restraint. The best file for almost every Singapore SME is a short one: allow everything, block the handful of paths that genuinely have no business being crawled, declare your sitemap, and stop. Every additional rule is a future failure mode, and the upside of aggressive blocking on a site of normal size is close to zero. Remember the distinction that causes most of the damage: this file controls crawling, not indexing, and using it to try to hide a page from search results will achieve the opposite of what you intended. Check the file after every launch, every migration and every major plugin update, and put a quarterly reminder in your calendar for the minute it takes to load it in a browser. If you inherited your site from a previous developer and have never looked at yours, look today, and if you would rather we did, our contractor SEO and wider technical work always starts with exactly this file.

Our clients who ask about blocking AI crawlers are usually trying to solve a licensing worry with a technical tool, and the two are not the same problem. In our experience, an aggressive robots.txt written out of anxiety costs far more in accidental indexation loss than the marginal training-data exposure it prevents ever does.

Frequently Asked Questions

Does every Singapore website need a robots.txt file?

Not strictly. If the file is absent, crawlers assume everything is permitted and proceed normally, which is the correct outcome for most small sites anyway. We still recommend having one, for two reasons: it lets you declare your sitemap location, and its absence means you have no record of intent, so nobody notices when a plugin or a host silently creates one. A minimal file you wrote deliberately is safer than no file at all.

How do I check whether my robots.txt is blocking something important?

Load your domain followed by /robots.txt in a browser and read it. Look for any Disallow line ending in a single slash with nothing after it, which blocks everything, and for any path that overlaps your commercial pages. Then use the URL Inspection tool in Search Console on your homepage and two key service pages, which reports explicitly whether crawling is blocked. Both checks together take under five minutes.

Can robots.txt remove a page from Google search results?

No, and this is the most consequential misunderstanding in the topic. Disallowing a page prevents crawling but does not remove an already indexed page from results. It will typically continue to appear as a bare URL with no description. To remove a page, allow crawling and add a noindex tag to the page, or use the removal tool in Search Console for urgent cases, then apply noindex for the permanent fix.

What happens if I block my CSS or JavaScript files?

Google renders your pages the way a browser does, so blocking stylesheets or scripts prevents it from seeing the page as your visitors see it. This used to be a common recommendation and is now clearly harmful. Blocked resources can make a responsive site look unusable to Google’s renderer, which affects mobile assessment and page experience evaluation. Leave your theme assets crawlable.

Should I block my WordPress admin directory?

Blocking the admin path is conventional and harmless, with one important exception: the admin-ajax file inside it is used by many front-end features such as forms, filters and load-more buttons. Block the directory but add an Allow line permitting that specific file. Most SEO plugins configure this correctly by default. Remember that this is not a security measure, since the file is public and anyone can read what you have listed.

Is robots.txt case sensitive?

The paths are, yes. Disallow: /Products/ will not block /products/. The directive names themselves are not case sensitive, so User-agent and user-agent both work, but every path you write must match the exact capitalisation used in your live URLs. This trips up sites that mix capitalisation across sections, which is more common on older Singapore sites built before lowercase URL conventions settled.

How long does it take Google to notice a robots.txt change?

Google caches the file and typically re-fetches it within a day, sometimes faster for active sites. If you have made an urgent correction, such as removing an accidental site-wide block, you can prompt a re-fetch through the robots.txt report in Search Console. Recovery of lost indexing after such a fix is slower than the fetch itself and can take days to weeks depending on the size of the site.

Can I use robots.txt to stop competitors scraping my content?

No. The file is honoured voluntarily and scrapers ignore it entirely. Worse, publishing a list of the directories you would prefer nobody visit is a useful hint to anyone acting in bad faith. Genuine protection comes from server-level rate limiting, bot management through a content delivery network, or authentication. Treat robots.txt purely as a politeness protocol for legitimate search crawlers.

My developer put Disallow: / on the live site. How much damage is done?

Potentially a great deal, and the speed of the damage depends on how quickly Google re-crawls your site. Remove the line immediately, request a re-fetch of the file in Search Console, then inspect and request indexing on your most important pages. Sites caught within days usually recover most visibility within a few weeks. Sites left blocked for months can take considerably longer, because pages drop out of the index progressively rather than all at once.

Do I need separate robots.txt files for my English and Chinese pages?

No. Robots.txt operates at the domain level and one file governs the entire domain, including all language folders on it. If your Chinese version lives on a separate domain or subdomain, that address needs its own file at its own root. For most Singapore bilingual sites running both languages in folders on one domain, a single file covering both is correct and simpler to maintain.