OpenAI, Google and Microsoft back a one-step AI crawler opt-out for websites

OpenAI, Google, Microsoft and other major AI companies said on 31 October that they will support a common standard allowing website operators to block AI training data collection in a single step. The standard, which emerged from Internet Engineering Task Force discussions, does not settle everything: coverage, enforcement and exceptions remain unclear.

What the standard changes

Under the current system, a website must block each crawler one by one in robots.txt, and update the file every time a new crawler appears. The new standard points to a single "no AI training data collection" setting that applies in bulk, while still allowing search engine crawlers through and blocking only crawls used for AI training.

Microsoft put the case plainly in its official blog: website operators should not be forced to choose between having their content appear in Bing search results and deciding how generative models use their work.

That speaks to the real position of small and mid-sized companies. For teams building independent sites for overseas markets, two fears tend to arrive together: content that search engines do not index is wasted work, and content fed to models for free is material grown for someone else. The two used to be bound together, leaving only a binary choice. At the level of rules, they are now being separated.

For overseas teams, the operating task has changed

What has changed is not whether to produce content, but that "who sees it" and "who crawls it" can for the first time be managed separately.

Take a typical case: a home-goods team selling overseas runs product specifications, installation instructions and selection comparisons on its site, written for search users, and that part should stay open to search engines. But pages such as customer cases, pricing logic and channel policy are often not meant to be absorbed wholesale by models. Under the new standard's approach, this becomes a distinction made at the settings level, not negotiated company by company.

The most common counter-argument is that nobody reads a small company's content, so blocking makes no difference. That holds under one condition: your site really has only a few introductory pages, with no original selection content and no accumulated customer Q&A. But once your content starts being copied by peers, or answered directly to potential customers by AI, the issue has nothing to do with scale.

The reverse also needs saying: this standard does not protect your trade secrets, and it does not stop anyone from manually copying your pages. It governs one thing, crawls used for AI training, and should not be expected to solve the whole problem of content being plagiarised.

Publishing content and having it trained on are turning from one thing into two; whether they can be managed separately will decide whether overseas teams still dare to put the real substance online.

Beyond the rules, your own data still needs managing

The standard governs others coming to crawl. It does not govern how you store and use data yourself.

Take a team running Hong Kong Type 9 family office business. Client identity documents, due diligence records and asset change approvals should never appear on a public web page at all, and discussing crawler standards is beside the point for them. The operating task for this kind of business is not whether to publish, but whether every internal step leaves a trace.

This is the part SHEYU AIFO takes on: the full KYC/CDD/EDD compliance process, including identity verification, document archiving, due diligence grading, risk assessment version management, and automated compliance reminders with audit log tracking; on the asset side, coverage of equities, bonds, funds, derivatives, insurance, cash, property, private equity, trust beneficiary rights and art collections, with daily and monthly valuation snapshots and transaction flow tracking; key operations go through dual review and approval, with KYC updates, asset adjustments and contract signing leaving tamper-proof records.

The boundary should also be stated: it is a custom enterprise product, does not support self-service registration and requires going through sales or leaving contact details; it addresses compliance and the asset view inside a licensed family office, not crawler blocking for an ordinary independent site. Choose the wrong tool and even good functions go unused.

One thing that can be done now

There is no need to wait for the standard to be fully in place. Start by sorting your site's pages: which are for search users, which are for signed clients only, and which should never have gone online at all. Once that is done, you may find that only a handful of pages genuinely require agonising over whether to block.

The rules are moving forward, but judging which content is worth publishing remains the owner's own job.