A new technical vocabulary has appeared around websites remarkably quickly.
OAI-SearchBot. GPTBot. Google-Extended. llms.txt. Markdown for Agents. AI crawlers. Agent accessibility.
Some of it represents meaningful changes to how machines interact with the web.
Some has been oversold.
For recruitment companies trying to decide what actually belongs on the technical roadmap, separating those two things matters.
Start with the crawlers
Not all AI-related crawlers have the same purpose.
OpenAI provides a particularly useful example.
OAI-SearchBot is associated with surfacing websites in ChatGPT search results.
GPTBot relates to crawling content that may be used to improve OpenAI's foundation models.
ChatGPT-User is associated with certain requests initiated by users.
Those distinctions matter because a publisher may be perfectly comfortable allowing its website to be discovered and cited while taking a different position on model training.
Google has a similar distinction through Google-Extended. Controlling Google-Extended does not remove a website from normal Google Search or affect its Search ranking, although it can affect certain Gemini uses.
There isn't one universal answer about which bots a recruitment company should allow.
The correct policy depends on what the organisation wants its information used for.
What has changed is that this policy now deserves conscious consideration.
llms.txt isn't an AI ranking file
This is probably the most important misconception to clear up.
There is no established evidence that creating an llms.txt file improves rankings or guarantees citations in ChatGPT, Gemini, Perplexity or other major AI products.
Google has specifically said it doesn't use llms.txt for Search.
That doesn't make the concept meaningless.
llms.txt is a proposed convention for giving language models a concise, structured guide to useful website information, usually pointing towards Markdown resources.
Version 2 of the proposal arrived in August 2026 and expanded the approach to include additional ways of associating Markdown content and llms.txt information with webpages.
Whether the specification becomes universally adopted remains unresolved.
Treating it as an experimental machine-discovery layer is reasonable.
Treating it as the new XML sitemap is premature.
Markdown is becoming more relevant
The broader idea behind llms.txt is already appearing elsewhere.
Cloudflare introduced Markdown for Agents in 2026.
When enabled, compatible systems can request a page using Accept: text/markdown and receive a cleaner Markdown version rather than the normal HTML response.
There is a straightforward efficiency argument behind this.
A webpage contains material necessary to display and operate the human experience. An AI system interested primarily in the information may not require all of it.
A cleaner representation can reduce unnecessary processing and token consumption.
This isn't proof of an AI visibility advantage.
It is evidence that major web infrastructure providers are now designing explicitly for machine consumption.
That's an important distinction.
Don't forget ordinary HTML
None of these developments remove the need for excellent HTML.
Semantic headings, meaningful links, properly labelled controls, structured content and accessible forms remain foundational.
OpenAI has documented that its agent can use ARIA information when understanding and interacting with websites.
This creates an overlap between accessibility and agent readiness, although accessibility should never be reduced to an AI optimisation technique.
Building accessible interfaces remains important because people need them.
The fact that machines can also make use of that structure is an additional benefit.
What this means for recruitment
Recruitment websites have an advantage because much of their information is naturally structured.
A vacancy isn't an abstract article.
It has a title, salary, location, employment type, description, skills, dates and potentially a recruiter and hiring organisation.
Consultants have names, expertise and locations.
Offices have geographic information.
Sectors contain related jobs and expertise.
Salary information relates to roles and markets.
Technical implementation should reinforce those relationships.
That is likely to remain useful regardless of whether any particular emerging AI specification survives.
The objective isn't to make a recruitment website “AI compliant”.
There currently isn't a universal standard deserving that description.
The objective is much simpler.
Make important information accessible.
Make its meaning clear.
Make relationships understandable.
Decide deliberately which automated systems can access it.
And keep enough flexibility in the underlying platform to adopt useful standards as they mature.
That is less exciting than promising an llms.txt file will get a company into ChatGPT.
It's also considerably more defensible.
