Is Your Recruitment Website Blocking AI Bots? How to Check Your AI Visibility in 2026

Recruitment agencies are being told they need to optimise their websites for AI search.

But before thinking about GEO, AEO, AI citations or any other new acronym, there is a more fundamental question:

Can AI search systems access and understand your recruitment website in the first place?

Some websites can unintentionally restrict crawlers through robots.txt rules, firewalls, CDN configurations, hosting settings or third party systems.

And for recruitment companies the problem can be more complicated because jobs may sit separately from the main website.

Your homepage might be accessible.

Your jobs might not be.

That matters.

Can AI bots crawl recruitment websites?

Generally yes, provided the website allows the relevant crawler access.

However, “AI bots” is becoming too broad a term.

Different crawlers can have completely different purposes.

OpenAI, for example, operates OAI SearchBot for ChatGPT search alongside GPTBot, which website owners can control separately for model training purposes. OpenAI explicitly allows publishers to permit search crawling while preventing GPTBot access. (OpenAI Developers)

Perplexity similarly provides controls for its web crawlers. (Perplexity)

This distinction is important because a company may reasonably decide it does not want its content used to train AI models while still wanting that content discoverable when somebody searches for its services.

The question should therefore not be:

“Are we blocking AI?”

It should be:

“Which automated systems are we allowing, which are we blocking and what commercial outcome are we trying to achieve?”

Why AI crawlability matters for recruitment agencies

Recruitment websites contain highly structured and commercially valuable information.

This can include:

• current vacancies
• role titles
• salaries
• skills
• industries
• locations
• consultant expertise
• employer services
• candidate advice
• salary guides
• market data
• FAQs
• client case studies
• reviews and testimonials

These are exactly the kinds of signals search engines and AI retrieval systems can use when answering recruitment related questions.

Examples include:

“Which recruitment agencies specialise in cybersecurity in Melbourne?”

“What does a senior financial controller earn in Sydney?”

“Find healthcare staffing firms in Dallas.”

“Who recruits renewable energy engineers in London?”

“What recruitment agencies specialise in executive search in Brisbane?”

“Find current Python developer jobs in Dublin.”

If your recruitment website clearly contains the answer, but the relevant page cannot be crawled or interpreted, you reduce the opportunity for that information to be discovered.

Google AI search still relies heavily on traditional SEO foundations

There has been a lot of unnecessary panic about SEO becoming obsolete.

Google's current guidance says almost the opposite.

Google states that its existing SEO best practices continue to apply to generative AI features such as AI Overviews and AI Mode. Pages need to be crawlable and eligible for indexing, and Google recommends clear technical structure, useful original content and a good page experience. (Google for Developers)

So the fundamentals still matter:

• crawlability
• indexing
• useful content
• logical website architecture
• internal linking
• site speed
• mobile usability
• structured data where appropriate
• sensible URLs
• strong page titles and descriptions

AI search has changed discovery.

It hasn't made good website architecture irrelevant.

Can Cloudflare block AI bots?

Yes, depending on how it is configured.

Cloudflare now provides detailed controls around automated AI traffic.

It distinguishes different crawler purposes rather than treating all AI bots identically, and on 15 September 2026 it is introducing updated defaults for new domains in certain configurations. Search traffic remains treated differently from training and agent traffic. (Cloudflare Docs)

This does not mean Cloudflare is bad for AI visibility.

It means your settings matter.

The same principle applies to other CDNs, firewalls and security systems.

Website owners should know whether legitimate search crawlers are being allowed, challenged, rate limited or blocked.

What is robots.txt?

A website's robots.txt file gives compliant crawlers instructions about which areas of the website they may request.

You can normally view it by visiting:

yourwebsite.com/robots.txt

A recruitment agency should check whether important public content has accidentally been restricted.

Potentially important areas include:

• jobs
• job detail pages
• sector pages
• industry pages
• location pages
• consultant profiles
• salary guides
• blogs
• FAQs
• case studies

However, robots.txt should not be treated as a website security mechanism.

It relies on crawler compliance.

Content that genuinely needs to be private should be protected using appropriate authentication, firewall or server controls. (Google for Developers)

Your ATS may be the part you forgot to check

Recruitment websites have an architectural issue most normal business websites do not.

Jobs may originate inside an ATS.

Where they are ultimately published matters.

A recruitment website may use:

youragency.com

while its jobs actually appear on another hostname, subdomain or ATS controlled URL.

Each environment can have different technical rules.

So when checking AI accessibility, don't just test the homepage.

Test:

• the job search page
• an individual job URL
• a registration page
• consultant job listings
• job location pages
• specialist job boards

If your most valuable recruitment content is somewhere AI systems struggle to access, that deserves attention.

Why first party recruitment job pages matter

A better recruitment website architecture brings job content into the agency's own branded digital experience.

That has several advantages.

Every job can have its own meaningful URL.

For example:

youragency.com/jobs/senior-accountant-melbourne

rather than sending candidates to a disconnected third party ATS experience.

With Shazamme, integrated job pages can also include structured JobPosting schema, giving search systems clearer information about the vacancy. (shazamme.com)

Structured job information can include fields such as:

• title
• description
• location
• salary
• employment type
• hiring organisation
• publication date
• expiry date

That makes a job page easier for machines to interpret and also supports Google for Jobs eligibility when implemented correctly.

What should a modern recruitment website handle automatically?

This is where the underlying website platform makes a significant difference.

Website owners should not need to manually reinvent basic technical SEO every time they publish a page.

On Shazamme websites, a range of technical foundations are handled natively.

Automatic XML sitemap

Published sites automatically generate a sitemap containing discoverable website content. Dynamic page records and blog content can be included automatically. (Duda Support)

Sitemaps help search engines discover URLs, particularly on larger websites.

robots.txt

A default robots.txt file is automatically generated, with the ability to replace it when more specific crawler rules are required. (Duda Support)

Automatic llms.txt

Published Shazamme sites automatically generate an llms.txt file containing public URLs and page metadata while filtering out drafts and no index pages. (Shazamme Support Portal)

This is an emerging convention rather than a proven ranking factor.

Google explicitly states that it does not use llms.txt for Google Search or its generative AI search features. (Google for Developers)

However, companies including OpenAI and Perplexity themselves now publish llms.txt documentation indexes, demonstrating that the format has practical utility for machine readable documentation. (OpenAI Developers)

For that reason, having the file generated automatically is useful future preparation.

It should not be sold as magic.

Built in performance optimisation

A significant amount of site optimisation happens automatically.

This includes:

• automatic image optimisation
• WebP image generation
• lazy loading
• video optimisation
• CDN delivery of static resources
• non critical CSS deferral
• CSS minification
• critical CSS extraction
• JavaScript code splitting
• WebFont optimisation
• Brotli compression
• caching optimisation
• preloading of important assets

These features are designed around site performance and Google's Core Web Vitals. (Duda Support)

The commercial point is simple:

The fewer performance fixes that depend on separate plugins and manual maintenance, the fewer moving pieces there are to break.

Secure HTTPS

SSL certificates are supported natively, with HTTPS redirection enabled by default when correctly configured, and the platform supports HSTS to strengthen transport security. (Duda Support)

Semantic HTML

The underlying platform uses semantic HTML5 elements where possible, including proper heading structures.

That is good for accessibility and gives machines clearer structural information about page content. (Duda Support)

Structured business information

Business information can be stored centrally and used to generate Local Business structured data for the primary business location. (Duda Support)

Recruitment websites can then add further recruitment specific schema where appropriate.

Dynamic pages with dynamic SEO

Dynamic content is particularly useful for recruitment websites because agencies often need scalable pages for industries, locations, consultants, services and other structured content.

SEO fields and page header information can be connected to the underlying data, allowing large numbers of useful pages to maintain individual SEO information rather than duplicating the same metadata everywhere. (Duda Support)

301 redirect management

Redirects can be managed centrally, which is particularly important when migrating an existing recruitment website.

Losing established URLs during a redesign is one of the easiest ways to throw away years of accumulated search value. (Duda Support)

Does llms.txt improve AI rankings?

At the moment, nobody credible should promise that.

Google explicitly says it does not use llms.txt for ranking or eligibility within Google Search's generative features. (Google for Developers)

Other AI focused organisations are adopting the format for machine readable documentation, but that is different from proving it improves ranking within ChatGPT, Claude or Perplexity.

The sensible approach is:

Generate it if your technology can do so automatically.

Keep it accurate.

Don't build your entire AI strategy around it.

And perhaps don't pay someone several thousand dollars to create a text file.

How can I test whether ChatGPT can access my recruitment website?

Start with a simple test.

Give ChatGPT a public URL and ask it to describe what is on the page.

Repeat the test with:

• your homepage
• an individual live job
• a sector page
• a consultant page
• a salary article
• an employer service page

OpenAI states that OAI SearchBot is the crawler responsible for surfacing websites within ChatGPT search results. (OpenAI Developers)

Remember that a successful user requested fetch does not prove every crawler has identical access.

Different search, training and user initiated systems can behave differently.

How can I test Claude?

Give Claude an exact public URL and ask it to retrieve and summarise the page.

Anthropic provides real time web search and web fetch capabilities within Claude products and APIs. (Claude Platform Docs)

Again, treat this as a useful smoke test rather than a complete crawler audit.

How do I know whether AI bots are actually reaching my website?

The best technical answer is server and CDN logs.

Your technical team can inspect requests from recognised crawlers and see the HTTP responses being returned.

Examples include:

200 OK

The page was successfully served.

403 Forbidden

Access was refused.

429 Too Many Requests

The crawler was rate limited.

A security challenge or CAPTCHA can also prevent automated systems from reading a page even though normal visitors can access it.

How should recruitment agencies audit AI crawlability?

Use this checklist.

1. Check robots.txt

Visit yourdomain.com/robots.txt.

2. Check your sitemap

Visit yourdomain.com/sitemap.xml.

3. Check llms.txt

Visit yourdomain.com/llms.txt if your website technology supports it.

4. Test important URLs

Don't only test the homepage.

5. Test live jobs

Confirm that jobs have their own accessible URLs.

6. Check JobPosting schema

Use structured data testing tools.

7. Check your ATS architecture

Find out whether jobs remain on your domain or are hosted somewhere else.

8. Check your firewall and CDN

Ask whether legitimate AI search crawlers are being challenged or blocked.

9. Check site speed and Core Web Vitals

Technical performance still matters for users and search engines.

10. Check Google Search Console

This one has become even more useful.

In June 2026, Google introduced a dedicated Generative AI performance report within Search Console, giving website owners greater visibility into impressions generated through AI features such as AI Overviews and AI Mode. (Google for Developers)

That means recruitment agencies can increasingly measure AI discovery rather than simply guessing.

What question should I ask my recruitment website provider?

Skip:

“Is my website AI ready?”

It is too vague.

Instead ask:

Are legitimate AI search crawlers able to access our public website pages and job URLs without being blocked, challenged or rate limited by our hosting, firewall, CDN, robots.txt configuration or ATS integration?

Then ask:

Do our individual jobs have their own crawlable URLs and JobPosting schema?

Then:

What technical SEO and AI readiness features are native to the platform, and which depend on third party plugins?

Those answers will tell you far more than an AI readiness badge on a sales brochure.

Do recruitment agencies need GEO specialists?

Sometimes specialist technical help is useful.

But recruitment companies should be cautious about paying for a completely new layer of “GEO optimisation” before checking their existing SEO, website architecture and content.

Google's current guidance is particularly relevant here.

Google says traditional SEO foundations remain important for generative AI search and warns against unnecessary AI specific tricks. It recommends technically accessible websites and useful, original, people first content. (Google for Developers)

For recruitment companies, that means concentrating on the things that actually make the agency understandable:

• strong sector pages
• location expertise
• useful salary content
• detailed consultant profiles
• genuine market insights
• current jobs
• structured job data
• employer FAQs
• candidate FAQs
• testimonials and proof
• case studies
• original expert opinion

AI needs something worth finding once it gets through the front door.

Frequently Asked Questions

Can blocking GPTBot stop my recruitment website appearing in ChatGPT search?

Not necessarily. OpenAI separates GPTBot from OAI SearchBot. OAI SearchBot is used for ChatGPT search, while GPTBot can be controlled separately. Website owners should therefore understand which crawler they are blocking. (OpenAI Developers)

Does Google use llms.txt?

No. Google currently says Google Search, including its generative AI features, does not use llms.txt as a ranking or visibility signal. (Google for Developers)

Should recruitment agencies delete robots.txt?

No. robots.txt has legitimate crawler management uses. It should be audited rather than blindly removed.

Can an ATS affect AI visibility?

Yes. If the ATS hosts job pages separately, those pages may operate under different crawling, indexing and security rules from the primary website.

Should recruitment jobs have individual URLs?

Yes. Individual accessible job URLs allow each vacancy to function as its own searchable piece of content and support structured JobPosting data.

What is the most important thing for AI visibility?

There is no single trick.

A strong recruitment AI visibility strategy combines:

technical accessibility, useful content, structured data, first party job pages, authority, speed, clear website architecture and genuine recruitment expertise.

Final thought

The AI search conversation is moving very quickly.

But don't let that trick you into ignoring the fundamentals.

You don't need a hundred new hacks.

You need a website that is technically sound, easy to crawl, easy to understand, fast, structured properly and packed with recruitment knowledge worth finding.

Get that right first.