What makes a recruitment website machine-readable?
A machine-readable recruitment website exposes its core content, relationships and actions through crawlable HTML, accurate structured data, stable URLs, clear labels and regularly refreshed job information. It does not rely on visual design or JavaScript alone to communicate what the business does.
Machine readability is not a separate layer built only for AI. Done properly, it improves accessibility, search eligibility, data quality and user experience at the same time.
Which crawlers should a recruitment site allow?
Crawler policy is a business decision, not a copy-and-paste robots.txt trend.
OpenAI separates search inclusion from model training. OAI-SearchBot is used for ChatGPT search, while GPTBot relates to potential model training. A business can allow search discovery while choosing a different policy for training. Review the current official crawler documentation and published IP ranges rather than relying on an old bot list.
Google’s AI Overviews and AI Mode draw from the Google Search index. Pages therefore need to remain crawlable, indexable and eligible for snippets. Bing recommends maintaining XML sitemaps and using IndexNow to signal changes quickly.
Does JavaScript stop AI from reading a site?
Not automatically. Google can process JavaScript, and browser agents can render pages.
The risk is that JavaScript-heavy implementations add failure points and delay.
Use server-side rendering, static generation or progressive enhancement for critical content where practical. At minimum, test the rendered page as a crawler and as a keyboard user. Confirm that jobs, consultant profiles, location details, navigation and calls to action are present and usable without relying on fragile interactions.
Which structured data matters for recruitment?
Use structured data that accurately reflects visible content:
- JobPosting: One live vacancy per page, with complete descriptions, correct location, dates, employment type and a working apply action.
- Organization: Legal and trading names, logo, contact details and relevant profiles.
- Person: Consultant and leadership profiles with genuine expertise and clear organisational relationships.
- LocalBusiness: Real locations with consistent names, addresses, phone numbers and opening information.
- Article: Authored insights with publication and modification dates.
- BreadcrumbList: Page hierarchy that helps systems and users understand context.
- VideoObject: Useful video content with a descriptive title, thumbnail and transcript or supporting copy.
Structured data must match the visible page. Hidden salary data, incorrect locations or expired job markup can create policy and trust problems.
How should job pages be managed?
Recruitment sites have a higher freshness burden than most corporate websites because inventory changes constantly.
Build a job publishing workflow that:
- creates a stable, descriptive URL on the agency domain
- maps ATS fields into complete visible content
- validates required and recommended JobPosting properties
- updates validThrough and availability immediately
- removes markup or returns the correct status when a vacancy closes
- notifies search systems of additions and removals
- preserves a useful expired-job experience without pretending the role remains open
The ATS feed should not be treated as a design afterthought. It is one of the largest structured datasets an agency publishes.
What should be tested every quarter?
- robots.txt and crawler access
- rendered HTML and internal links
- XML sitemap coverage
- Indexing API or IndexNow workflows where relevant
- canonical URLs
- structured data validity and visible-data matching
- expired job handling
- Core Web Vitals and mobile usability
- keyboard navigation and accessible names
- application completion on mobile
- AI descriptions of services, locations and people
Technical SEO has not been replaced. It has become the plumbing for both search and agents. Nobody notices good plumbing until the other sort floods the house.
