WWW Readiness for Artificial Intelligence

One of the basic files of every website is:

\/robots.txt

You can find it, for example, at:

https:\/\/www.domain.com\/robots.txt

The file specifies the rules for bots that crawl the website.

Google, for instance, checks the rules specified in robots.txt before loading the website and decides which parts it is allowed to crawl based on them.

Thus, modern website configuration must address not only Googlebot but also individual AI bots.

An example of a simplified configuration might look like this:

User-agent: OAI-SearchBot Allow: \/ User-agent: Claude-SearchBot Allow: \/ User-agent: Googlebot Allow: \/ User-agent: * Allow: \/ Sitemap: https:\/\/www.domain.com\/sitemap.xml

However, the specific configuration needs to be prepared individually according to the type of website.

For example, administration, internal search, cart, order processes, or specific parameterized addresses should not unnecessarily be made accessible to bots.

Another important component is the file:

sitemap.xml

This helps bots determine:

• which pages the website contains,
• which products exist,
• which articles are available,
• which pages have been updated,
• how the website is structured.

For extensive websites or e-shops, there may be separate maps for example for:

• products,
• categories,
• articles,
• images,
• language versions.

The sitemap must be current. If a website contains 20,000 products, of which half no longer exist, and the sitemap still links to old addresses, it creates unnecessary requests and complicates the work for bots.

A very important topic for modern websites is JavaScript.

The website may look perfect on a monitor, but a large portion of its content may only be generated after JavaScript is executed in the web browser.

This can pose a problem for some automated systems.

Therefore, we check whether important information exists directly in the resulting HTML document, or whether appropriate server-side rendering or other solutions are used.

The bot should be able to easily determine:

• the name of the company,
• main services,
• products,
• prices,
• locations,
• contact details,
• important product parameters,
• category texts,
• FAQ,
• references,
• other relevant information.

The web must not be "readable only by humans".

Artificial intelligence can process text very well. Nevertheless, it is significantly helped if the page is logically structured.

We therefore check, for example:

<title>

<meta name="description">

<h1>

<h2>

<h3>

<main>

<article>

<section>

<nav>

<header>

<footer>

Correctly used structure helps to determine:

• what is the main topic of the page,
• what is the heading,
• what is additional information,
• what is navigation,
• what is the actual content,
• how individual parts of the text relate to each other.

For example, a page titled only:

Services

provides the machine significantly less information than:

Custom kitchen and furniture production – Uherské Hradiště and surroundings

The correctly prepared HTML header is also important.

It includes, for example:

<title>Custom Kitchen Production | Company XY</title> <meta name="description" content="Custom production of kitchens and furniture in Uherské Hradiště and the surroundings. Design, production, and installation."> <link rel="canonical" href="https://www.domain.com/kitchens-on-order">

Depending on the type of website, we also add:

• canonical URL,
• language variants using hreflang,,
• Open Graph metadata,
• metadata for social networks,
• information for crawlers,
• correct HTTP status codes.

Duplicate addresses are a problem not only for classic SEO.

If, for example, the same content appears on five different URLs, the robot must determine which address is actually the primary one.

One of the most important technical layers of the modern web is structured data.

This is not text intended for visitors.

This is machine-readable information that explicitly tells robots:

This is a company.

This is a product.

This is a price.

This is the company address.

This is a phone.

This is an article.

This is a FAQ.

This is an author.

This is a service.

The most commonly used format is JSON-LD according to the Schema.org standard.

Google directly states that it uses structured data to better understand the content of a page.

For a local business, for example, you can utilize the structure type LocalBusiness. Schema.org allows describing, for example, the address, phone, location, logo, offered services, and other information.

A simplified example:

<script type="application/ld+json"> { "@context": "https://schema.org", "@type": "LocalBusiness", "name": "Company XY", "url": "https://www.domain.sk", "telephone": "+421123456789", "address": { "@type": "PostalAddress", "addressLocality": "Uherské Hradiště", "addressCountry": "CZ" } } </script>

For an e-shop, structured data can describe, for example, products, prices, availability, or other parameters.

For machines, this is significantly clearer than trying to infer all information purely from the visual appearance of the page.

A technically correct website is not enough.

AI search is based on questions.

Therefore, the methods of content creation are also changing.

The user does not only search for:

"carpenter Uherské Hradiště"

but may ask:

"Who can make a custom-built wardrobe for me in Uherské Hradište?"

"How much does a custom kitchen cost?"

"Who will come to measure the kitchen at home?"

"Which company manufactures furniture around Uherské Hradiště?"

If the website provides quality and clear answers to such questions, it gives the AI system more usable information.

Therefore, during optimization, we also check:

• individual service pages,
• locations,
• FAQ,
• product descriptions,
• references,
• contact information,
• company information,
• expert articles,
• clarity and precision of the texts.

A large portion of AI queries will be local.

For example:

"Find me a tire service in Kunovice."

"Where can I get a pergola made in Zlín?"

"Who repairs heat pumps in Uherské Hradiště?"

If it is not clear from the website where the company operates, AI has no safe way to ascertain this information.

That’s why we check, for example:

• the address,
• the city,
• the region,
• the serviced localities,
• the contact page,
• the structured data LocalBusiness,
• the connection of individual services with the locality.

However, it is not correct to create hundreds of practically identical pages with only a changed name of the municipality.

The content must be primarily high-quality and useful.

Speed is important not just for users but also for automated systems.

If a bot requests a page and the server responds after several seconds or repeatedly returns errors, processing the website becomes more complex and expensive.

That’s why we check:

• the size of HTML,
• images,
• WebP/AVIF,
• lazy loading,
• JavaScript,
• CSS,
• database queries,
• cache,
• Brotli/GZIP compression,
• HTTP/2 or HTTP/3,
• server speed,
• CDN,
• loading of external scripts.

Google uses Core Web Vitals among other metrics to assess user experience.

Currently, the main metrics include:

LCP – Largest Contentful Paint recommended value up to 2.5 seconds.

INP – Interaction to Next Paint recommended value under 200 ms.

CLS – Cumulative Layout Shift recommended value under 0.1.

Previously, websites were primarily visited by users and a few major search engines.

Today, additional automated systems are being added.

AI search engines, indexers, agents, and other services create another layer of automated traffic.

For some websites, the number of these accesses can be significant.

Therefore, it is necessary to check:

• the server capacity,
• the database,
• cache,
• the number of requests,
• server logs,
• rate limiting,
• protection against uncontrolled robots,
• the difference between a legitimate bot and a malicious scraper.

The aim is not simply to 'allow all bots'.

The goal is to allow access to those systems that make sense for the web, while protecting the web from unwanted automated load..

This is a very common problem today.

A website may have a properly configured robots.txt, but in front of the server, for example, there could be:

• Cloudflare,
• Web Application Firewall,
• DDoS protection,
• bot protection,
• custom firewall.

This can block the bot even before it reaches the website.

OpenAI, for example, points out that website protections can block legitimate bots and return them a response 403 Forbidden.

Therefore, it is not enough to check only the HTML pages.

It is necessary to check the entire path:

bot → DNS → CDN → firewall → server → application → database

With the development of automation and AI, the possibilities for automated vulnerability scanning, request generation, and web application analysis are also increasing.

Website security is therefore becoming increasingly important.

During a technical audit, we check, for example:

• outdated libraries,
• PHP version and other technologies,
• form security,
• SQL injection,
• XSS,
• CSRF,
• access rights,
• administrative interface,
• file uploads,
• API,
• HTTP headers,
• SSL/TLS,
• error pages,
• logging,
• protection against brute-force attacks,
• rate limiting.

It is important to distinguish between a legitimate AI bot and an automated attacker pretending to be a bot.

The text in the User-Agent header alone is not a reliable proof of the visitor's identity.

The bot must also correctly understand what happened to a specific address.

That’s why we check, for example:

200 OK the page exists.

301 Redirect the page has been permanently moved.

404 Not Found the page does not exist.

410 Gone the content has been removed.

429 Too Many Requests too many requests.

5xx server error.

If the website displays a page with the text 'product does not exist' instead of an actual 404 error but returns HTTP 200, the robot may consider the erroneous page as valid content.

The robot must be able to navigate the website.

Therefore, we analyze:

• main menu,
• breadcrumb navigation,
• links between services,
• links between categories,
• related products,
• articles,
• references.

An important page should not exist as an 'island' to which there is no link from anywhere on the website.

Images must also be correctly described.

We check for instance:

• file names,
• alt text,
• dimensions,
• compression,
• lazy loading,
• image availability to robots,
• access via CDN.

For example, the file:

IMG_45892.jpg

gives no information.

In contrast:

kuchyne-na-miru-uherske-hradiste.jpg

along with the correct ALT description, provides significantly more context.

In connection with AI, the file is increasingly discussed:

\/llms.txt

It is a draft standard aimed at providing AI agents with a concise and well-structured overview of important information about the website.

It may include, for example:

• company description,
• main parts of the website,
• important documents,
• links to services,
• documentation,
• other information suitable for LLM processing.

It is important to emphasize that llms.txt is not yet a universal internet standard with guaranteed support for all AI systems.

However, it is an interesting addition that is rapidly evolving and makes sense to prepare it for some projects. The proposal was created in 2024 and is continuously being developed.

However, it must not replace quality HTML, structured data, a sitemap, or the correct robots.txt.

After making adjustments, it is advisable to monitor who is actually visiting the website.

From the server logs, we can find out, for example:

• Googlebot,
• OAI-SearchBot,
• GPTBot,
• ClaudeBot,
• Claude-SearchBot,
• other bots.

We can monitor:

• the number of their visits,
• requested URLs,
• error responses,
• server speed,
• amount of transferred data,
• unusual load.

This allows for gradual adjustments of the configuration based on actual traffic.