LLM Ready

Added

• New "Enable llms.txt" setting (enableLlmsTxt, default true) — turns off /llms.txt and the /.well-known/llms.txt redirect without disabling the rest of the plugin, so .md URLs, content negotiation and discovery tags keep working. Previously the only way to remove the route was the global enabled setting, which took everything else with it. When off, the URL rules are never registered so the paths 404 naturally, the controller actions refuse to run (keeping them unreachable via their actions/… URLs), and the home page stops advertising an llms.txt alternate — there is no .md for the bare home page, so with llms.txt off it has no Markdown alternate to point at. Existing installs are unaffected: the default leaves the route on. Thanks to @nikolenko-dmitriy for the request (#23)

Changed

• "AI Bot User-Agent Detection" now defaults to off. Serving Markdown on the canonical URL based on the request's User-Agent is what made cache poisoning possible in the first place: the response varies by a header shared caches don't key on, so a bot request could be stored and replayed to real visitors as raw Markdown. Declaring Vary: User-Agent is the correct fix on paper but unusable in practice — the header has effectively unbounded cardinality, so honouring it would give every browser build its own cache entry, which is exactly why Cloudflare and others ignore Vary for HTML. Cache-correct and cache-efficient can't both hold, so the canonical URL now keeps a single representation by default.

  Crawlers are unaffected in the ways that matter: /llms.txt lists every entry's .md URL, and every HTML page still carries the <link rel="alternate"> tag and the Link header pointing at its Markdown alternate. Each of those is its own URL whose response never varies, so all of it stays cacheable. Accept: text/markdown content negotiation is unchanged and still on.

  Upgrading from 1.5.x or earlier? Nothing changes automatically — but please make this change yourself. An upgrade migration writes enableUserAgentDetection: true explicitly for any install that was relying on the old default, so the upgrade is behaviour-neutral; if you had already set the value either way, your choice is left alone, and only fresh installs get the new default. The migration deliberately doesn't decide for you: silently switching a working feature off during an upgrade would be its own kind of surprise.

  Recommended action: if a CDN or shared cache sits in front of your site — Cloudflare, Fastly, Varnish, or a platform edge such as Servd — go to the plugin settings and turn AI Bot User-Agent Detection off. That is the configuration this release steers toward, and it removes the variation behind #24 rather than only neutralising it with cache headers. Your crawlers keep working through .md URLs, /llms.txt, and the discovery tag and header. If your site is served straight from its origin with nothing caching in front, leaving it on is fine and costs you nothing. See "Why User-Agent detection is off by default" in DOCUMENTATION.md. Follow-up to the cache-poisoning report in (#24)

• Entries marked noindex in SEOmatic or Ether SEO are now excluded from all Markdown output. noindex is an explicit "don't surface this URL" signal, so LLM Ready now honours it the way a search engine would: the entry is dropped from /llms.txt and listing pages, its .md URL returns a 404, its canonical URL stops serving Markdown to AI bots and Accept: text/markdown requests, and its HTML page stops advertising a Markdown alternate in both the <link rel="alternate"> tag and the Link header. Live preview is exempt, so authors can still check a noindex entry's Markdown from the control panel.

  Note for existing installs. This changes output on upgrade. If you run SEOmatic or Ether SEO and have entries marked noindex today, those entries will disappear from /llms.txt and their .md URLs will begin returning 404 as soon as the cached output is rebuilt. That is the intended behaviour — but if you were relying on noindex entries staying available as Markdown, review them before upgrading. Sites with neither plugin installed are unaffected, and no lookups run.

  Both noindex and none count (none is shorthand for noindex, nofollow); nofollow, noarchive and nosnippet on their own do not. SEOmatic is read through its own resolver, so an entry inheriting noindex from its section or global meta bundle counts, not only one with a per-entry override — which means that for SEOmatic sites, building /llms.txt now runs SEOmatic's meta-container resolution once per listed entry. Results are memoised per request and /llms.txt remains cached for Cache TTL seconds, so the cost lands on a cache miss rather than every request. Every lookup fails open: if an SEO plugin's API throws or returns something unexpected, the entry is treated as indexable and a warning is logged. Thanks to @Mathew-WD for the request (#19)

• On the sites that deliberately keep canonical-URL User-Agent detection enabled, Markdown on that URL is best-effort behind a shared cache. The Markdown variant is no longer storable, and the HTML variant doesn't declare a Vary dependency on User-Agent, so an AI bot requesting a page a CDN has already cached as HTML receives that HTML. Nothing is ever served incorrectly; the feature just doesn't fire on a cache hit. Crawlers still reach the Markdown through the explicit .md URL and through the rel="alternate" discovery tag and Link header, none of which are affected — which is the reasoning behind the new default above.

Fixed

• Markdown served on an entry's canonical URL now sends Cache-Control: private, no-store and Vary: User-Agent, Accept. The plugin chooses the Markdown representation from the request's User-Agent and Accept headers, but the response carried no cache directives, so a shared cache keying only on the URL could store the Markdown variant from a single AI-bot request and replay it to every later visitor — real browsers included — until the entry expired. Affected pages rendered as raw Markdown for everyone and then recovered on their own once the cache refreshed, which made the fault look mysterious and location-dependent. Thanks to @aedan-umd for the detailed report and the fix (#24, #25).
• Blitz no longer stores the Markdown served on an entry's canonical URL. Cache-Control: private, no-store stops a shared cache sitting in front of the site, but a page cache running inside Craft never sees those headers — Blitz decides what to store from the response format and its own URI patterns, and doesn't inspect Cache-Control at all. Its default of cacheNonHtmlResponses = false happened to spare this response, since it is FORMAT_RAW rather than FORMAT_HTML, but that protection was incidental: turning that documented Blitz setting on was enough for a single AI-bot request to be stored under the canonical URI and replayed to every later visitor — served as Content-Type: text/html, so browsers tried to render the raw Markdown as a page. LLM Ready now opts out through Blitz's own API instead of relying on a header Blitz never reads. .md URLs and /llms.txt stay fully cacheable, since each is its own URL with a single representation. Found while verifying the fix for (#24)
• Nested entries no longer take the page down with a TypeError. A Matrix entry — and any other field-owned entry — belongs to a field rather than a section, so its sectionId is null. Give one its own URI format and template and it becomes routable, at which point the plugin passed that null straight into isSectionEnabled(int $sectionId) and fataled. getSectionConfig() and isSectionEnabled() now accept ?int and treat a null section as not enabled, since a nested entry has nothing to toggle in the settings UI. The guard covers all four paths that look a section up: the content-negotiation handler, the discovery-tag injector, the .md route, and renderMarkdown(). Thanks to @patrikkaprinay for the report (#21)
• /llms.txt no longer fatals when a live entry in an enabled section has no URL. Entry::getUrl() returns null for entries that have no URI — entries used as globals or as structural data — and formatEntryLink() handed that null to rtrim(), which throws a TypeError on PHP 8 before the existing !$url guard on the very next line could skip the entry. Thanks to @S-n-d for the report and the diagnosis (#20)
• The analytics dashboard's date-range and site filters now actually filter. The dashboard appended '?' + params to an action URL that already carried a query string, so the result held two ? separators. Which way that broke depended on the install's control-panel URL format: either the request 404'd — and the dashboard then threw a TypeError trying to read stats off the failed response — or the range value was folded into another parameter's value and silently ignored, so every range returned the same default 30 days of data. The parameters are now passed to Craft.getActionUrl(), which merges them correctly for both formats. Thanks to @markdrzy for the report (#22)

Security

These fixes were surfaced by an independent security review of the plugin. Thank you to the reporter for the careful and well-documented findings.

• The analytics dashboard no longer embeds server-side data into its inline <script type="application/json"> blocks with an unescaped json_encode. A logged requestPath containing a literal </script> sequence could otherwise close the block and inject arbitrary HTML into the authenticated control panel. The chartData and siteBaseUrl payloads are now encoded with JSON_HEX_TAG | JSON_HEX_AMP | JSON_HEX_APOS | JSON_HEX_QUOT, so HTML-special characters are emitted as \uXXXX escapes.
• The analytics data endpoint (analytics/data) now validates the range parameter against the values the dashboard offers (7, 30, 90, all), falling back to 30 for anything else. Previously an arbitrary integer was passed straight to DateTime::modify(), letting a caller push the query's start date far into the past (a full-table scan) or the future (skewed results).
• The botName and requestType filter parameters on analytics/data are now bounded: each comma-separated list is trimmed, de-duplicated and capped at 50 values, and requestType is additionally constrained to the known set (entry, llmstxt, listing, negotiated). This prevents a caller from forcing a many-thousand-element SQL IN (...) clause on every analytics query.
• The dedicated-LLM-template path guard in MarkdownService now uses a positive character allowlist ([A-Za-z0-9_\-./]) in addition to the existing .. / leading-/ checks, rejecting backslash traversal, null bytes and stray percent-encoding rather than relying on a blocklist alone.

Security

• The analytics dashboard's Chart.js library is now bundled with the plugin instead of being loaded from a third-party CDN (cdn.jsdelivr.net). The dashboard renders in the authenticated control panel, so a compromised or MITM'd CDN response would have executed with full CP privileges. The vendored copy is pinned to Chart.js 4.4.7.
• The analytics data endpoints are now scoped to the sites the current user is allowed to edit. Previously any user with the "View the analytics dashboard" permission could request analytics for any siteId, disclosing request paths and bot activity for sites they otherwise couldn't access. Requests for a non-editable site now return a 403, and the dashboard's site switcher lists only editable sites.

Fixed

• Deleting a site no longer leaves orphaned section settings in project config. The plugin now prunes its llm-ready.sectionSettings.*.{siteUid} entries when a site is removed, so a later project-config/apply (or a fresh install of the config) no longer aborts referencing a site that no longer exists.
• The project config add/update/remove handlers and the install-time rebuild no longer abort with a SiteNotFoundException when a section setting references a missing site. Sites::getSiteByUid() throws rather than returning null for an unknown UID, which defeated the existing === null guards; lookups now resolve to null and skip the orphaned entry as intended.
• The install-time rebuild of the section settings table is now best-effort: a single entry that fails to save no longer aborts the plugin install. Failures are logged and skipped, since the table is a cache that the config event listeners repopulate when project config is applied.

Added

• Expanded the built-in AI bot detection list with current crawlers: Claude-SearchBot and Claude-User (Anthropic), Perplexity-User (Perplexity), and Meta-ExternalAgent, Meta-ExternalFetcher, and Meta-WebIndexer (Meta) (#16)
• New botUserAgents config setting — a full replacement for the built-in bot user-agent list, for installs that want complete control. additionalBotUserAgents continues to append on top (#16)
• New excludeBotUserAgents config setting — remove specific entries from the effective list (e.g. drop a single default) without re-listing the whole list (#16)

Changed

• Removed Claude-Web (legacy, not in Anthropic's current crawler docs) and FacebookBot (a general crawler, not AI-specific) from the default detection list. Add either back via additionalBotUserAgents if you still want it (#16)
• Removed Google-Extended and Applebot-Extended from the default list — these are robots.txt opt-out tokens, not request User-Agents, so they never appear in a User-Agent header and matching them was a no-op (#16)

Documentation

• Brought the config.php template fully in sync with the available settings — it now includes excludeSelector, autoInjectLinkHeader, titleField, authorOverride, and the analytics options that were added in 1.4.0/1.4.1 but never documented in the template, and the descriptionField / titleField comments now describe the dot-notation, () method-call, Generated Field, and SEOmatic-resolver syntax
• Documented the HTTP Link discovery header (Auto-inject Link Header) and the analytics dashboard/purge permissions in DOCUMENTATION.md
• Overhauled AI-INSTALL.md to match 1.4.0/1.4.1/1.5.0: corrected the minimum Craft version to 5.9.18, added an SEO-plugin detection step (SEOmatic / Ether SEO / SEOmate / SEO Fields) for the Description/Title fields, an analytics setup step, an HTTP Link header verification test, and the Exclude Selector / Auto-inject Link Header / Title Field / Author Override settings

Fixed

• Auto-injected Link header is now emitted on HEAD requests as well as GET. Per RFC 9110, a HEAD response must carry the same headers as the equivalent GET, and some clients (uptime monitors, link checkers, curl -I) only issue HEAD (#7)

Changed

• Bumped minimum Craft CMS to ^5.9.18 (was ^5.5.0) so consumers no longer install Craft versions affected by GHSA-gj2p-p9m4-c8gw, GHSA-qrgm-p9w5-rrfw, and GHSA-33m5-hqp9-97pw, all patched in Craft 5.9.18
• Stopped committing composer.lock — distributed plugins shouldn't ship lock files, since consumers resolve dependencies against their own. This also clears noise from Dependabot scans of transitive dependencies that don't actually affect consumers

Added

• New "Exclude Selector" setting under Content Extraction — strip decorative or non-content elements (e.g. carousels, [data-nosnippet]) from the HTML before Markdown conversion (#3)
• "Description Field" now supports dot notation for traversing nested fields and sub-objects (e.g. seo.seoDescription inside a ContentBlock field, or seo.description for an Ether SEO field), () method-call syntax (e.g. metaData.getMetaDescription() for SEO Fields), and Generated Field handles (#4)
• Native SEOmatic resolver: set Description Field to seomatic:description (or seomatic:og-description, seomatic:twitter-description) to use SEOmatic's full resolution chain — per-entry override → section default → global default, with Twig token parsing. No Generated Field required
• New "Title Field" front-matter setting — point at any field handle/path to override the front-matter title: value, with the same syntax as Description Field. Falls back to the entry's native title when unresolved (#6)
• New "Author Override" front-matter setting — write a single authoritative author name (e.g., an editorial team) to every entry's front matter instead of leaking individual editor names (#6)
• New CP dashboard widget — compact summary of last-30-day analytics for the current site (total requests, top bot, top page) with a click-through to the full dashboard. Hidden when analytics are disabled or the user lacks the llm-ready:viewAnalytics permission (#5)
• New "Auto-inject Link Header" setting (default on) — adds an HTTP Link response header (RFC 8288) pointing at the Markdown alternate, alongside the existing <link rel="alternate"> HTML tag. Useful for crawlers that inspect headers without parsing HTML (#7)
• New SEO-PLUGINS.md documenting how to wire LLM Ready into SEOmatic, Ether SEO, SEOmate, and Studio Espresso SEO Fields
• New user permissions under "LLM Ready": "View the analytics dashboard" and the nested "Purge analytics data". Admins have both by default

Changed

• When "Description Field" is explicitly configured but resolves to an empty value, the entry's description is now omitted rather than silently falling back to auto-extraction from other fields (#4)

Security

• Analytics dashboard, JSON data endpoint, and purge action now require the corresponding permission. Previously any CP user could view analytics and trigger a purge. Existing non-admin users will lose access until granted the new permissions

Changed

• Chart legend filtering is now additive — clicking a bot or type shows only that item instead of hiding it; clicking it again shows all
• Clicking a chart legend item now updates the bot breakdown table, request types table, most accessed pages table, and stats cards to reflect the selected filter

Changed

• Chart legend items now show a pointer cursor on hover to indicate they are clickable for toggling datasets

Added

• Documentation on how to block specific bots via robots.txt

Added

• Toggle on the Requests Over Time chart to view stacked breakdowns by bot or by request type

Fixed

• Strip trailing slash from entry URLs before appending .md, preventing broken links like /about/.md in llms.txt and <link rel="alternate"> discovery tags
• Homepage discovery <link> tag now points to /llms.txt instead of the non-existent /.md
• Homepage analytics requests are now logged with a meaningful path and displayed as "Homepage" in the analytics dashboard

Fixed

• "Last Seen" dates in the analytics bot breakdown now correctly display in the Craft system timezone instead of UTC

Added

• 301 redirect from /.well-known/llms.txt to /llms.txt so LLMs checking the RFC 8615 well-known path are directed to the canonical location
• Documentation for the analytics dashboard, including explanations of the four request types (entry, listing, llmstxt, negotiated) and data retention

Added

• Opt-in analytics dashboard tracking AI bot visits to .md pages, /llms.txt, and content negotiation responses
• Bot breakdown table showing request counts and last-seen timestamps per crawler (GPTBot, ClaudeBot, PerplexityBot, etc.)
• Requests over time bar chart powered by Chart.js with date range filtering (7d / 30d / 90d / all time)
• Most accessed pages table with links to the served Markdown page and entry edit page
• Request type breakdown (entry, llmstxt, listing, negotiated)
• Multi-site support for analytics with site selector
• Configurable data retention period (default 90 days) with manual purge from dashboard
• Console command llm-ready/analytics/purge for cron-based data cleanup
• CP section for the analytics dashboard

Changed

• Analytics dashboard edit links are only shown to users with permission to view the entry
• Use Yii's registerLinkTag() for discovery tag injection instead of manual HTML string replacement
• Use Yii's getAcceptableContentTypes() for content negotiation instead of manual Accept header parsing
• Use Yii's Html::decode() for HTML entity decoding instead of raw html_entity_decode()

Fixed

• Logged-in users without section edit permissions no longer get a 403 error on public .md pages

Fixed

• Homepage singles no longer appear in the plugin settings page, since they can't serve .md URLs

Added

• Entry descriptions in /llms.txt and listing pages — each link now includes a brief description following the llms.txt spec format
• New "Description Field" setting to specify a field handle for entry descriptions (e.g., summary, excerpt)
• Auto-extract fallback that pulls a description from the first text field when no description field is configured
• Config file support — copy src/config.php to config/llm-ready.php to manage settings in code instead of the control panel

Fixed

• Homepage singles no longer appear in /llms.txt and listing pages with broken /.md URLs
• Sections with no listable entries (e.g. homepage singles) no longer show empty headings in /llms.txt

Added

• Markdown endpoint via .md URL suffix — append .md to any entry URL to get a Markdown version
• Content negotiation support — serve Markdown for requests with Accept: text/markdown header
• AI bot user-agent detection — automatically serve Markdown to known AI crawlers (GPTBot, ClaudeBot, Amazonbot, PerplexityBot, and others)
• Smart HTML-to-Markdown conversion using league/html-to-markdown with configurable CSS selectors for content extraction
• Dedicated LLM template support — assign a Twig template per section/site that outputs raw Markdown directly
• Auto-generated /llms.txt site index following the llms.txt specification
• YAML front matter with entry metadata (title, date, author, canonical URL, section)
• Auto-injection of <link rel="alternate" type="text/markdown"> discovery tags into HTML pages
• Listing page support — append .md to a section's base URL for a Markdown list of entries
• Per-section enable/disable control with optional LLM template configuration stored in project config for version control and multi-environment sync
• Markdown response caching via Craft's cache component with automatic invalidation on entry save/delete
• X-Robots-Tag: noindex header on Markdown responses to prevent search engine indexing (configurable)
• Content-Type: text/markdown; charset=utf-8 header with explicit charset to prevent encoding issues
• Link canonical header pointing to the HTML version of the page
• Graceful fallback to field-level content extraction when template rendering fails
• Multi-site support with independent settings per section/site combination
• Permission checks on all Markdown endpoints — logged-in users without view permission receive a 403
• Template path traversal protection and XPath injection prevention