We recently went through all our documentation to help make it more visible to search engines and particularly AI scrapers. Ironically the advice given was provided by an AI LLM, so let’s trust that it’s providing the right information. In this post, I’ll dig into front matter and using Cursor to help us update all our documentation.
What is front matter
Front matter is a block of metadata at the top of a document to help describe what’s on the document and how to categorise it. What you put into the front matter is up to you. Some example fields would be title, description, publication date, keywords, and categories. If you’re doing technical documentation, then other helpful fields could include language, sdks, and complexity.
And apparently it's front matter, not frontmatter?
Updating our front matter
The existing front matter we had on our pages looked something like:
---
page_id: 95d9b022-c6a1-43b0-8924-3026c1337012
title: "This cool page"
sidebar:
order: 2
relatedArticles:
- f4e916d4-58e6-4e9b-a8d9-aac74cb2631d
- 86438027-c3b7-47d0-b295-9ec3d541e0b9
app_context:
- m: user
s: details
description: "This is an awesome page with tons of epic information"
featured: false
deprecated: false
---
While reviewing our pages, we noticed that a lot of them had a generic description not applicable the page content or were missing a description altogether.
To handle the onslaught of AI agents and LLMs referencing our documentation in conversations, we wanted to make sure it had more context to respond with. Well, this was at least the AI based recommendation provided.
So our new front matter ended up looking like:
---
page_id: 95d9b022-c6a1-43b0-8924-3026c1337012
title: "This cool page"
sidebar:
order: 2
relatedArticles:
- f4e916d4-58e6-4e9b-a8d9-aac74cb2631d
- 86438027-c3b7-47d0-b295-9ec3d541e0b9
app_context:
- m: user
s: details
description: "This is an awesome page with tons of epic information."
topics:
- manage-users
sdk:
- nodejs
languages:
- javascript
audience:
- admins
- developers
complexity: beginner
keywords:
- multi-organization
- user management
- sign in experience
- enterprise authentication
updated: 2025-07-29
featured: false
deprecated: false
ai-summary: "This is an awesome page with tons of epic information."
---
With this added block of text, we’ll be providing a lot more goodies for the crawlers to consume. Particularly important to us was the sdk and keywords, which weren’t there before. These will help people find related docs when working with one our SDKs. And of course more keywords is always good.
Mass updates with Cursor
Our documentation site has over 350 pages. Updating each page individually and then populating the related content would be a serious chore. AI and Cursor to the rescue.
The context used in Cursor was:
Here's a sample template of the full front matter needed for all our documentation. I want to update the front matter for all docs in the documentation folder.
Keep the existing page_id, title, sidebar.order, app_context, topics, and relatedArticles. Add the others from the context below.
topics: please use the folder one above in the tree, so in this case it'll be "authenticate"
sdk: if there's mention of an sdk, then include them, otherwise leave it blank
languages: if there's mention of a coding language, then add it, otherwise leave it blank
audience: take your best guess
complexity: make a guess from beginner to expert
keywords: use your judgement based on the article
updated: figure out the commit date for the file
featured: leave as false for all articles
deprecated: leave as false for all articles
ai-summary: create a one line summary used for ai llms to index
This worked remarkably well and provided an easy diff view for each page to quickly eyeball the change. But based on the volume, I ended up doing the Accept All option.
The best part was the analysis of the document to write out the metadata, especially for things like the description, ai-summary, sdk, and keywords. Having the AI agent analyse the document and figure out what to put in there saved a lot of human brain power.
When looking at our Cursor dashboard, I’d guess that preparing and then completing this activity would have used about 10,000 lines of agent edits. Not sure how that converts to requests with Cursor, but I’d wager that at least a few hundred out of the included 500 requests were consumed with this.
One part that didn’t work too well was actually updating every file. I had to run through this a few times since it would seemingly pick random files throughout the structure, but not all of them. It wasn’t an issue through since it’s easy to see all the modified files in the folder structure and then re-prompt Cursor to grab the missed files.
What it looks like on the page
So once deployed, this content gets written to the <head> section of the HTML files. Here's the relevant parts on our Next.js App Router SDK page based on this front matter update.
<meta name="ai-summary" content="Complete guide for Next.js App Router SDK including installation, configuration, middleware setup, route protection, and authentication integration for Next.js 13+ applications."/>
<meta name="keywords" content="Next.js SDK, App Router, Server Side Components, middleware, authentication, route protection, environment variables"/>
<meta name="description" content="Complete guide for Next.js App Router SDK including installation, configuration, middleware setup, route protection, and authentication integration for Next.js 13+ applications."/>
<meta name="sdk" content="nextjs"/>
<meta name="languages" content="javascript, typescript, jsx, tsx"/>
<meta name="audience" content="developers"/>
<meta name="complexity" content="intermediate"/>
I think for a future pass, we'll try to optimise the ai-summary to be something more suitable to the AI LLMs, but this seems good for now.
Using AI in our docs already
We currently have an AI agent built into our docs that powers both the search and chat functions, which uses InKeep under the hood.
When speaking with their team, they said that the AI agent we use won't benefit too much from the extra front matter because they analyse our pages in detail already. The biggest benefit would be breadcrumbs to help provide category context, which we already include on our pages.
In summary
Hopefully these changes will have a positive impact with how search engines and the AI chat bots index our documentation. And to be honest, I don't even know how to measure the performance against these changes, but I guess it's better try something rather than do nothing. Using an AI tool to make the changes definitely saved a ton of time and effort providing the summaries and other contextual information.