Add Dynamic Sitemap to Next.js Website Using Pages or App Directory

Updated on February 27, 2024 · 6 min read|

As the field of web development continues to evolve, ensuring your website is optimized for search engines is essential for achieving visibility and success. One crucial aspect of this optimization is having a well-structured sitemap that communicates your website's hierarchy and content to search engine crawlers effectively. A sitemap is a file that lists all the pages of a website. It provides valuable information about each page, such as when it was last updated and how often it changes. This information can be beneficial for search engines when crawling and indexing your website. Combined with other features, such as a table of contents, a sitemap can significantly improve your website's SEO performance.

In this post, we'll explore how to add a dynamic sitemap to a Next.js website. We'll start by adding the sitemap using the "pages" directory, and then update it to leverage the new "app" directory introduced in Next.js version 13. Our goal is to implement a dynamic sitemap that automatically updates as pages are added or removed from the website. This approach is particularly beneficial for SEO purposes, especially for sites with regularly updated content, such as blog websites.

What is a sitemap and do I need it for my website?

A sitemap is a file containing a list of all the pages on a website, along with their relationships to one another. It helps search engines crawl and understand the website's structure. Sitemaps can also provide essential information about each page, such as when it was last updated and how often it changes. Having a sitemap is especially important for larger websites or those with complex structures, making it easier for search engines to discover and index all the site's pages. According to Google, sitemaps are beneficial for new websites with few external links and websites with content that isn't well-linked. In any case, a sitemap can improve a website's SEO by providing valuable information to search engines and enhancing site navigation.

Static vs dynamic sitemaps

There are two main types of sitemaps: static and dynamic. A static sitemap is manually created, listing all the pages on a website. Typically created in XML format, this type of sitemap is submitted to search engines to help them discover and index all site pages. Conversely, a dynamic sitemap is automatically generated and updated by the website's server whenever a page is added or removed. This type of sitemap is useful for larger or more complex websites where manually tracking all pages is difficult. Although both sitemap types serve the same purpose, dynamic sitemaps are generally more efficient, as they eliminate the need for manual updates. For this reason, we'll add a dynamic sitemap to a Next.js website, which will automatically update when new pages are added, such as when publishing a new blog post.

Adding a sitemap to the pages directory

In this section, we'll add a dynamic sitemap to a Next.js website using the "pages" directory. We'll create a new file called a sitemap.xml.js file inside the pages folder and add a default SiteMap component there.

js
// pages/sitemap.xml.js
 
export default function SiteMap() {}

js
// pages/sitemap.xml.js
 
export default function SiteMap() {}

The sole purpose of this component is to render the sitemap page when the sitemap.xml endpoint is hit. The actual content will come from the getServerSideProps function, which will get the URLs for all the posts, call the sitemap rendering function and write the response with the text/xml content-type header.

js
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

js
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

Since the purpose of getServerSideProps here is to send a custom response with the sitemap when the /sitemap.xml URL is accessed, we return an empty props object for the SiteMap component. getSortedPostsData is our API for getting sorted blog posts.

Are you currently using a Next.js website and considering adding a continuous deployment pipeline to it?

If so, I've written an article that might be helpful for you: Automate Your Deployment Workflow: Continuously Deploying Next.js website to DigitalOcean using GitHub Actions.

The final step is to implement the generateSiteMap function, which will extract the URLs from all the posts, add the links to all the static pages and render everything in the XML format.

js
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!-- Add the static URLs manually -->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

js
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!-- Add the static URLs manually -->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

Here we are only adding the URLs, however, you could also add other attributes, like lastmod or priority. sitemaps.org has more details about all the possible attributes.

Another thing to note is that we are generating the XML as a string and not JSX, so we have to use JavaScript template literals here.

Make sure to replace the URLs with your own and after starting the app and navigating to yoursite.com/sitemap.xml, you'd be able to see the dynamically generated sitemap.

Adding a sitemap to the app directory

Next.js version 13 introduced a new "app" directory, which is meant to be a replacement for the "pages" directory in the long term. As a result of this change, Server-side rendering (using getServerSideProps) has been replaced with the Server Components. In our case, this means that we can no longer rely on getServerSideProps to render our sitemap page.

Next.js 13.2 and lower

Fortunately, we can use Route Handlers to achieve the same behavior in Next.js 13.2 and lower. Route Handlers allow creating custom request handlers for specific routes using the Web Request and Response APIs. They serve as a replacement for API Routes from the "pages" directory.

Digging deeper into the documentation, we find that Route Handlers can also be used to return non-UI content, which is what our sitemap is. First, we need to move the sitemap.xml.js file into the app directory and change its name to route.js, the standard filename for defining Route Handlers. Additionally, we'll nest it inside the sitemap.xml folder.

Here's a comparison of the old "pages" and new "app" directory structures:

shell
// Before
blog/
  pages/
    sitemap.xml.js


// After
blog/
  app/
    sitemap.xml/
      route.js

shell
// Before
blog/
  pages/
    sitemap.xml.js


// After
blog/
  app/
    sitemap.xml/
      route.js

After that, we will modify the route.js to create a custom GET request handler for this route, which will return a custom response with the generated sitemap. Additionally, we'll remove the getServerSideProps function and the SiteMap component as we no longer need them.

js
// app/sitemap.xml/route.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!--We manually set the two URLs we know already-->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export function GET() {
  const posts = getSortedPostsData();
  const body = generateSiteMap(posts);
 
  return new Response(body, {
    status: 200,
    headers: {
      "Cache-control": "public, s-maxage=86400, stale-while-revalidate",
      "content-type": "application/xml",
    },
  });
}

js
// app/sitemap.xml/route.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!--We manually set the two URLs we know already-->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export function GET() {
  const posts = getSortedPostsData();
  const body = generateSiteMap(posts);
 
  return new Response(body, {
    status: 200,
    headers: {
      "Cache-control": "public, s-maxage=86400, stale-while-revalidate",
      "content-type": "application/xml",
    },
  });
}

Here we add s-maxage of 1 day and stale-while-revalidate directives. More info about them is available on MDN.

Next.js 13.3 and higher

Next.js version 13.3 introduced the file-based metadata API, which simplifies working with page metadata by exporting a Metadata object. This not only makes it easier to manage static and dynamic sitemaps but also eliminates the need for manual sitemap generation.

To take advantage of this, we can create a sitemap.js file that handles most of the repetitive logic involved in creating the website's sitemap.

shell
// Before
blog/
  app/
    sitemap.xml/
          route.js

// After
blog/
  app/
    sitemap.js

shell
// Before
blog/
  app/
    sitemap.xml/
          route.js

// After
blog/
  app/
    sitemap.js

In the sitemap.js file, our main task is to map the posts and static URLs to the properties of a Sitemap object, and then return this object.

js
// app/sitemap.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
export default async function sitemap() {
  const posts = getSortedPostsData.map(({ id, date }) => ({
    url: `${URL}/blog/${id}`,
    lastModified: date,
  }));
 
  const routes = ["", "/portfolio", "/blog"].map((route) => ({
    url: `${URL}${route}`,
    lastModified: new Date().toISOString(),
  }));
 
  return [...routes, ...posts];
}

js
// app/sitemap.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
export default async function sitemap() {
  const posts = getSortedPostsData.map(({ id, date }) => ({
    url: `${URL}/blog/${id}`,
    lastModified: date,
  }));
 
  const routes = ["", "/portfolio", "/blog"].map((route) => ({
    url: `${URL}${route}`,
    lastModified: new Date().toISOString(),
  }));
 
  return [...routes, ...posts];
}

Additionally, we include the lastModified field to display the date of the last modification for each page. This information can be beneficial for search engines when crawling and indexing your website.

With these changes in place, visiting yoursite.com/sitemap.xml should now display the updated sitemap, which is rendered from the "app" directory.

Summary

Adding a sitemap to a website is an important aspect of search engine optimization, for large websites in particular. It helps search engines understand the structure of a site, discover all its pages, and crawl them efficiently.

This article demonstrated how to add a dynamic sitemap to a Next.js website, which is automatically updated whenever new pages are added or removed. It was shown how to create a dynamic sitemap for Next.js websites using either the "pages" or "app" directory.

A dynamic sitemap is more efficient than a static one, especially for larger or more complex websites. By implementing the techniques discussed in this article, you can ensure that your websites are easily discoverable by search engines, improving their overall SEO performance.