Add a Dynamic Sitemap to Next.js Website Using Pages or App Directory

In this post, we will explore how to add a dynamic sitemap to a Next.js website. First, we add the sitemap to a website using the "pages" directory and then update it to take advantage of the new "app" directory. We'll implement a dynamic sitemap, which automatically updates as new pages are added or removed from the website. This is more useful for SEO purposes, especially for sites with regularly added content, like blog websites. Before diving into the implementation details, we'll briefly talk about what sitemaps are and their uses, as well as discuss the main differences between static and dynamic sitemaps. By the end of this post, you will have a fully functional dynamic sitemap for your Next.js website.

What is a sitemap and do I need it for my website?

A sitemap is a file that contains a list of all the pages of a website and their relationships to one another. It is used to help search engines crawl and figure out the structure of the website. A sitemap can also provide important information about each page, such as when it was last updated and how often it changes. Having a sitemap is particularly important for larger websites or those with a complex structure, as it can make it easier for search engines to discover and index all the pages on the site. However, according to Google, sitemaps are also beneficial for new websites, which do not have many external links to them and websites with content that is not well-linked to each other. In any case, a sitemap can help improve the website's SEO by providing valuable information to search engines and making the website navigation easier.

Static vs dynamic sitemaps

There are two main types of sitemaps: static and dynamic. A static sitemap is created manually and lists all the pages on a website. This type of sitemap is usually created in XML format and submitted to search engines to help them discover and index all the pages on the site. On the other hand, a dynamic sitemap is automatically generated and updated by the website's server whenever a new page is added or removed. This type of sitemap can be useful for larger or more complex websites where it may be difficult to manually keep track of all the pages. While both types of sitemaps serve the same purpose of helping search engines understand the structure of the website, dynamic sitemaps are generally considered more efficient because they eliminate the need for manual updates. For this reason, we'll add a dynamic website sitemap to a Next.js website. It will be automatically updated whenever new pages are added, which would happen for example, when a new blog post is published.

Adding a sitemap to the pages directory

Firstly, let's discuss how we would add a sitemap to a site based on the "pages" directory. To start, we create a sitemap.xml.js file inside the pages folder and add a default SiteMap component there.

jsCopy
// pages/sitemap.xml.js
 
export default function SiteMap() {}

jsCopy
// pages/sitemap.xml.js
 
export default function SiteMap() {}

The sole purpose of this component is to render the sitemap page when the sitemap.xml endpoint is hit. The actual content will come from the getServerSideProps function, which will get the URLs for all the posts, call the sitemap rendering function and write the response with the text/xml content-type header.

jsCopy
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

jsCopy
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

Since the purpose of getServerSideProps here is to send a custom response with the sitemap when the /sitemap.xml URL is accessed, we return an empty props object for the SiteMap component. getSortedPostsData is our API for getting sorted blog posts.

The final step is to implement the generateSiteMap function, which will extract the URLs from all the posts, add the links to all the static pages and render everything in the XML format.

jsCopy
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!-- Add the static URLs manually -->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

jsCopy
// pages/sitemap.xml.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!-- Add the static URLs manually -->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export async function getServerSideProps({ res }) {
  const posts = getSortedPostsData();
 
  // Generate the XML sitemap with the blog data
  const sitemap = generateSiteMap(posts);
 
  res.setHeader("Content-Type", "text/xml");
  // Send the XML to the browser
  res.write(sitemap);
  res.end();
 
  return {
    props: {},
  };
}
 
export default function SiteMap() {}

Here we are only adding the URLs, however, you could also add other attributes, like lastmod or priority. sitemaps.org has more details about all the possible attributes.

Another thing to note is that we are generating the XML as a string and not JSX, so we have to use JavaScript template literals here.

Make sure to replace the URLs with your own and after starting the app and navigating to yoursite.com/sitemap.xml, you'd be able to see the dynamically generated sitemap.

Adding a sitemap to the app directory

Next.js version 13 introduced a new "app" directory, which is meant to be a replacement for the "pages" directory in the long term. As a result of this change, Server-side rendering (using getServerSideProps) has been replaced with the Server Components. In our case, this means that we can no longer rely on getServerSideProps to render our sitemap page.

Luckily, we can use Route Handlers to achieve the same behavior. Route Handlers enable creating custom request handlers for specific routes using the Web Request and Response APIs. They are a replacement for API Routes from the "pages" directory.

Looking deeper into the documentation we can see that the Route Handlers can also be used to return non-UI content, which is what our sitemap is. First, we'll need to move the sitemap.xml.js into the app directory and change its name to route.js, which is the standard name for the file where Route Handlers are defined. Additionally, we'll nest it inside the sitemap.xml folder.

Here's the comparison of the old "pages" and new "app" directory structure:

bashCopy
// Before
blog/
  pages/
    sitemap.xml.js
 
// After
blog/
  app/
    sitemap.xml/
         route.js

bashCopy
// Before
blog/
  pages/
    sitemap.xml.js
 
// After
blog/
  app/
    sitemap.xml/
         route.js

After that, we will modify the route.js to create a custom GET request handler for this route, which will return a custom response with the generated sitemap. Additionally, we'll remove the getServerSideProps function and the SiteMap component as we no longer need them.

jsCopy
// app/sitemap.xml/route.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!--We manually set the two URLs we know already-->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export function GET() {
  const posts = getSortedPostsData();
  const body = generateSiteMap(posts);
 
  return new Response(body, {
    status: 200,
    headers: {
      "Cache-control": "public, s-maxage=86400, stale-while-revalidate",
      "content-type": "application/xml",
    },
  });
}

jsCopy
// app/sitemap.xml/route.js
 
import { getSortedPostsData } from "../lib/posts";
 
const URL = "https://claritydev.net";
 
function generateSiteMap(posts) {
  return `<?xml version="1.0" encoding="UTF-8"?>
   <urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
     <!--We manually set the two URLs we know already-->
     <url>
       <loc>${URL}</loc>
     </url>
     <url>
       <loc>${URL}/portfolio</loc>
     </url>
      <url>
       <loc>${URL}/blog</loc>
     </url>
     ${posts
       .map(({ id }) => {
         return `
           <url>
               <loc>${`${URL}/blog/${id}`}</loc>
           </url>
         `;
       })
       .join("")}
   </urlset>
 `;
}
 
export function GET() {
  const posts = getSortedPostsData();
  const body = generateSiteMap(posts);
 
  return new Response(body, {
    status: 200,
    headers: {
      "Cache-control": "public, s-maxage=86400, stale-while-revalidate",
      "content-type": "application/xml",
    },
  });
}

Here we add s-maxage of 1 day and stale-while-revalidate directives. More info about them is available on MDN.

Now if we go to the same yoursite.com/sitemap.xml URL we should see the same sitemap, but it is now rendered from the "app" directory.

Summary

Adding a sitemap to a website is an important aspect of search engine optimization, for large websites in particular. It helps search engines understand the structure of a site, discover all its pages, and crawl them efficiently.

This article demonstrated how to add a dynamic sitemap to a Next.js website, which is automatically updated whenever new pages are added or removed. It was shown how to create a dynamic sitemap for Next.js websites using either the "pages" or "app" directory.

A dynamic sitemap is more efficient than a static one, especially for larger or more complex websites. By implementing the techniques discussed in this article, you can ensure that your websites are easily discoverable by search engines, improving their overall SEO performance.

References and resources

• Google sitemaps overview
• MDN Cache Control
• MDN Web Request API
• MDN Web Response API
• Next-sitemap package
• Next.js API Routes documentation
• Next.js Route handlers documentation
• Next.js getServerSideProps documentation
• Next.js new app directory
• Sitemaps protocol