Removing Duplicates with Map In JavaScript

Updated on March 24, 2023 · 5 min read|

Removing duplicates from an array is a common task in JavaScript development, and using the Set constructor is a popular way to achieve this. However, while it works great for arrays of primitive values, it doesn't work as expected for arrays of arrays or objects. This is because Set stores either primitive values or object references. As a result, all values in the array have distinct references (unless an object/array is declared externally and referenced multiple times within the array), resulting in the original array remaining unchanged when converted to Set. In this article, we'll explore an alternative approach using the Map data structure, which maintains key uniqueness, to solve this problem. We'll also walk through a real-world use case for Map, demonstrating how it can be used to efficiently collect unique values from an array of objects in a React app.

Removing duplicates from an array with Set

Removing duplicate items from an array in JavaScript is a common practice, and one way to achieve this is by using the Set constructor. This involves wrapping an existing array in a Set and then converting it back to an array.

javascript
const arr = [1, 2, 'a', 'b', 2, 'a', 3, 4];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [1, 2, "a", "b", 3, 4]

javascript
const arr = [1, 2, 'a', 'b', 2, 'a', 3, 4];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [1, 2, "a", "b", 3, 4]

It is worth noting that Set uses same-value-zero equality to determine value equality. As a result, a NaN value will be considered equal to another NaN, despite the fact that NaN !== NaN. This means that if the original array contains multiple instances of NaN, only one of them will be kept when the array is transformed into a Set.

If you're interested in learning more about NaN and its type comparison, you may find this article helpful: What Is the Type of NaN?.

This works great for arrays of primitive values, however, when applying the same approach to an array of arrays or objects, the result is quite disappointing:

javascript
const arr = [[1, 2], {'a': 'b'}, {'a':2}, {'a':'b'}, [3, 4], [1, 2]];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [[1, 2], {'a': 'b'}, {'a':2}, {'a':'b'}, [3, 4], [1, 2]]

javascript
const arr = [[1, 2], {'a': 'b'}, {'a':2}, {'a':'b'}, [3, 4], [1, 2]];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [[1, 2], {'a': 'b'}, {'a':2}, {'a':'b'}, [3, 4], [1, 2]]

This is because Set stores primitive values or object references, and in our case, all values in the array have different references.

It's worth noting that this approach will work if a non-primitive element is declared externally and then referenced multiple times within the array, since all its occurrences in the array will point to the same element. However, such a use case is not that common.

javascript
const obj = { a: "b" };
const tuple = [1, 2];
const arr = [tuple, obj, { a: 2 }, obj, [3, 4], tuple];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [[1, 2], {a: "b"}, {a:2}, [3, 4]]

javascript
const obj = { a: "b" };
const tuple = [1, 2];
const arr = [tuple, obj, { a: 2 }, obj, [3, 4], tuple];
const uniqueArr = [...new Set(arr)];
 
console.log(uniqueArr); // [[1, 2], {a: "b"}, {a:2}, [3, 4]]

Before the introduction of Set in JavaScript, filtering out duplicates from an array was most often done using a combination of filter and indexOf methods.

javascript
const arr = [1, 2, "a", "b", 2, "a", 3, 4];
const uniqueArr = arr.filter((element, index) => {
  return arr.indexOf(element) === index;
});
console.log(uniqueArr); // [1, 2, "a", "b", 3, 4]

javascript
const arr = [1, 2, "a", "b", 2, "a", 3, 4];
const uniqueArr = arr.filter((element, index) => {
  return arr.indexOf(element) === index;
});
console.log(uniqueArr); // [1, 2, "a", "b", 3, 4]

However, this approach is inefficient because, for each element iterated by filter, there is an additional iteration over the whole array by the indexOf method.

Using Map to remove duplicates from an array of objects

A not-so-well-known fact is that the Map data structure maintains key uniqueness, meaning that there can be no more than one key-value pair with the same key in a given Map. While knowing this won't help us magically transform any array into an array of unique values, certain use cases can benefit from Map's key uniqueness.

To demonstrate this, let's consider a sample React app that displays a list of books and includes a dropdown menu to filter books by their authors.

jsx
const App = () => {
  const books = [
    {
      id: 1,
      name: "In Search of Lost Time ",
      author: { name: "Marcel Proust", id: 1 }
    },
    { id: 2, name: "Ulysses", author: { name: "James Joyce", id: 2 } },
    {
      id: 3,
      name: "Don Quixote",
      author: { name: "Miguel de Cervantes", id: 3 }
    },
    { id: 4, name: "Hamlet", author: { name: "William Shakespeare", id: 4 } },
    {
      id: 5,
      name: "Romeo and Juliet",
      author: { name: "William Shakespeare", id: 4 }
    },
    { id: 6, name: "Dubliners", author: { name: "James Joyce", id: 2 } }
  ];
  const [selectedAuthorId, setSelectedAuthorId] = useState(null);
 
  const filteredBooks = () => {
    if (!selectedAuthorId) {
      return books;
    }
    return books.filter(book => String(book.author.id) === selectedAuthorId);
  };
 
  return (
    <div className="books">
      <select
        className="books__select"
        onChange={({ target }) => setSelectedAuthorId(target.value)}
      >
        {/* TODO Show author options */}
      </select>
      <ul className="books__list">
        {filteredBooks().map(book => (
          <li className="books__item">
            {book.name} by {book.author.name}
          </li>
        ))}
      </ul>
    </div>
  );
};

jsx
const App = () => {
  const books = [
    {
      id: 1,
      name: "In Search of Lost Time ",
      author: { name: "Marcel Proust", id: 1 }
    },
    { id: 2, name: "Ulysses", author: { name: "James Joyce", id: 2 } },
    {
      id: 3,
      name: "Don Quixote",
      author: { name: "Miguel de Cervantes", id: 3 }
    },
    { id: 4, name: "Hamlet", author: { name: "William Shakespeare", id: 4 } },
    {
      id: 5,
      name: "Romeo and Juliet",
      author: { name: "William Shakespeare", id: 4 }
    },
    { id: 6, name: "Dubliners", author: { name: "James Joyce", id: 2 } }
  ];
  const [selectedAuthorId, setSelectedAuthorId] = useState(null);
 
  const filteredBooks = () => {
    if (!selectedAuthorId) {
      return books;
    }
    return books.filter(book => String(book.author.id) === selectedAuthorId);
  };
 
  return (
    <div className="books">
      <select
        className="books__select"
        onChange={({ target }) => setSelectedAuthorId(target.value)}
      >
        {/* TODO Show author options */}
      </select>
      <ul className="books__list">
        {filteredBooks().map(book => (
          <li className="books__item">
            {book.name} by {book.author.name}
          </li>
        ))}
      </ul>
    </div>
  );
};

For simplicity, the books array is hardcoded here, although in a real-world app, the data will be probably fetched from an API.

Now that the app is almost complete, we just need to render the dropdown of authors to filter by. One approach would be to collect the id and name of each author from our list of books into a separate array and render it as options inside the select. However, this list should contain unique authors, or else authors of more than one book will appear in the dropdown multiple times.

Because the author's data is contained inside an object, we can't just apply the Set trick to get unique values only. One option would be to first get all the ids for the authors into an array, then apply Set to it to get unique ones, and after that iterate over the authors array once more to collect their names based on the ids. However, this sounds like a lot of work.

Another solution, similar to the way for filtering out duplicates from an array in JavaScript before the introduction of Set, is to use a combination of filter and findIndexOf.

js
const authorOptions = books
  .filter((book, index, array) => {
    const bookIndex = array.findIndex((b) => book.author.id === b.author.id);
    return index === bookIndex;
  })
  .map((book) => ({ id: book.author.id, author: book.author.name }));

js
const authorOptions = books
  .filter((book, index, array) => {
    const bookIndex = array.findIndex((b) => book.author.id === b.author.id);
    return index === bookIndex;
  })
  .map((book) => ({ id: book.author.id, author: book.author.name }));

While this is better than the first approach, it is still quite verbose, and for our particular use case, we need to run an extra map iteration over the result to transform it into the shape we need for the options. Luckily, there's an easier solution.

We can extract an array of id-name pairs from the books list and transform it into a Map. This approach automatically takes care of preserving only the pairs with unique keys, avoiding the need for multiple iterations and unnecessary code.

javascript
  const authorOptions = new Map([
    ...books.map(book => [book.author.id, book.author.name])
  ]);

javascript
  const authorOptions = new Map([
    ...books.map(book => [book.author.id, book.author.name])
  ]);

That's it! Now we have a Map of unique key-value pairs that we can feed directly into our select component. It's worth noting that when a Map preserves key uniqueness, the last inserted item with the existing key remains in the Map, while previous duplicates are discarded.

javascript
const map1 = new Map([[1,3], [2,3]]);
 
const map2 = new Map([[1,2]]);
 
const merged = new Map([...map1, ...map2]);
 
console.log(merged.get(1)); // 2
console.log(merged.get(2)); // 3

javascript
const map1 = new Map([[1,3], [2,3]]);
 
const map2 = new Map([[1,2]]);
 
const merged = new Map([...map1, ...map2]);
 
console.log(merged.get(1)); // 2
console.log(merged.get(2)); // 3

However, in our example app, all the author id-name pairs are unique, so we don't need to worry about accidentally overriding any data.

Now we can combine everything into the final version of the component.

jsx
const App = () => {
  const books = [
    {
      id: 1,
      name: "In Search of Lost Time ",
      author: { name: "Marcel Proust", id: 1 }
    },
    { id: 2, name: "Ulysses", author: { name: "James Joyce", id: 2 } },
    {
      id: 3,
      name: "Don Quixote",
      author: { name: "Miguel de Cervantes", id: 3 }
    },
    { id: 4, name: "Hamlet", author: { name: "William Shakespeare", id: 4 } },
    {
      id: 5,
      name: "Romeo and Juliet",
      author: { name: "William Shakespeare", id: 4 }
    },
    { id: 6, name: "Dubliners", author: { name: "James Joyce", id: 2 } }
  ];
  const [selectedAuthorId, setSelectedAuthorId] = useState(null);
 
  const authorOptions = new Map([
    ...books.map(book => [book.author.id, book.author.name])
  ]);
 
  const filteredBooks = () => {
    if (!selectedAuthorId) {
      return books;
    }
    return books.filter(book => String(book.author.id) === selectedAuthorId);
  };
 
  return (
    <div className="books">
      <select
        className="books__select"
        onChange={({ target }) => setSelectedAuthorId(target.value)}
      >
         <option value=''>--Select author--</option>
        {[...authorOptions].map(([id, name]) => (
          <option value={id}>{name}</option>
        ))}
      </select>
      <ul className="books__list">
        {filteredBooks().map(book => (
          <li className="books__item">
            {book.name} by {book.author.name}
          </li>
        ))}
      </ul>
    </div>
  );
};

jsx
const App = () => {
  const books = [
    {
      id: 1,
      name: "In Search of Lost Time ",
      author: { name: "Marcel Proust", id: 1 }
    },
    { id: 2, name: "Ulysses", author: { name: "James Joyce", id: 2 } },
    {
      id: 3,
      name: "Don Quixote",
      author: { name: "Miguel de Cervantes", id: 3 }
    },
    { id: 4, name: "Hamlet", author: { name: "William Shakespeare", id: 4 } },
    {
      id: 5,
      name: "Romeo and Juliet",
      author: { name: "William Shakespeare", id: 4 }
    },
    { id: 6, name: "Dubliners", author: { name: "James Joyce", id: 2 } }
  ];
  const [selectedAuthorId, setSelectedAuthorId] = useState(null);
 
  const authorOptions = new Map([
    ...books.map(book => [book.author.id, book.author.name])
  ]);
 
  const filteredBooks = () => {
    if (!selectedAuthorId) {
      return books;
    }
    return books.filter(book => String(book.author.id) === selectedAuthorId);
  };
 
  return (
    <div className="books">
      <select
        className="books__select"
        onChange={({ target }) => setSelectedAuthorId(target.value)}
      >
         <option value=''>--Select author--</option>
        {[...authorOptions].map(([id, name]) => (
          <option value={id}>{name}</option>
        ))}
      </select>
      <ul className="books__list">
        {filteredBooks().map(book => (
          <li className="books__item">
            {book.name} by {book.author.name}
          </li>
        ))}
      </ul>
    </div>
  );
};

If you are curious about other ways Map can help you optimize your code, I've written a separate article: Simplifying Code with Maps In JavaScript and React.

Conclusion

In conclusion, removing duplicate values from an array is a common task in JavaScript development, and using the Set constructor is a popular way to achieve this. However, it falls short when dealing with arrays of objects or arrays of arrays. In this post, we explored an alternative approach to remove duplicates from an array by using the Map data structure. By harnessing the power of Map's key uniqueness, we have demonstrated a real-world use case in a React app, where we efficiently collected unique values from an array of objects.

Using the Map approach allows for preserving only the unique key-value pairs, which is not possible with the Set approach when dealing with arrays of non-primitive values. This method is particularly useful when working with arrays of objects, and when we need to filter out duplicates based on a particular key value.

We also explored other techniques for removing duplicates, such as using the filter in combination with the indexOf or findIndex methods, but these can be less efficient and more verbose compared to using Map.

Overall, whether you are dealing with arrays of primitives, objects, or arrays, it's good to know that there are multiple ways to remove duplicates and that choosing the right approach depends on your specific use case.

Removing Duplicates with Map In JavaScript

Removing duplicates from an array with Set

Using Map to remove duplicates from an array of objects

Conclusion

References and resources

Related posts