DataWeave Delight: Removing duplicates from an array

If we want to remove duplicate items from an array in DataWeave we can use the distinctBy function from the dw::Core module. The first argument is an array and the second argument is a criteria function. The criteria function gets an item of the array and the index value of that item in the array. The return value of the function defines the uniqueness of that item. This can simply be the item value itself or something more complex like a calculation or selector.

In the following example we use several criteria functions to remove duplicate items from an array:

Source

%dw 2.0

var items = [
    { id: "DW", name: "DataWeave" },
    { id: "CLJ", name: "Clojure" },
    { id: "DW", name: "DataWeave" }
]
var s = "aabccdeff"

output application/json 
---
{ 
    // An array with numbers where we want to remove duplicates.
    numbers1: [1, 5, 5, 6, 2, 3, 3, 1] distinctBy ((item) -> item),

    // We can also use a bit more complex function to define uniqueness.
    numbers2: [1, -2, 2, 3, -1] distinctBy (item) -> abs(item),

    // We can use shorthand notation $ and
    // distinctBy also works on objects in the array.
    items1: items distinctBy $,

    // Instead of doing a distinctBy on the whole object
    // we can also use matching function and use properties
    // from the object to define unique objects.
    items2: items distinctBy ((item, index) -> item.id),

    // With the help of toArray and joinBy we can remove duplicates
    // from a string value.
    string: dw::util::Coercions::toArray(s) distinctBy $ joinBy ""
}

Output

{
  "numbers1": [
    1,
    5,
    6,
    2,
    3
  ],
  "numbers2": [
    1,
    -2,
    3
  ],
  "items1": [
    {
      "id": "DW",
      "name": "DataWeave"
    },
    {
      "id": "CLJ",
      "name": "Clojure"
    }
  ],
  "items2": [
    {
      "id": "DW",
      "name": "DataWeave"
    },
    {
      "id": "CLJ",
      "name": "Clojure"
    }
  ],
  "string": "abcdef"
}

Written with DataWeave 2.4.