DataWeave Delight: Unzipping Arrays

In a previous blog post we learned about the zip function. DataWeave also gives us the unzip function that will do the opposite for an array with arrays. The input argument of the unzip function is an array where the elements are also arrays. This could be created by the zip function or just defined as data structure directly. The unzip function will take from each array the same index element and return it as an array with the index elements. For example with the input array [[1, "A"], [2, "B"]] will be unzipped to [[1, 2], ["A", "B"]]. When the number of elements in the arrays that need to unzipped are not equal, the unzip function will only return the elements from the index with the most elements.

In the following example we use the unzip function with different input arrays:

Source

%dw 2.0

var fruitPrices = [["Apple", 2.30], ["Pear", 1.82], ["Banana", 2.06]]

var fruitPricesIncomplete = [["Apple", 2.30], ["Pear", 1.82], [2.06]]

output application/json
---
{
    // unzip will break up each array into separate arrays.
    unzip: unzip(fruitPrices),

    // When the arrays to break up don't have the same 
    // number of elemnts unzip can only return elements
    // from the index that has the most elements.
    unzipIncomplete: unzip(fruitPricesIncomplete),

    // When the arrays to unzip have more than 2 elements
    // the results will be the number of arrays equal to the number of elements.
    // In this case we get 3 arrays as a result as the array to
    // unzip has 3 elements.
    unzipMoreElements: unzip([[1, "a", "A"], [2, "b", "B"]])
}

Output

{
  "unzip": [
    [
      "Apple",
      "Pear",
      "Banana"
    ],
    [
      2.30,
      1.82,
      2.06
    ]
  ],
  "unzipIncomplete": [
    [
      "Apple",
      "Pear",
      2.06
    ]
  ],
  "unzipMoreElements": [
    [
      1,
      2
    ],
    [
      "a",
      "b"
    ],
    [
      "A",
      "B"
    ]
  ]
}

Written with DataWeave 2.4.

DataWeave Delight: Zipping Arrays

DataWeave has a zip function in the dw::Core module. The function will merge two arrays into a new array. Each element in the new array is also an array and will have a value from the two original arrays from the same index grouped together. So for example we have an input array ["A", "B"] and another input array [1, 2]. The result of the zip function will be [["A", 1], ["B", 2]]. The size of the resulting array is the same as the minimal size of both input arrays. Any value from an array that cannot be merged is simply ignored and left out of the resulting array.

In the following code example we use the zip function for different arrays:

Source

%dw 2.0

import take from dw::core::Arrays

var fruit = ["Apple", "Pear", "Banana"]
var prices = [2.30, 1.82, 2.06]

output application/json
---
{
    // Create new array where each element is
    // an array with first element a fruit and
    // second element a price value. 
    zip: fruit zip prices,

    // When we have an array that contains arrays
    // of 2 items (a pair) we can easily turn it into an object
    // with key/value pairs using reduce.
    zipObj: fruit zip prices 
        reduce ((item, acc = {}) -> acc ++ {(item[0]): item[1]}),

    // The resulting array will have no more items
    // then the smallest array that is used with the zip function.
    // In the following example the second array only has 2 items
    // so the resulting array also has 2 items in total. 
    // The fruit "Banana" is now ignored.
    zipMinimal: fruit zip (prices take 2)
}

Source

{
  "zip": [
    [
      "Apple",
      2.30
    ],
    [
      "Pear",
      1.82
    ],
    [
      "Banana",
      2.06
    ]
  ],
  "zipObj": {
    "Apple": 2.30,
    "Pear": 1.82,
    "Banana": 2.06
  },
  "zipMinimal": [
    [
      "Apple",
      2.30
    ],
    [
      "Pear",
      1.82
    ]
  ]
}

Written with DataWeave 2.4.

DataWeave Delight: Partition An Array

In DataWeave we can partition the items in an array using a predicate function by using the partition function from the dw::core::Arrays module. The function takes an array as first argument and a predicate function as second argument. The predicate function should return true or false for each item of the array. The result is an object with the key success containing all items from the array that returned true for the predicate function and a key failure for the items that returned false.

In the following example code we use the partition function on an array:

Source

%dw 2.0

import partition from dw::core::Arrays

var items = ["language", "DataWeave", "username", "mrhaki", "age", 48]

output application/json 
---
{ 
    // Partition by item is of type String or not.
    example1: items partition ((item) -> typeOf(item) == String),

    // Partition by checking if item have value 1 or 4 or not
    // using shorthand notation.
    example2: (0 to 5) partition ([1, 4] contains $)
}

Output

{
  "example1": {
    "success": [
      "language",
      "DataWeave",
      "username",
      "mrhaki",
      "age"
    ],
    "failure": [
      48
    ]
  },
  "example2": {
    "success": [
      1,
      4
    ],
    "failure": [
      0,
      2,
      3,
      5
    ]
  }
}

Written with DataWeave 2.4.

DataWeave Delight: Splitting An Array Or Object

The module dw::core::Arrays has extra functions that are useful when working with arrays in DataWeave. In this post we will look at the functions splitAt and splitWhere. We can use these functions to split an array into two arrays. The result of both functions is actually a Pair type, which is defined as an object with the keys l and r. An example is { "l": 1, "r": 2 } which we can read as the left side of the pair has value 1 and the right side of the pair has value 2. The result of the splitAt and splitWhere function will have one part of the split array in the left side of the pair and the rest of the array in the right side.

The splitAt function takes the array as first argument and an index value as second argument. The index value should indicate at which position the array should be split. The function can be used with infix notation as well. The splitWhere function takes as first argument also an array, but the second argument is a predicate function. All items starting from the first item for which the predicate function returns true will be assigned to the r key of the Pair result, and all preceding items to the l key. We can use the infix notation here as well.

In the following example code we use the splitAt and splitWhere function on an array:

Source

%dw 2.0

import splitWhere, splitAt from dw::core::Arrays

output application/json
---
{
    // Split the range at index 4. 
    splitAt: (0 to 8) splitAt 4,

    // Split at the position where the predicate returns true for the first time.
    splitWhere: (0 to 8) splitWhere ((item) -> item > 5)
}

Output

{
  "splitAt": {
    "l": [
      0,
      1,
      2,
      3
    ],
    "r": [
      4,
      5,
      6,
      7,
      8
    ]
  },
  "splitWhere": {
    "l": [
      0,
      1,
      2,
      3,
      4,
      5
    ],
    "r": [
      6,
      7,
      8
    ]
  }
}

Although the function work on arrays we can use them on a object as well. We first turn the object into an array of objects with a single key and value using the pluck function. We can use this array with the splitAt and splitWhere functions. Then we can use the reduce function to transform the values in the Pair to an object with multiple keys and values again.

In the next example we use this mechanism on an object:

Source

%dw 2.0

import splitWhere, splitAt from dw::core::Arrays

// Helper functions
// ----------------
// Transform object to array with key/value pairs
fun objectToArray(obj: Object): Array<Object> = obj pluck ((value, key) -> (key): value)

// Transform array with key/value pairs to object
fun arrayToObject(items: Array<Object>): Object = items reduce ((item, acc = {}) -> acc ++ item)

var obj = {
    language: "DataWeave",
    alias: "mrhaki",
    age: 48, 
    country: "NL"
}

output application/json 
---
{
    // We can use splitAt on object if we first transform an object
    // to an array of key/value pairs, apply the splitAt function and 
    // transform the resulting Pair into an object again.
    splitAtObject: objectToArray(obj) 
        splitAt 3 
        mapObject ((value, key) -> (key): arrayToObject(value)),

    // We can use splitWhere in the same manner for objects.
    splitWhereObject: objectToArray(obj) 
        splitWhere ((item) -> typeOf(item[0]) == Number)
        mapObject ((value, key) -> (key): arrayToObject(value))
}

Output

{
  "splitAtObject": {
    "l": {
      "language": "DataWeave",
      "alias": "mrhaki",
      "age": 48
    },
    "r": {
      "country": "NL"
    }
  },
  "splitWhereObject": {
    "l": {
      "language": "DataWeave",
      "alias": "mrhaki"
    },
    "r": {
      "age": 48,
      "country": "NL"
    }
  }
}

Written with DataWeave 2.4.

DataWeave Delight: Taking or dropping items from an array

The module dw::core::Array has several functions to create a new array by taking or dropping a number of elements of the source array. With the take function has as first input argument the original array and the second is a number indicating the number of items to take. For example [1, 2] take 1 will return [1]. Notice that we can use the functions also in the infix notation. The drop function works similarly, but will skip the number of items we define as second argument to the function. So [1, 2] drop 1 will return [2].

In the following example we use the take and drop function with several numbers to use or skip items from the original array:

Source

%dw 2.0

import take from dw::core::Arrays

var list = ['Simple', 'list', 'with', 5, 'items']

output application/json
---
{
    // Return array with first element "Simple"
    "take 1 item": list take 1,

    // Return array with first 2 elements.
    "take 2 items": list take 2,

    // Empty list returned as we take 0 items.
    "take 0 items": list take 0,

    // Return array with all elements as 6 is larger 
    // than number of items in the list.
    "take 6 items": list take 6
}

Output

{
  "take 1 item": [
    "Simple"
  ],
  "take 2 items": [
    "Simple",
    "list"
  ],
  "take 0 items": [
    
  ],
  "take 6 items": [
    "Simple",
    "list",
    "with",
    5,
    "items"
  ]
}

Source

%dw 2.0

import drop from dw::core::Arrays

var list = ['Simple', 'list', 'with', 5, 'items']

output application/json
---
{
    // Return array where first element of original 
    // array is skipped.
    "drop 1 item": list drop 1,

    // Return array with elements from index 3.
    "drop 3 items": list drop 3,

    // Empty array returned as we drop all items.
    "drop 5 items": list drop 5,

    // Return empty array as 6 is larger than
    // number of items in the list.
    "drop 6 items": list drop 6,

    // Range as array of numbers.
    "drop from range": (0 to 10) drop 8
}

Output

{
  "drop 1 item": [
    "list",
    "with",
    5,
    "items"
  ],
  "drop 3 items": [
    5,
    "items"
  ],
  "drop 5 items": [
    
  ],
  "drop 6 items": [
    
  ],
  "drop from range": [
    8,
    9,
    10
  ]
}

We can also specify a condition using a predicate function to determine how many items to take or drop. In that case we use the takeWhile and dropWhile functions. Instead of specifying a number of items we use predicate function. The predicate function should return true or false. The function is applied to the items of the original array and as long as the return value is true it will be included in the resulting array with takeWhile or skipped with dropWhile.

In the next example we use both functions:

Source

%dw 2.0

import takeWhile, dropWhile from dw::core::Arrays

var list = (0 to 10)

var object = {
    name: "mrhaki", 
    livesIn: "Tilburg", 
    worksAt: 'JDriven'
}

// Helper function to get key from object
// with only 1 key/value pair.
fun getKey(obj: Object): Key = keysOf(obj)[0]

output application/json
---
{
    // Return array with first items smaller than 6.
    take: list takeWhile ((item) -> item < 6),

    // Once a false is returned for the condition
    // no more items are processed.
    // This time we use shorthand notation for the function argument.
    irregularTake: [1, 5, 7, 2, 4, 6] takeWhile $ < 6,

    // Return array where first items smaller than 6 are skipped.
    drop: list dropWhile ((item) -> item < 6),

    // With a bit of transformation we can also use the 
    // functions on objects. 
    // First we transform the object into an array, 
    // apply the function and transform to object again.
    objectTake: object
        pluck ((value, key) -> (key): value)
        takeWhile ((item) -> sizeOf(getKey(item)) == 4)
        reduce ((result, acc = {}) -> acc ++ result),

    objectDrop: object
        pluck ((value, key) -> (key): value)
        dropWhile ((item) -> getKey(item) ~= "name")
        reduce ((result, acc = {}) -> acc ++ result)        
}

Output

{
  "take": [
    0,
    1,
    2,
    3,
    4,
    5
  ],
  "irregularTake": [
    1,
    5
  ],
  "drop": [
    6,
    7,
    8,
    9,
    10
  ],
  "objectTake": {
    "name": "mrhaki"
  },
  "objectDrop": {
    "livesIn": "Tilburg",
    "worksAt": "JDriven"
  }
}

Written with DataWeave 2.4.

DataWeave Delight: Reversing the items in an array

To reverse the order of the items in the array we have two possible solutions in DataWeave. The first solution uses the index operator to access items in an array ([...]). We can define a range as value for the index operator to get all elements that have a index value that is part of the range. For example we could use [1 to 3] to get the second up to the fourth items from the array. The value -1 can be used to indicate the last index in the array, so we could get a new array where the items are in reversed order by using the range [-1 to 0].

The second option is to use the orderBy function for an array from the dw::Core module. The function takes as first argument an array and the second argument is a criteria function that is applied for each element. The criteria function will get the array element value and index. The output of the function is used for the ordering of the resulting array. To reverse the order we can negate the index value of the criteria function and we get a new array in reverse order: ((item, index) -> -index).

In the following example we use both ways to reverse the order of an array:

Source

%dw 2.0

var list = [2, 1, 4, 5, 3]

output application/json
---
{
    // Using index operator.
    reverseIndices: list[-1 to 0],

    // Using orderBy with criteria function.
    reverseOrderBy: list orderBy ((item, index) -> -index),

    // Using orderBy with shorthand notation for criteria function.
    reverseOrderByShorthand: list orderBy -$$
}

Output

{
  "reverseOrderBy": [
    3,
    5,
    4,
    1,
    2
  ],
  "reverseIndices": [
    3,
    5,
    4,
    1,
    2
  ],
  "reverseOrderByShorthand": [
    3,
    5,
    4,
    1,
    2
  ]
}

Written with DataWeave 2.4.

DataWeave Delight: Removing duplicates from an array

If we want to remove duplicate items from an array in DataWeave we can use the distinctBy function from the dw::Core module. The first argument is an array and the second argument is a criteria function. The criteria function gets an item of the array and the index value of that item in the array. The return value of the function defines the uniqueness of that item. This can simply be the item value itself or something more complex like a calculation or selector.

In the following example we use several criteria functions to remove duplicate items from an array:

Source

%dw 2.0

var items = [
    { id: "DW", name: "DataWeave" },
    { id: "CLJ", name: "Clojure" },
    { id: "DW", name: "DataWeave" }
]
var s = "aabccdeff"

output application/json 
---
{ 
    // An array with numbers where we want to remove duplicates.
    numbers1: [1, 5, 5, 6, 2, 3, 3, 1] distinctBy ((item) -> item),

    // We can also use a bit more complex function to define uniqueness.
    numbers2: [1, -2, 2, 3, -1] distinctBy (item) -> abs(item),

    // We can use shorthand notation $ and
    // distinctBy also works on objects in the array.
    items1: items distinctBy $,

    // Instead of doing a distinctBy on the whole object
    // we can also use matching function and use properties
    // from the object to define unique objects.
    items2: items distinctBy ((item, index) -> item.id),

    // With the help of toArray and joinBy we can remove duplicates
    // from a string value.
    string: dw::util::Coercions::toArray(s) distinctBy $ joinBy ""
}

Output

{
  "numbers1": [
    1,
    5,
    6,
    2,
    3
  ],
  "numbers2": [
    1,
    -2,
    3
  ],
  "items1": [
    {
      "id": "DW",
      "name": "DataWeave"
    },
    {
      "id": "CLJ",
      "name": "Clojure"
    }
  ],
  "items2": [
    {
      "id": "DW",
      "name": "DataWeave"
    },
    {
      "id": "CLJ",
      "name": "Clojure"
    }
  ],
  "string": "abcdef"
}

Written with DataWeave 2.4.

DataWeave Delight: Counting items in array with predicate

The dw::core::Arrays module has some nice functions that works with arrays. One of these functions is countBy. The function takes an array as first argument and a predicate function that accepts an item from the array and should return true or false. The total number of items that adhere to the predicate is the result of the countBy function.

In the following examples we use the countBy function with different predicates:

Source

%dw 2.0

import countBy from dw::core::Arrays

var languages = ['DataWeave', 'Groovy', 'Java', 'Clojure']

output application/json
---
{ 
    // Count items that have more than 6 characters.
    sample1: languages countBy ((item) -> sizeOf(item) > 6), 

    // Count number of even numbers.
    sample2: 1 to 10 countBy (number) -> isEven(number),

    // Using shorthand notation $ to count upper case characters.
    sample3: ["AB", "cD", "Ef", "GH"] countBy dw::core::Strings::isUpperCase($)
}

Output

{
  "sample1": 2,
  "sample2": 5,
  "sample3": 2
}

Written with DataWeave 2.4.

DataWeave Delight: Turn string value into array of characters

A string value can be seen as an array of characters and if we want to transform our string value to an array we can use the toArray function in the dw::util::Coercions module. Once we have transformed our string to an array we can use all functions that work on arrays. The nice thing about DataWeave is that some functions that work on arrays already have an overloaded version that accepts a string value. Then we don't have to explicitly use the toArray function, but we can simply use our original value when we invoke the function.

In the following example we use the toArray function and also see the groupBy function that already accepts a string value and treats it is as an array:

Source

%dw 2.0

import toArray from dw::util::Coercions
import isUpperCase from dw::core::Strings

output application/json
---
{
    // toArray will convert a string to an array of characters
    toArray: toArray("DataWave"),
    
    // joinBy function only accepts an array argument
    toArrayJoinBy: toArray("abcd") joinBy "-",

    // groupBy function has been overloaded to accept string,
    // but internally makes it an array of characters
    groupByString: "AbCd" groupBy ((character, index) -> isUpperCase(character))
}

Output

{
  "toArray": [
    "D",
    "a",
    "t",
    "a",
    "W",
    "e",
    "a",
    "v",
    "e"
  ]
  "toArrayJoinBy": "a-b-c-d",
  "groupByString": {
    "true": "AC",
    "false": "bd"
  }
}

Written with DataWeave 2.4.

DataWeave Delight: Using pluck function to transform object to array

Sometimes we get a data structure with keys and values and we want to create an array with data from the data structure. DataWeave gives us the pluck function to achieve this. The input argument is an object and a lambda and the output must be an array. We use a lambda to define the mapping from our object to elements in the resulting array. The lambda has three arguments: the value for a key, the key itself and the index of the key in the object. We have a handle on all data to pluck the information and transform it to elements in an array.

In the following examples we have object structure with keys book1 and book2 that we want to transform to an array. We use the infix notation of the function so our input object is followed by the pluck function and lambda:

Source

%dw 2.0

// Object structure has already an array like structure,
// but uses keys (book1, book2) for each book object.
// Using pluck we can turn this object into an array.
var books = {
    book1: {
        title: "Lord of the rings",
        price: 30.0
    },
    book2: {
        title: "Hitch-hiker's guide to the galaxy",
        price: 42.0
    }
}

output application/json
---
{ 
    // Transform object books to array with object 
    // containing fields with index, key and book details
    booksArray: books pluck ((book, key, index) -> {position: index, key: key} ++ book),

    // We can easily find all titles from the object using pluck.
    // We cannot use books.*title as books is an object not an array.
    // Alternatives to get the same result is to use valuesOf function,
    // to get all book objects and then map to get the title:
    // * valuesOf(books) map (book) -> book.title
    // * valuesOf(books).title
    bookTitles: books pluck (book) -> book.title,

    // The pluck function also allows the dollar-sign to reference the value.
    // $$ is the key and $$$ the index.
    bookPrices: books pluck $.price
}

Output

{
  "booksArray": [
    {
      "position": 0,
      "key": "book1",
      "title": "Lord of the rings",
      "price": 30.0
    },
    {
      "position": 1,
      "key": "book2",
      "title": "Hitch-hiker's guide to the galaxy",
      "price": 42.0
    }
  ],
  "bookTitles": [
    "Lord of the rings",
    "Hitch-hiker's guide to the galaxy"
  ],
  "bookPrices": [
    30.0,
    42.0
  ]

In the next example we add an if/else to the lambda:

Source

%dw 2.0

var messages = {
  "message0": {
    "text": "Hi, mrhaki"
  },
  "message1": {
    "text": "Good evening, Hubert"
  },
  "message3": {
    "text": "Hello, DataWeave"
  },
  "created_with": "DataWeave 2.4"
}

output application/json
---
messages pluck ((value, key, index) -> if (key ~= "created_with") value else value.text)

Output

[
  "Hi, mrhaki",
  "Good evening, Hubert",
  "Hello, DataWeave",
  "DataWeave 2.4"
]

Written with DataWeave 2.4.

DataWeave Delight: Join array elements into a string

DataWeave has a stanard function to join array elements into a string value: joinBy in the dw::Core module. The joinBy function takes an array as first argument and a string value that is used as separator. The array should have string values or values that can be coerced into string values.

In the following example we use the function joinBy that will join elements in an array with a given separator:

Source

%dw 2.0

output application/json  
---
{
  // Join with default separator ("") using infix notation
  join: ["a", "b", "c"] joinBy "",

  // Join with ", " as separator
  separator: joinBy(["a", "b", "c"], ", "),
  
  // Elements can also be non-string values
  // if they can be coerced into string values
  range: [1, 2, 3, 4] joinBy "|"
}

Output

{
  "join": "abc",
  "separator": "a, b, c",
  "range": "1|2|3|4"
}  

Written with DataWeave 2.4.

DataWeave Delight: Finding highest or lowest value in an array

DataWeave has several function to find the highest or lowest value in an array. The items in the array need to be of the same type and implement the Comparable interface. To find the highest value we can use the max function. The function returns the maximum value based on the type of the item in the array. If we want to find the highest value based on other criteria we can use the maxBy function. This function takes a lambda as last argument where we can return a Comparable value that is used to determine the highest value. To find the lowest value in an array we have similar functions: min and minBy.

In the following example we use several ways to find the highest and lowest value in arrays with items of different types:

Source

%dw 2.0

// Array with numbers
var numbers = [10, 2, 3, 1, 20]

// Array with string values
var languages = ["DataWeave", "Java", "Groovy", "Clojure"]

// Array with objects representing a user
var users = [
	{ name: "mrhaki", age: 47 },
	{ name: "alice",  age: 32 },
	{ name: "bob",    age: 61 }
]

output application/json
---
{
	// Find maximum for numbers
	maxNumbers: max(numbers),
	
	// Find minimum for numbers
	minNumbers: min(numbers),
	
	// Find max for srings looks at first character
	maxLanguages: max(languages),
	
	// Find min for strings looks at first character
	minLanguages: min(languages),
	
	// Find language with max length
	maxByLength1: maxBy(languages, (lang) -> sizeOf(lang)),
	
	// Find max with infix notation
	maxByLength2: languages maxBy (lang) -> sizeOf(lang),
	
	// Find max with infix notation and $ to reference lambda argument
	maxByLength3: languages maxBy sizeOf($),
	
	// Find language with minimal length
	minByLength: languages minBy ((lang) -> sizeOf(lang)),
	
	// Find oldest user with maxBy on objects
	oldestUser: users maxBy ((user) -> user.age),
	
	// Find youngest user with minBy on objects
	youngestUser: users minBy (user) -> user.age
}

Output

{
  "maxNumbers": 20,
  "minNumbers": 1,
  "maxLanguages": "Java",
  "minLanguages": "Clojure",
  "maxByLength1": "DataWeave",
  "maxByLength2": "DataWeave",
  "maxByLength3": "DataWeave",
  "minByLength": "Java",
  "oldestUser": {
    "name": "bob",
    "age": 61
  },
  "youngestUser": {
    "name": "alice",
    "age": 32
  }
}

Written with DataWeave 2.4.