Skip to main content

I’m having some issues in regards as to how to approach handling pagination when creating a channel and accompanying datamappers.

The issue I’m trying to solve is trying to set up a channel that queries a MDM device portal and is to act as a sort of CMDB gateway for our phones. However, the MDM portal (Workspace ONE UEM API) pagination is hard set to a max value of 500, and I need to check for updates on a total of ~ 12000 devices. 

The version of the API I‘m currently integrating with only has GET requests available, and using the regular HTTPS channel and datamapper I’m unsure of how to iterate through the pages of the  response. The endpoint URL is https://….net/API/mdm/devices/search
The page number is returned in the last section of the JSON response:

…
"page": 1,
"page_size":500,
"total": 10
}

However, I’m usure of how to use this part of the response to get the next page when using a HTTPS channel. 
 

To get the phones created as items I’ve performed a manual workaround by updating the channel Endpoint URL to manually step through the pages (https://….net/API/mdm/devices/search?page=0 → https://….net/API/mdm/devices/search?page=1, etc.) But this can hardly be the best method to solve this issue. 


I’ve seen there is an idea proposed, where this would get better supported out of the box:
 

​​​​​​
However, how has this issue been addressed with the current limitations in place?

Is it possible to change to the next page or perform other dynamic operations from the result fetched by a datamapper?

Has this problem been solved using a different approach? 

This isn’t really supported at present. If you can upvote the idea and add an example JSON payload then that will help us when/if we implement this. 

Channel sources don’t have any way to perform a dynamic query.

You might be able to do this by combing 3 channels - 1 to periodically trigger the process; 1 to get the next page of data & then trigger itself if there are more pages; and 1 to actually process the devices.

 

Channel 1

  • Timer Source
  • assyst Message Bus Mapper
    • Body = 0 (the page to get)
    • Queue = MDM Get Page

Channel 2

  • Assyst Destination Source
  • Message Format = Free Text
  • Queue = MDM Get Page
  • HTTP Mapper
  • assyst Message Bus Mapper
    • Queue = MDM Data
    • Body = mappedB0].response
      • send the JSON response to channel 3 to process
  • assyst Message Bus Mapper
    • Queue =MDM Get Page
    • Body = (inbound * 1) + 1
      • add 1 to the inbound value (converting to a number by multiplying by 1)
    • No Value When
      • Some JS logic to decide if there is another page or not. Ideally this doesn’t involve parsing the 500 device response JSON, as that may cause memory issues.

Channel 3

 

  • Assyst Destination Source
  • Message Format = JSON
  • Queue = MDM Data
  • CMDB Gateway style mappers to process each device 
  • Each importsshould consist of <page size> records/

I *think* that would work. Alternatively I’d write a small shell script to grab all of the pages; combine into a single JSON document and then send that to ETM - that’s probably easier.

Reply


Terms & ConditionsCookie settings