GraphQL API endpoint with pagination

[Twitch]

Hi EM Community,

I am hoping to get some advice if anyone has dealt with a similar situation that I am having regarding repeating call requests to a GraphQL endpoint where a cursor string is returned and needs to be fed back into the loop to call the next set of items.

As a preface, I did read the below article on pagination but am still confused on my scenario.

How to pull data from web APIs with pagination

The “Repeat” action introduced in v4.5 can be used to pull data from web APIs that require pagination. Such APIs don’t return all the requested data in one response, but instead return a limited set (page) of records (e.g. 10 records per page) and require specifying a page in requests (e.g. p=2). The calling application must sequentially pull page by page from such API until it pulls all the required data. Unfortunately, pagination is not specified in the web standards. Therefore, in real life …

Here is the documentation for the API I am calling. I don't really have any expectation for anyone to read this but just putting it here anyway.

Migration guide - 2023-10

I am including an image below my use case/problem explanation to show the flow.

Ok... The issue.

The platform of the API I am calling has boards that are essentially act like glorified excel workbooks. I need to query the board I am attempting to poll the records from to get a cursor string to start off a repeat loop of requests until all of the items are returned.

The cursor string from the first graphql query needs to be passed into a second and entirely different query to request the next set of records. I need to repeat this until all the items are returned and exhausted.

If I split into two modules, and have the repeat action in the first to call the second module where the second query resides, I am having trouble figuring out how to pass back the query string that is returned from the module 2 query back into the repeat action in call one to continue the loop until all the records are returned.

Some notes:

1. I have to specific the number of records/items I want returned in the body of my query. If the number of items is greater than the number of items remaining in the board, the cursor string will be empty for every returned record and after these are returned, I need to stop the loop.

2. If the number of items being requested to be returned is less than the count of the remaining items in the board, the same cursor string is returned for every record in the response. I need to take the cursor string value from one of these and pass it back to the module where the repeat is being done to continue the loop.

3. Not sure if relevant, but the length of the cursor string looks like it is always 70 characters.

Hopefully the image I am attaching helps clear up some gaps.

Any ideas anyone has would be sincerely and greatly appreciated!!!

[dgudkov]

Hi Sean,

The main module (1) should do the following:

1. Perform the initial web request (without cursor) and produce 3 columns: cursor, ID, and name.
2. Use the "Repeat" action to iterate module (2)

Design the logic of the iterated module (2) as follows:

1. It should expect in the "Input" action three columns: cursor, ID, and name.
2. If the cursor is empty, then it should just return an empty dataset in the result table
3. If the cursor is not empty, then it should perform a web request (with cursor) to pull the next page from the API and return in the result table three columns: cursor, ID, and name.

The idea is to make "Repeate" do one more loop even when the result of the web request has no cursor but has data. That extra loop will return an empty dataset (because the web request will be skipped) and it will cause the loop to exit (and append results into one table).

I've created a dummy project to illustrate the logic:
cursor-paging.morph (6.1 KB)

PS. Special thanks for the detailed description in the question.