oAuth2 connection - Cornerstone(CSOD)

[Rykie]

Hi Team

I need some advice on getting my API to work.

The following:

1. I completed the OAuth details
2. Go to this
   
3. Received: The connector has been authorised
4. When running the base URL with a test
5. I get
   

What else could I check?

Further question: Does the redirect URL matter - I used: http://localhost:12345
Thanks

[Rykie]

Or do I have to follow this:
How to perform OAuth Client Credentials Flow - Articles - EasyMorph Community - Data preparation professionals and enthusiasts

[olysak]

Hi @Rykie ! That mostly depends on a service you’re trying to use. What is this service? But if you get the ‘Connector has been authorized’ message then OAuth part is working correctly, and issue may be in your not having required credentials to access specific endpoint or even misspelling the endpoint name etc

[Rykie]

Hi @olysak. Thanks for your quick answer. I can get this to work using the example in the article above. After getting the token, I then use iterate and push the token to another web request. It extracts a maximum of 10,000 rows and generates a column: next link. How do I feed the next link into another web request?

This works:

1. Iterate through module1
2. Get the 1st 10000 rows
3. Filter on Body
4. Parse JSON
5. Derive table
6. Delete all columns, but Next Link, deduplicate (1 row remain)
7. Peak Token to table
8. Iterate through module2
9. Repeat the process -
10. At the end append all results

There must be a better way - ? Repeat iterate, but the link field only become available at the end of iterate of each round

[olysak]

Yes, basically you need to repeat the call, first time using your initial endpoint URL and subsequent times using 'next link' returned to you by the API, repeat until you either get no data or no 'next link'. This is rather common scenario, 'pagination', API providers use it to even-out the load from large requests.

We have an article about how to deal with such cases

How to pull data from web APIs with pagination

The “Repeat” action introduced in v4.5 can be used to pull data from web APIs that require pagination. Such APIs don’t return all the requested data in one response, but instead return a limited set (page) of records (e.g. 10 records per page) and require specifying a page in requests (e.g. p=2). The calling application must sequentially pull page by page from such API until it pulls all the required data. Unfortunately, pagination is not specified in the web standards. Therefore, in real life …

[Rykie]

Thanks, @olysak

I am trying to get a combination of pushing the token and calling the next page with a Link.
Will let you know how it goes.

[Rykie]

@olysak is it possible to get an example where nexLink is used instead of page numbers please? I cannot get the Query API to work using “Repeat” and pushing token. Thx

[olysak]

Please, see here:

How to query Salesforce using EasyMorph 4.6

While a dedicated Salesforce-related action and connector are already on our to-do list and planned for release in v5 later this year, you can already start querying Salesforce using existing EasyMorph tools! In this article, I will explain how to create and configure a connection to Salesforce. Also, a working sample project is attached. Creating Connected App in Salesforce First, you need to create a new Connected App. Note that using Web API in Salesforce requires you to have an Enterprise …

There is a sample project attached to the opening post that does just this. There is REPEAT action in the Query Salesforce module that repeatedly calls Sf Query module until its output is empty.

This Sf Query module expects input dataset to have 'URI' column which it uses as an actual URL to query. And in this Sf Query module after the call is being done the response is parsed and there is a conditional that if response had nextRecordsUrl column then original URI column should be replaced with it:

In effect, the first time REPEAT action calls Sf Query module with URI column containing the initial URL and on the second, third and so forth Sf Query is called with URI column containing nextRecordsUrl from the previous iteration

[Rykie]

Thank you @olysak. I am close - not working 100% yet. I will get back to you.

[Rykie]

@olysak -

I think I miss one part - do not understand this:
In the example I saw: [URL] = /?q.

The following: Main
I have a token and Path I need to push to the Sf Query

My starting Path will always be: CheckInParticipant

On SFQuery

I do not understand the Skip action condition, as I will always have a path:

My Iterate WEb request on Sf Query Raw looks like:

Path - from the Main and Authorisation from the main
I tried add URL parameters - and think that is where I should add the nextpage, but could note get it to work

The last action: Modify Columns on Sf Query Raw
Do I remove everything from the Path before CheckInParticipant or keep CheckInParticipant i.e. https://xxx/api/Objects/CheckInParticipant?$skiptoken=eyJjIjpmYWxzZSwicyI6W3sibiI6ImNoZWNrSW5fcGFydGljaXBhbnRJZCIsIn

Thanks

[olysak]

Hi @Rykie

The exact details of how your skiptoken should be utilized depends on API you’re trying to use. Sometimes this token should be appended to the URL, like first if request was

https://something/endpoint?q=some

and that request returned skiptoken in the body, like

{
    ....
    'skiptoken': 'abcdefg123'
}

then next request should be:
https://something/endpoint?q=some&token=abcdefg123, meaning everything stays the same except one more URL parameter appended with the skiptoken value.

Sometimes, pagination is implemented in the form of next page URL being completely replaced with nextPageUrl token from the prevoius step – that’s the case with Salesforce example. In this case, for example, if your first request was

https://something/endpoint?q=some

and that request returned for example nextPage in the body, like

{
    ....
    'nextPage': 'https://whatever/whatever?whatever=1234'
}

then next request should be:
https://whatever/whatever?whatever=1234, meaning you don’t modify the original URL but blindly use next page URL that was given to you by the API.

The exact case depends on what says the documentation to the API you’re trying to use.

One more thing, about the REPEAT action - the key concept that may be causing the misunderstanding is how the iteration is implemented in EasyMorph. The first time a module is being called by REPEAT action it gets input dataset that equals to the dataset you push to REPEAT action. But the second, third and so on times the input dataset of the called module is the output of the last iteration.

See
https://easymorph.com/learn/iterations.html
the Repeat action section.

So, you should determine:

1. How exactly the URL formed by Iterate Web Request should look during the first call with initial input dataset? How should I make it look the way API expects it to be?
2. How exactly the next page token/url is returned after the call?
3. How do I make the output of the first iteration look so that, when fed to the p.1 instead of ‘input dataset’, the resulting URL for the Iterate Web Request should point to the query for the second page.

The iterations and their implementation in EasyMorph can be somewhat tricky ‍

[Rykie]

Thank you @olysak.

I worked out what the 1st page and next page URLs should look like.
(Using OData server driven paging protocol.)

What I am stuck with now, is authorisation. On my 1st call, the repeat pushes the acces-token as part of the dataset. How do I keep that token as I need it for iterations 2,3…
On my 2nd iteration, I get the error:

[olysak]

@Rykie

On my 2nd iteration, I get the error:

Make sure that your Sf Query Raw module has access_token column in its output and that column contains access token. Remember that first iteration's output is second iteration input and if the module requires access_token column to be present in the data, it always must be present.

Additionally,

If I understand correctly that you're manually utilizing the acces_token and this token doesn't change from page to page, it can be more convenient to use module parameters to pass this token.

Go to the module you're trying to call repeatedly and to the parameters section. Add an access_token parameter there:

In your REPEAT action there is section Assign parameters of the project/module using these columns... pick your newly created parameter in the module and assign it's value using 'first value of column' option, pointing it to the column in the input dataset that contains your access token:

Then, in the called module you can pick access_token for your Iterate Web request action from the parameter instead of dragging it through your datasets:

[Rykie]

Thanks, @Olysak.

I am still missing a part. I know about the parameter - but can’t get that to work either.
I am not sure if I need another subquery.

Currently:

1. Main Query
   

2. I use this to move to SF Query Raw
   

3. On SF Query Raw
   

4. Iterate Web request
   

If I use the Parameter, comes with format value is invalid as I have the following setup in my SF Query Raw
- probably as the value does not stay.

In your response, you indicated I should keep a column Token in my SF Query Raw - but I do not know how to keep it as with iterate web, it is no longer available.

When I use the column option in value for Authorisation, I get the 1st value for nextlink.

Should I have a 3rd module:
Currently I have main (with repeat) that push data to SF Query Raw that use Iterate Web request and it gives me the 1st NextLink as I cannot get Authorisation back in the SF Query Raw.

Sorry - so close, but so far…

[olysak]

@Rykie

In your response, you indicated I should keep a column Token in my SF Query Raw - but I do not know how to keep it as with iterate web, it is no longer available.

When I use the column option in value for Authorisation, I get the 1st value for nextlink.

I suppose you could keep this token as a parameter, since it doesn't change from iteration to iteration.

@Rykie

If I use the Parameter, comes with format value is invalid as I have the following setup in my SF Query

Could you please clarify what do you mean 'comes with format value is invalid'?

You should:

1. Create parameter 'access_token' in your sumbodule
2. In your repeat action settings, set access_token parameter to equal what you need to be passed to the API, input column with access_token that's fed to Repeat action should probably look like Bearer d85f22a8fefc254a256478b80903fac416499ba13ac7914978dc573bdc8d7f77
3. In your iterated module, pass this parameter to Web Request action as a Authorization header.

As far as I see in the screenshot, you do both 1 and 2 but in the next screenshot as far as I see you keep using access_token from the dataset, not from the parameter

[Rykie]

Thanks, Olysak.

I did all of that, but this is the error:

Please note my Base URL data Exporter has no authentication and I need to add the Access token generated in a previous step - and yes - it does come with Bearer …

So, I 1st generate the access token, pass it via Repeat to sub module and in Iterate web Request when I then use as value the parameter, I get a format of value ‘’ is invalid - I am sure it is on the 2nd run, where the value is not available anymore.

[Rykie]

Big Thank you @olysak for getting this to work.

I appreciate your support.

R

[olysak]

You’re welcome!