Hi Team
I need some advice on getting my API to work.
The following:
What else could I check?
Further question: Does the redirect URL matter - I used: http://localhost:12345
Thanks
Or do I have to follow this:
How to perform OAuth Client Credentials Flow - Articles - EasyMorph Community - Data preparation professionals and enthusiasts
Hi @Rykie ! That mostly depends on a service you’re trying to use. What is this service? But if you get the ‘Connector has been authorized’ message then OAuth part is working correctly, and issue may be in your not having required credentials to access specific endpoint or even misspelling the endpoint name etc
Hi @olysak. Thanks for your quick answer. I can get this to work using the example in the article above. After getting the token, I then use iterate and push the token to another web request. It extracts a maximum of 10,000 rows and generates a column: next link. How do I feed the next link into another web request?
This works:
There must be a better way - ? Repeat iterate, but the link field only become available at the end of iterate of each round
Yes, basically you need to repeat the call, first time using your initial endpoint URL and subsequent times using 'next link' returned to you by the API, repeat until you either get no data or no 'next link'. This is rather common scenario, 'pagination', API providers use it to even-out the load from large requests.
We have an article about how to deal with such cases
Thanks, @olysak
I am trying to get a combination of pushing the token and calling the next page with a Link.
Will let you know how it goes.
@olysak is it possible to get an example where nexLink is used instead of page numbers please? I cannot get the Query API to work using “Repeat” and pushing token. Thx
Please, see here:
There is a sample project attached to the opening post that does just this. There is REPEAT action in the Query Salesforce module that repeatedly calls Sf Query module until its output is empty.
This Sf Query module expects input dataset to have 'URI' column which it uses as an actual URL to query. And in this Sf Query module after the call is being done the response is parsed and there is a conditional that if response had nextRecordsUrl column then original URI column should be replaced with it:
In effect, the first time REPEAT action calls Sf Query module with URI column containing the initial URL and on the second, third and so forth Sf Query is called with URI column containing nextRecordsUrl from the previous iteration
@olysak -
I think I miss one part - do not understand this:
In the example I saw: [URL] = /?q.
The following: Main
I have a token and Path I need to push to the Sf Query
My starting Path will always be: CheckInParticipant
On SFQuery
I do not understand the Skip action condition, as I will always have a path:
My Iterate WEb request on Sf Query Raw looks like:
Path - from the Main and Authorisation from the main
I tried add URL parameters - and think that is where I should add the nextpage, but could note get it to work
The last action: Modify Columns on Sf Query Raw
Do I remove everything from the Path before CheckInParticipant or keep CheckInParticipant i.e. https://xxx/api/Objects/CheckInParticipant?$skiptoken=eyJjIjpmYWxzZSwicyI6W3sibiI6ImNoZWNrSW5fcGFydGljaXBhbnRJZCIsIn
Thanks
Hi @Rykie
The exact details of how your skiptoken should be utilized depends on API you’re trying to use. Sometimes this token should be appended to the URL, like first if request was
https://something/endpoint?q=some
and that request returned skiptoken in the body, like
{
....
'skiptoken': 'abcdefg123'
}
then next request should be:
https://something/endpoint?q=some&token=abcdefg123, meaning everything stays the same except one more URL parameter appended with the skiptoken value.
Sometimes, pagination is implemented in the form of next page URL being completely replaced with nextPageUrl token from the prevoius step – that’s the case with Salesforce example. In this case, for example, if your first request was
https://something/endpoint?q=some
and that request returned for example nextPage in the body, like
{
....
'nextPage': 'https://whatever/whatever?whatever=1234'
}
then next request should be:
https://whatever/whatever?whatever=1234, meaning you don’t modify the original URL but blindly use next page URL that was given to you by the API.
The exact case depends on what says the documentation to the API you’re trying to use.
One more thing, about the REPEAT action - the key concept that may be causing the misunderstanding is how the iteration is implemented in EasyMorph. The first time a module is being called by REPEAT action it gets input dataset that equals to the dataset you push to REPEAT action. But the second, third and so on times the input dataset of the called module is the output of the last iteration.
So, you should determine:
Iterate Web Request should look during the first call with initial input dataset? How should I make it look the way API expects it to be?Iterate Web Request should point to the query for the second page.The iterations and their implementation in EasyMorph can be somewhat tricky 
Thank you @olysak.
I worked out what the 1st page and next page URLs should look like.
(Using OData server driven paging protocol.)
What I am stuck with now, is authorisation. On my 1st call, the repeat pushes the acces-token as part of the dataset. How do I keep that token as I need it for iterations 2,3…
On my 2nd iteration, I get the error:
Make sure that your Sf Query Raw module has access_token column in its output and that column contains access token. Remember that first iteration's output is second iteration input and if the module requires access_token column to be present in the data, it always must be present.
Additionally,
If I understand correctly that you're manually utilizing the acces_token and this token doesn't change from page to page, it can be more convenient to use module parameters to pass this token.
Go to the module you're trying to call repeatedly and to the parameters section. Add an access_token parameter there:
In your REPEAT action there is section Assign parameters of the project/module using these columns... pick your newly created parameter in the module and assign it's value using 'first value of column' option, pointing it to the column in the input dataset that contains your access token:
Then, in the called module you can pick access_token for your Iterate Web request action from the parameter instead of dragging it through your datasets:
Thanks, @Olysak.
I am still missing a part. I know about the parameter - but can’t get that to work either.
I am not sure if I need another subquery.
Currently:
Main Query
I use this to move to SF Query Raw
On SF Query Raw
Iterate Web request
If I use the Parameter, comes with format value is invalid as I have the following setup in my SF Query Raw
- probably as the value does not stay.
In your response, you indicated I should keep a column Token in my SF Query Raw - but I do not know how to keep it as with iterate web, it is no longer available.
When I use the column option in value for Authorisation, I get the 1st value for nextlink.
Should I have a 3rd module:
Currently I have main (with repeat) that push data to SF Query Raw that use Iterate Web request and it gives me the 1st NextLink as I cannot get Authorisation back in the SF Query Raw.
Sorry - so close, but so far…
I suppose you could keep this token as a parameter, since it doesn't change from iteration to iteration.
Could you please clarify what do you mean 'comes with format value is invalid'?
You should:
access_token parameter to equal what you need to be passed to the API, input column with access_token that's fed to Repeat action should probably look like Bearer d85f22a8fefc254a256478b80903fac416499ba13ac7914978dc573bdc8d7f77Authorization header.As far as I see in the screenshot, you do both 1 and 2 but in the next screenshot as far as I see you keep using access_token from the dataset, not from the parameter
Thanks, Olysak.
I did all of that, but this is the error:
Please note my Base URL data Exporter has no authentication and I need to add the Access token generated in a previous step - and yes - it does come with Bearer …
So, I 1st generate the access token, pass it via Repeat to sub module and in Iterate web Request when I then use as value the parameter, I get a format of value ‘’ is invalid - I am sure it is on the 2nd run, where the value is not available anymore.
You’re welcome!
