I’m considering EasyMorph, but I have a couple of questions after reading through the material on the website. I hope someone can answer my questions.
I subscribe to stocks and futures data service called CSI. All the data resides on my hard drive in a proprietary, encrypted, and compressed format. They have two ways to retrieve data. One can use their interface called ‘Unfair Advantage’ and it will create csv files. Or, since they have an API, one can programmatically access the data directly. It is this second capability for which I wish to know if EasyMorph can utilize. Can EasyMorph access data through an API?
Can EasyMorph use data recursively? For example, suppose one had a table of data upon which calculations are made. While calculating the table, can EasyMorph access the result of a prior calculation in order to use it on the calculation of the current line of a table?
If I’m unclear on any point, please don’t hesitate to ask me questions and in answering I will hopefully clarify my question.
At this point the recommended way to access external APIs is to use the "Run program" transformation to call 3rd party utilities (e.g. curl or wget) for accessing APIs. If the API requires simply a GET request to download a file, then the "Download file" transformation may be of help.
This sounds a bit vague to me. Do you mind providing an example?
Thanks, Dmitry, for the answer regarding using ‘run program’ along with a 3rd party utility. One question about this: would the data be saved to disk and then EasyMorph would read the disk file? Or, would the data be returned dynamically?
[edit] Also, can I pass parameters to a utility in the ‘run program’ command?
Regarding your request for clarification let me give some examples:
Suppose one had a table of stock data and that he wished to find out if the closing price (one of the fields/columns) for any day of the data is greater than the closing price 5 days ago. As I understand it, EasyMorph has a ‘shift’ function that would duplicate the closing price and shift the column back 5 rows. Then, each row would have two closing price columns: the current day’s and the one from 5 days ago. This is all very straight-forward and does not require any recursive calculation. So, let’s look at example 2.
Suppose the same table of stock data. This time a calculation is made on some combination (call it calc ‘a’) of high, low, and close price data to produce a value. In addition, an ‘if’ statement is required that says, 'if today’s ‘a’ calculation is greater than yesterday’s result, then keep the current ‘a’ calculation as the current result, else make some ‘b’ calculation of some other combination of high, low, and close as the result. This is recursive. That is, the current line’s data requires the result of the prior calculation. Is there a way to do this without having to iterate on every line of the table individually.
Both scenarios are possible. It depends on how the external utility works. If it saves the result into a file, then EasyMorph can read it afterwards. If it outputs into the console, then the "Run program" transformation has the capability to capture the output of external application into a new dataset.
See the example below. Did I understand you correctly?
UPDATE
Oops, didn't realize that the dates are descending, not ascending. So the shift shoud've been done up, not down, and the IF expression should work the other way.
OK, I think now I understood your question. If you need the “replaced a” (the Result in the example) used for the next day calculation as the “yesterday’s a” then it’s indeed a recursion and is not possible in EasyMorph without using iterations.
Out of curiosity, I made an example with recursive calculation using iterations. The result of previous calculation is transferred between iteration steps through an external file.
First, I’m assuming that EasyMorph does calculation as in my example 1 using Matrix calculations (rather than line by line). So, my question is this: how fast are iterations in comparison to the ‘matrix’ calculations?
Second, my recursive example was very generic. May I send a real-world recursive calculation? I promise it won’t be difficult. I just want to see if EasyMorph can handle it. I’ll send it tomorrow.
Matrix calculations (we call them array calculations) are much faster than iterations. Probably 2-3 orders of magnitude faster because they are more lightweight and use CPU parallelization. The iteration example in my previous post had 250 rows and ran for 2 seconds, which is comparable to an array calculation on about 100K rows.
Sure, send it to support(а)easymorph.com
Thanks,
Dmitry
PS. Recursive calculations is an interesting topic. I suspect we might be able to introduce a simple transformation for it, because it's easy in Excel, so shouldn't be hard for EasyMorph too. Still analyzing the case, though. It would be interesting to look at your real-world calculation.
I have attached a zip file that contains a page out of an article describing a transformation of data that includes recursion. It’s very simple. The page shows the Excel formulas.
It is simply a 5 column spreadsheet containing Date, Open, High, Low, and Close for a stock. The goal of the calculation is to modify the Open, High, Low, Close data transforming them into a modified Open, High, Low, Close.
The calculation is very simple and it includes recursion.
Can you use this to illustrate how EasyMorph would work?
Also, regarding Run Program - can I send parameters to the program being executed?
I’ve updated the example above to use recursive Heiken-ashi OHLC calculation as per provided description. I’m also attaching a spreadsheet with reconciliation of haOpen calculated in EasyMorph and in Excel. They’re identical (except for minor rounding errors in last decimal digit).
The total run time is about 2 seconds on my laptop (250 rows).
The recursive logic is implemented in the helper project. It reads state.csv which contains previous day data (haOpen and haClose) needed to calculate today’s haOpen. Today’s haClose isn’t calculated recursively, so it’s just passed from the main project through a parameter. When today’s haOpen and haClose are calculated the helper project overwrites state.csv with the new values. They will be used in the next iteration.
