Multiline text file - how to capture details?

1

Hi
I’m looking through log files from our sortation system, and in a specific telegram type (carton manifest), I get multiple lines per header.
I know how to get to the header, but how do I link to the details underneath?

In below screenshot, I take the line, parse it and split into columns (simple to start with - timestamp and message). However, I don’t get the details for the header lines.

The telegram is identified by telegram type = 63
Each of those telegrams can contain 1 or more lines underneath.

Is it possible?

image
image.png1319×981 82.5 KB

2

I'm not sure I understand what that would be. Can you point at the details of the header lines on your screenshot?

3

Hi
I can see me being in a hurry writing the original post, made interpretation almost impossible :laughing:

I actually think I managed to get something working, but was asking to see if there was an easier way to do it.

In the example below, I have a project with 2 modules; 1 for importing and processing Item Manifest telegrams, 1 for importing and processing Carton Manifest telegrams.
Source of data is an example log file containing first 300 rows of data of a real log file.

The telegram build is as described below.

image
image956×720 73 KB

1, A and B all relates to the Item Manifests;
A = the string to decipher - documentation underneath
B = example of string

2, C, D and E all relates to the Carton Manifests;
C = the string to decipher - documentation underneath
D = comment regarding the issue - items in carton are put in separate lines within the telegram, but on new line with no link, when reading in EasyMorph or NotePad++
E = example of string - only for first item, so the issue is not presented in the example

When reading the log file in Notepad++, it is obvious what to see the issue;

image
image1486×860 82.9 KB

So, in my own solution, I read the file, and based on some logic in import column (Column 1) I defined a type for the relevant lines;

Type = if 
contains([Column 1],'text:63') then 'Start' else if 
(len([Column 1])=36 and left([Column 1],2)='00') then 'Data' else if 
[Column 1]="##[" then 'end' else ''

Then I defined start of the string among the endless number of lines;

Start = if [Type]='Start' then replace([Column 1],
' DE(D) sortschemehandler.hostif.BaseP5 Sending status telegram to the host (or putting in buffer): text:','|')

Then I split the column 'Start' to get a time stamp and the message.
Then parsed the message using telegram documentation from excerpt above.
Made 2 new calculated columns from the extra lines within the telegram- SKU ID and Number of items sorted.
Calculated an end of the telegram based on the type defined in first step.
Fill Down from telegram header to 'end' to get header details on SKU ID / Detail level.
Count records in header.
Filter for only relevant telegram types.

Starting point being the log file shown above, or in EasyMorph:

image
image1040×886 53 KB

Current end point being a clean table with detail and header information:

image
image1273×716 47.5 KB

Spoiler alert - I was originally looking for a shortcut to the 9 steps in table 'Modify' :smile:

Example.zip (6.8 KB)

4

This looks like a sectioned file. A sectioned file is a file where a set of consequent rows is a header, and then a set of consequent rows is items (details).

The idiomatic approach to parsing sectioned files is as follows:

  1. Create a marker column first in which for each row you define the row type - “Header” or “Item”.

  2. Create additional columns for header data using expressions such as

    IF [Row Type] = "Header" THEN //Some header data

  3. Use the “Fill down” action to “stretch” data in header rows down, so that it fills out the item rows (which were empty until now).

  4. Finally, remove header rows by removing all rows where [Row type] = “Header”

If you do everything right, then the rows with header data will become columns with header data.