Add File-Based CSV Format Option for Dataset Exchange in "Call Python" Action

Feature requests
1

Hi,
I've been using the "Call Python" action and I'm very pleased with how it enables seamless integration between EasyMorph workflows and Python scripts. The ability to pass parameters and datasets between the two environments has significantly enhanced my data processing capabilities. The current implementation with the Easymorph Python module works great, and I'd like to suggest an enhancement that would make this already excellent feature even more versatile.

Add an option to the "Call Python" action to pass input datasets and receive output datasets as CSV files, in addition to the current native format.

Currently, the "Call Python" action exchanges datasets with Python scripts using EasyMorph's native format through the Easymorph Python module.

Proposed Enhancement:

Add a format option in the "Call Python" action settings with two choices:

Native** (default): Current behavior using the easymorph module
CSV File**: Exchange datasets via CSV files

Implementation Details

When CSV File format is selected:

  1. Input Dataset Transfer:
  • EasyMorph writes the input dataset to a temporary CSV file
  • The file path is passed to the Python script as a predefined parameter (e.g., input_file or $INPUT_CSV_PATH)
  • The temporary file is automatically cleaned up after execution
  1. Output Dataset Transfer:
  • EasyMorph provides a temporary output file path to the Python script (e.g., output_file or $OUTPUT_CSV_PATH)
  • The Python script writes its results to this CSV file
  • EasyMorph reads the CSV file back as the result dataset
  • The temporary file is automatically cleaned up after execution
  1. CSV Configuration Options:
  • Delimiter (default: comma)
  • Quote character (default: double quote)
  • Encoding (default: UTF-8)
  • Include headers (default: yes)
  • Line terminator (default: system default)

Benefits

  • Performance : File-based transfer is efficient for large datasets
  • Reliability : Avoids potential issues with stdin/stdout buffering or encoding
  • Debugging : Temporary files can optionally be preserved for troubleshooting
  • Compatibility : Works with any Python CSV processing library
  • Memory Efficiency : Suitable for datasets too large to fit in memory

Use Cases

  • Processing large datasets with pandas or other data libraries
  • Integration with existing Python ETL scripts
  • Scenarios requiring specific CSV dialects or formats
  • Development and debugging workflows where file inspection is helpful

It would be great if you can look into this

Regards,
Knut P