💡 Announcement: Custom API Server

In version 5.4 (planned for release in November) we will be introducing a new major capability of EasyMorph Server - custom API endpoints. You will be able to configure a custom API server and use EasyMorph workflows to process incoming HTTP requests from your internal (no-code or not) applications. Here is how it’s going to work:

  • New Server tab “API endpoints” is used to configure custom API endpoints - endpoint URL paths and EasyMorph projects that will process HTTP requests sent to the endpoints. Different modules of the projects handle different HTTP methods, one module per method. URL path segments can be automatically assigned to module parameters.

  • The modules that handle requests, must have the new “Incoming HTTP request” action. Request URL, query parameters, headers, and body will be returned by the action as columns. Request body (e.g. JSON) can be parsed and processed using regular EasyMorph actions.

  • An HTTP response is configured by another new action “HTTP response”. With the action, you can configure a response’s headers (including cookies) and body. Also, there will be response options for downloading a file or a URL redirect. Response body (e.g. JSON) can be constructed using regular EasyMorph actions.

  • Endpoints can employ authentication either via an API key or using the HTTP Basic authentication.

With this new capability, you will be able to rapidly design API endpoints for serving internal applications – various automations, no-code apps, system-to-system real-time integrations, etc.

We did preliminary tests and even before optimizations, the Server can process tens of HTTP requests per minute which should be sufficient for most internal purposes. In the future, we’re planning to optimize project execution startup time and increase request throughput by an order of magnitude.

UI mockups

List of endpoints

Endpoint properties

Questions? Suggestions?

UPDATE
Changed the version and estimated release date to 5.4 / November.

2 Likes

Dmitry,

A great addition to the platform! Our shop has many custom internal apps and you are helping us expand the realm of automation possibilities. This will give us the ability to hand off complex data engineering tasks to EM and simplify the code base oof our apps.

Good stuff!

Will space users have control over this or only server admins have control over it? The reason is that the easyMorph server in our organization is managed by IT team.

Yes, space users will be able to control it. Custom API endpoints are a space feature.

1 Like

Great feature!
It’ll help us to implement EM in many new ways!

Environments in Custom API Server

Serving API requests is different from running ETL tasks because it’s real-time and this implies a new kind of technical requirements. One of the requirements is “hot” updating of endpoint configurations, i.e. replacing projects or changing parameter settings without downtime.

To address that requirement, we will be using environments (such as production, test, dev, etc.). One space will be able to have multiple environments. Each environment will have its own:

  • Base URL path
  • Endpoint configurations
  • API keys
  • Request sampling configuration
  • Error message configuration

It will be possible to promote an endpoint configuration from one environment (e.g. “Test”) to another environment (e.g. “Production”) with a single button click (also with a command of the “EasyMorph Server command” action).

An endpoint configuration is promoted as is, i.e. without changes. Basically, it overwrites the endpoint configuration in the target environment. After promotion, only the endpoint’s public URL changes (because the target environment has a different base URL path than the source one). The project path and the HTTP method handlers and their parameter settings don’t change. It means that if you tested an endpoint in one environment and it worked as expected, it’s guaranteed to work after promotion. Also, promotion is done in a way that guarantees zero downtime. As soon as pending requests are complete, new requests will be served using the new endpoint configuration.

An endpoint configuration in an environment can be disabled. For instance, after you promoted an endpoint configuration from “Test” to “Production” you can disable it in the test environment (to not expose unnecessary API methods) while it will continue working in production. Configurations of multiple endpoints can be disabled at once.

Here are UI mockups for managing environments. The mockups in the opening post have been updated as well.

List of environments:

Environment settings:

When designing workflows, it will be possible to retrieve sample requests from any environment. Therefore, for instance, you can retrieve a failed request from the production environment, debug/fix the workflow in the test environment, and then promote the bugfix into production.

In the initial release, only promoting endpoint configurations from one environment to another in the same space will be possible. Such a promotion guarantees zero downtime.

In future releases, we will add the ability to promote endpoint configurations from one space to another space. Such promotion will not guarantee zero downtime, but it will make it possible to develop/test endpoints in a different space (e.g. with a different set of connectors).

Even better!
Using multiple environment allow us to follow a test-production pattern, avoiding annoying problems like bugged project in production

UPDATE

We’ve simplified things a bit and removed environments as an additional abstraction layer. Instead, spaces can be configured as environments. Environments can be configured for various scenarios – shared connector repositories or separated, shared endpoint configurations or separated, full access permission to all environments or configured individually per environment, etc.

Also, we’ve settled on the difference in Server editions:

Feature Team Edition Enterprise Edition
Endpoints per space Unlimited Unlimited
Authentication API Key, HTTP Basic API Key, HTTP Basic
Spaces with API feature enabled 1 space Unlimited spaces
HTTP Methods GET, POST Any

The table above indicates that the Team edition can have API enabled in only 1 space (environment) and accept only GET and POST requests. This should be sufficient for most cases. More complex scenarios that require multiple environments or support for other HTTP methods such as PUT or DELETE will require the Enterprise edition.

Other remarks:

  • The way endpoints are configured and versioned will be compatible with git and other version control systems so it will be possible to promote changes via git.
  • It will be possible to promote endpoint configurations between two Servers (not just between two spaces of a Server) using git or folder copying.
  • Folder copying will be added as a standard operation in the “Files” tab) as it will be frequently used for endpoint versioning.
  • It will be possible to lock endpoints (disable editing) similarly to locking tasks. This can help reduce (but not eliminate) the chance of accidental changes in production environments.
  • Two spaces will be able to share the same set of endpoints but have different base URLs or connector repositories.
  • We will add new commands to the “Server command” action to enable the automation of complex deployment scenarios with automated testing, promotion between environments, file/folder management, etc. It may work especially well after the “Git command” action is introduced (it’s on the roadmap).
  • The “HEAD” HTTP method will be treated as “GET” but the response body will be removed so only headers remain.
  • Setting or modifying cookies will not be possible in the initial release but can be added later

Beta-testing of the API endpoints feature is expected to start in mid-November.

1 Like

The API Server feature is nearing the beta-testing stage. Below is a quick overview of the feature. It demonstrates how to create an API endpoint to serve incoming GET requests. The endpoint in the demo simply returns two parameters of an incoming web request as a JSON object. Not much practical value, but it demonstrates the process of creating an API endpoint.