Multiple sources and destinations
How to use steps, collect data from multiple systems and send different parts of the result to multiple destinations.
Machine translation
Mode steps It is needed when there is only one pair source and destination not enough. It allows you to get data from multiple systems, transfer it to a single JavaScript handler, and then send the result to multiple destinations.
How steps are performed
Automation separates the steps by field kind:
- All the steps
sourceexecuted in parallel; - the results of the sources are collected into an object based on their
id; - the handler is called once for the collected object;
- All the steps
destinationexecuted in parallel; - You can transfer the entire result or the selected field to each assignment.
Automation must contain at least one source and one destination. The maximum number of steps is 50.
Step Structure
| Field | Necessarily | Appointment |
|---|---|---|
id |
yes | The unique name of the step. Starts with a letter, contains letters, numbers, _ or -, no longer than 64 characters. |
kind |
yes | source or destination. |
type |
yes | Source and purpose: http or artifact. |
method |
no | HTTP method. By default, GET for the source and POST for the destination. |
url |
for HTTP | An absolute HTTP or HTTPS address. |
headers |
no | Request headers. They support ${secrets.NAME} and ${artifacts.alias.field}. |
body |
no | The body of the HTTP request for the source or destination. The selected transform result is sent without the destination body. |
response |
for the HTTP source | The format and encoding of the response. |
request |
for HTTP destination | The format and encoding of the data being sent. |
input |
for appointment only | Which part of the handler result or the connected artifact is sent in this step. |
An example with two sources
version: 1
trigger:
type: schedule
cron: "0 4 * * *"
timezone: Europe/Moscow
steps:
- id: orders
kind: source
type: http
method: GET
url: https://sales.example.com/api/orders
headers:
Authorization: "Bearer ${secrets.SALES_TOKEN}"
response:
format: json
- id: employees
kind: source
type: http
method: GET
url: https://accounting.example.com/export/employees.xml
headers:
X-API-Key: "${secrets.ACCOUNTING_KEY}"
response:
format: xml
encoding: windows-1251
- id: analytics
kind: destination
type: http
method: POST
url: https://analytics.example.com/import
headers:
Authorization: "Bearer ${secrets.ANALYTICS_TOKEN}"
input: ${transform.output.analytics}
request:
format: json
- id: archive
kind: destination
type: http
method: POST
url: https://archive.example.com/daily.csv
headers:
X-API-Key: "${secrets.ARCHIVE_KEY}"
input: ${transform.output.archive}
request:
format: csv
encoding: utf-8
options:
delimiter: ";"
transform:
type: javascript
timeout_seconds: 3
retry:
attempts: 3
delay_seconds: 10
What the handler gets
For the given configuration input it has the following form:
{
orders: /* декодированный ответ шага orders */,
employees: /* декодированный ответ шага employees */
}
The object's keys match with id sources. Therefore, it is better to keep identifiers short and clear.
Handler Example:
function transform(input, context) {
const orders = Array.isArray(input.orders?.items)
? input.orders.items
: [];
const employeeList = input.employees?.Employees?.Employee;
const employees = Array.isArray(employeeList)
? employeeList
: employeeList
? [employeeList]
: [];
const employeeById = {};
for (const employee of employees) {
employeeById[String(employee.Id)] = employee;
}
const rows = orders.map((order) => ({
orderNumber: String(order.id),
employeeName: employeeById[String(order.employee_id)]?.Name || "Не найден",
amount: Number(order.amount || 0),
}));
return {
analytics: {
generatedAt: context.started_at,
rows,
},
archive: rows,
};
}
Selecting the data to be assigned
Field input supports two options.
Transfer the entire result:
input: ${transform.output}
Pass the object field:
input: ${transform.output.analytics}
Nested fields are supported:
input: ${transform.output.exports.accounting}
If input omitted, the destination gets the entire handler result.
The selector only works with fields of objects. It does not support array indexes and arbitrary JavaScript expressions.
What happens in case of an error
- If any source fails, the handler and destinations will not start.
- If the handler fails, the assignments are not started.
- If one or more assignments fail, the startup status is assigned Mistake with a list of problematic steps.
- The general setting applies to HTTP assignments.
retry; the artifact is recorded in a single transaction.
Sources and assignments are executed in parallel within their own group, so you cannot rely on the order of execution between two sources or between two assignments.
When steps are not needed
Do not use steps only for the sake of future expansion. For easy transfer from one system to another configuration with source and destination shorter, clearer, and easier to maintain.
Go to the directory of all fields: Automation configuration.
To store the result within the company, add the purpose of artifact. Field name sets the name of the artifact; HTTP fields are prohibited in this step.
Input artifacts at all stages
Connect artifacts: {settings: service.settings} at the root of the configuration. The values are available in source, conditions, transform and destination. A source type: artifact with name returns the saved value; HTTP supports ${artifacts.settings.field} in the URL, headers, and JSON body. In steps The destination can choose input: ${artifacts.settings}. The terms are used path: artifacts.settings.field or input.field; value it can refer to a connected artifact. In the handler, read context.artifacts.settings.
The input values are loaded once at the start of the startup, including the verification one. If there is no artifact, the launch fails. For events, the source remains the envelope of the event, but the block artifacts it works the same way. The full example, type rules, limitations, and errors are in automation artifacts.