Skip to content

Commit 210fa7a

Browse files
dieribahugocasa
dieriba
and
hugocasa
authored
docs: sqs (windmill-labs#834)
* docs: sqs * update docs --------- Co-authored-by: HugoCasa <hugo@casademont.ch>
1 parent 02273e4 commit 210fa7a

File tree

8 files changed

+183
-6
lines changed

8 files changed

+183
-6
lines changed

‎docs/core_concepts/43_preprocessors/index.mdx

Lines changed: 49 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ Here are examples of a preprocessor function in [TypeScript](../../getting_start
2323
export async function preprocessor(
2424
/* args from the request body (e.g. webhook/http body args, msg for ws/kafka/nats, raw_email and parsed_email for email) */
2525
wm_trigger: {
26-
kind: 'http' | 'email' | 'webhook' | 'websocket' | 'postgres' | 'kafka' | 'nats',
26+
kind: 'http' | 'email' | 'webhook' | 'websocket' | 'postgres' | 'kafka' | 'nats', 'sqs',
2727
http?: {
2828
route: string // The route path, e.g. "/users/:id"
2929
path: string // The actual path called, e.g. "/users/123"
@@ -47,6 +47,16 @@ export async function preprocessor(
4747
status?: number
4848
description?: string
4949
length: number
50+
},
51+
sqs?: {
52+
queue_url: string,
53+
message_id?: string,
54+
receipt_handle?: string,
55+
attributes: Record<string, string>,
56+
message_attributes?: Record<string, {
57+
string_value?: string,
58+
data_type: string
59+
}>
5060
}
5161
}
5262
) {
@@ -89,12 +99,24 @@ class Nats(TypedDict):
8999
description: str | None
90100
length: int
91101

102+
class MessageAttribute(TypedDict):
103+
string_value: str | None
104+
data_type: str
105+
106+
class Sqs(TypeDict):
107+
queue_url: str
108+
message_id: str | None
109+
receipt_handle: str | None
110+
attributes: dict[str, str]
111+
message_attributes: dict[str, MessageAttribute] | None
112+
92113
class WmTrigger(TypedDict):
93-
kind: Literal["http", "email", "webhook", "websocket", "postgres", "kafka", "nats"]
114+
kind: Literal["http", "email", "webhook", "websocket", "postgres", "kafka", "nats", "sqs"]
94115
http: Http | None
95116
websocket: Websocket | None
96117
kakfa: Kafka | None
97118
nats: Nats | None
119+
sqs: Sqs | None
98120

99121
def preprocessor(
100122
# args from the request body (e.g. webhook/http body args, msg for ws/kafka/nats, raw_email and parsed_email for email)
@@ -127,9 +149,9 @@ The flow preprocessor takes the same arguments as the script preprocessor and sh
127149
<TabItem value="TypeScript">
128150
```TypeScript
129151
export async function preprocessor(
130-
/* args from the request body (e.g. webhook/http body args, msg for ws/kafka/nats, raw_email and parsed_email for email) */
152+
/* args from the request body (e.g. webhook/http body args, msg for ws/kafka/nats/sqs, raw_email and parsed_email for email) */
131153
wm_trigger: {
132-
kind: 'http' | 'email' | 'webhook' | 'websocket' | 'postgres' | 'kafka' | 'nats',
154+
kind: 'http' | 'email' | 'webhook' | 'websocket' | 'postgres' | 'kafka' | 'nats', 'sqs',
133155
http?: {
134156
route: string // The route path, e.g. "/users/:id"
135157
path: string // The actual path called, e.g. "/users/123"
@@ -149,6 +171,16 @@ export async function preprocessor(
149171
nats?: {
150172
servers: string[]
151173
subject: string
174+
},
175+
sqs?: {
176+
queue_url: string,
177+
message_id?: string,
178+
receipt_handle?: string,
179+
attributes: Record<string, string>,
180+
message_attributes?: Record<string, {
181+
string_value?: string,
182+
data_type: string
183+
}>
152184
}
153185
}
154186
) {
@@ -187,12 +219,24 @@ class Nats(TypedDict):
187219
description: str | None
188220
length: int
189221

222+
class MessageAttribute(TypedDict):
223+
string_value: str | None
224+
data_type: str
225+
226+
class Sqs(TypeDict):
227+
queue_url: str
228+
message_id: str | None
229+
receipt_handle: str | None
230+
attributes: dict[str, str]
231+
message_attributes: dict[str, MessageAttribute] | None
232+
190233
class WmTrigger(TypedDict):
191-
kind: Literal["http", "email", "webhook", "websocket", "postgres", "kafka", "nats"]
234+
kind: Literal["http", "email", "webhook", "websocket", "postgres", "kafka", "nats", "sqs"]
192235
http: Http | None
193236
websocket: Websocket | None
194237
kafka: Kafka | None
195238
nats: Nats | None
239+
sqs: Sqs | None
196240

197241
def preprocessor(
198242
# args from the request body (e.g. webhook/http body args, msg for ws/kafka/nats, raw_email and parsed_email for email)

‎docs/core_concepts/48_sqs_triggers/index.mdx

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# SQS Triggers
2+
3+
Windmill can connect to an [SQS](https://aws.amazon.com/sqs/) queue and trigger runnables (scripts, flows) in response to messages received.
4+
SQS triggers is a self-hosted Enterprise feature.
5+
6+
For more details on SQS, see the [AWS SQS Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html).
7+
8+
![SQS triggers](./sqs_triggers.png)
9+
10+
---
11+
12+
## How to Use
13+
14+
- **Pick an AWS Resource**
15+
- Select an existing AWS resource or create a new one.
16+
- The AWS resource must have permissions to interact with SQS.
17+
18+
- **Select the Runnable to Execute**
19+
- Choose the runnable (script or flow) that should be executed when a message arrives in the queue.
20+
- The message will be passed as a JSON object to the runnable.
21+
22+
- **Provide an SQS Queue URL**
23+
- Enter the **Queue URL** of the SQS queue that should trigger the runnable.
24+
- You can find the Queue URL in the AWS Management Console under SQS.
25+
- For more details, see the [SQS Queue URL Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-message-identifiers.html#sqs-queue-url).
26+
27+
- **Choose (Optional) Message Attributes**
28+
- Specify which message attributes should be included in the triggered event.
29+
- These attributes can carry metadata, such as sender information or priority levels.
30+
- For more details, see the [SQS Message Attributes Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes).
31+
32+
33+
## Example
34+
35+
Below are code examples demonstrating how to handle SQS messages in your Windmill scripts. You can either process messages directly in a basic script or use a preprocessor for more advanced message handling and transformation before execution.
36+
37+
### Basic Script Example
38+
39+
```TypeScript
40+
export async function main(msg: string) {
41+
// do something with the message
42+
}
43+
```
44+
45+
### Using a Preprocessor
46+
47+
If you use a [preprocessor](../43_preprocessors/index.mdx), the preprocessor function receives an SQS message with the following fields:
48+
49+
#### Field Descriptions
50+
51+
- **`queue_url`**: The URL of the SQS queue that received the message.
52+
- **`message_id`**: A unique identifier assigned to each message by SQS.
53+
- [More details](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html)
54+
- **`receipt_handle`**: A token used to delete the message after processing.
55+
- [More details](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-message-identifiers.html)
56+
- **`attributes`**: Metadata attributes set by SQS, such as `SentTimestamp`.
57+
- [Full list of system attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-system-attributes)
58+
- **`message_attributes`**: User-defined attributes that can be attached to the message.
59+
- `string_value`: The string representation of the attribute value.
60+
- `data_type`: The data type of the attribute (e.g., `"String"`, `"Number"`, `"Binary"`).
61+
- [More details on message attributes](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-metadata.html#sqs-message-attributes)
62+
63+
64+
```TypeScript
65+
export async function preprocessor(
66+
msg: string,
67+
wm_trigger: {
68+
kind: "sqs",
69+
sqs: {
70+
queue_url: string,
71+
message_id?: string,
72+
receipt_handle?: string,
73+
attributes: Record<string, string>,
74+
message_attributes?: Record<string, {
75+
string_value?: string,
76+
data_type: string
77+
}>
78+
}
79+
},
80+
) {
81+
// assuming the message is a JSON object
82+
const data = JSON.parse(msg);
83+
84+
return {
85+
content: data.content,
86+
metadata: {
87+
sentAt: wm_trigger.sqs.attributes.SentTimestamp,
88+
messageId: wm_trigger.sqs.message_id
89+
}
90+
};
91+
}
92+
93+
export async function main(content: string, metadata: { sentAt: string, messageId: string }) {
94+
// Process transformed message data
95+
console.log(`Processing message ${metadata.messageId} sent at ${metadata.sentAt}`);
96+
console.log("Content:", content);
97+
}
98+
```

‎docs/core_concepts/48_sqs_triggers/sqs_triggers.png

109 KB
Loading

‎docs/core_concepts/index.mdx

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,11 @@ On top of its editors to build endpoints, flows and apps, Windmill comes with a
4747
description="Trigger scripts and flows from NATS subjects."
4848
href="/docs/core_concepts/nats_triggers"
4949
/>
50+
<DocCard
51+
title="SQS"
52+
description="Trigger scripts and flows from Amazon SQS."
53+
href="/docs/core_concepts/sqs_triggers"
54+
/>
5055
</div>
5156

5257
## Windmill features

‎docs/getting_started/8_triggers/index.mdx

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Triggers from external events:
2222
- [Postgres triggers](#postgres-triggers)
2323
- [Kafka triggers](#kafka-triggers)
2424
- [NATS triggers](#nats-triggers)
25+
- [SQS triggers](#sqs-triggers)
2526
- [Scheduled polls](#scheduled-polls-scheduling--trigger-scripts)
2627

2728
:::info Scripts and Flows in Windmill
@@ -305,6 +306,18 @@ Windmill can connect to NATS servers and trigger scripts or flows when messages
305306
/>
306307
</div>
307308

309+
### SQS triggers
310+
311+
Windmill can connect to an Amazon SQS queues and trigger scripts or flows when messages are received. This enables event-driven processing from your AWS ecosystem. Preprocessors can transform the SQS message data before it reaches your script or flow.
312+
313+
<div className="grid grid-cols-2 gap-6 mb-4">
314+
<DocCard
315+
title="SQS triggers"
316+
description="Trigger scripts and flows from Amazon SQS messages."
317+
href="/docs/core_concepts/sqs_triggers"
318+
/>
319+
</div>
320+
308321
### Scheduled polls (Scheduling + Trigger scripts)
309322

310323
A particular use case for schedules are [Trigger scripts](../../flows/10_flow_trigger.mdx).

‎docs/script_editor/settings.mdx

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,18 @@ Windmill can connect to NATS brokers and trigger scripts or flows when messages
280280
/>
281281
</div>
282282

283+
### SQS triggers
284+
285+
Windmill can connect to Amazon SQS queues and trigger scripts or flows when messages are received. This enables event-driven processing from your AWS ecosystem. Preprocessors can transform the SQS message data before it reaches your script or flow.
286+
287+
<div className="grid grid-cols-2 gap-6 mb-4">
288+
<DocCard
289+
title="SQS triggers"
290+
description="Trigger scripts and flows from Amazon SQS messages."
291+
href="/docs/core_concepts/sqs_triggers"
292+
/>
293+
</div>
294+
283295
### Email
284296

285297
Scripts and flows can be triggered by email messages sent to a specific email address, leveraging SMTP.

‎sidebars.js

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,11 @@ const sidebars = {
150150
type: 'doc',
151151
id: 'core_concepts/nats_triggers/index',
152152
label: 'NATS'
153+
},
154+
{
155+
type: 'doc',
156+
id: 'core_concepts/sqs_triggers/index',
157+
label: 'SQS'
153158
}
154159
]
155160
},

‎src/components/Pricing.js

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1106,4 +1106,4 @@ export default function Pricing() {
11061106
/>
11071107
</div>
11081108
);
1109-
}
1109+
}

0 commit comments

Comments
 (0)