September 27, 2024

Process S3 Objects with Step Functions using CDK + TS

Claudio Ctin28 mins
Process S3 Objects with Step Functions using CDK + TS

Preface

Whenever a user needed to launch automation, our tech lead had to dive into an extensive manual process that took about two days to configure everything.
Picture this: an Excel file with over six tabs and data manipulation using Excel functions (a spreadsheet nightmare 👻).
Seeing this madness, I realised we could automate this process (trust me, it was a heated debate to reach this conclusion). Thus, the architecture below was born.

Remember, we’ve got some proprietary code, so this is a simplified and slightly fictionalised version to illustrate the solution.

The Solution

We needed to set up a process to trigger a Step Function every time an object was created in S3. However, direct invocation of a Step Function is not currently supported. Therefore, I created an EventBridge rule to monitor S3 for object creation. Once an object was created, the rule would invoke the Step Function to process the object using one of the Lambda tasks.
I used the Express type since I needed this to be a quick process. With this decision, I had to parallelise some steps (yes, my automation allowed for that), so, I used the Parallel state. Finally, if any errors occurred, I set up a notification using a topic for that.

Express x Standard
Standard: For long-running processes with exactly-once execution guarantees and advanced recovery features.
Express: For fast, high-throughput processes with at least one execution guarantee and billing based on time and memory.

Getting your hands dirty


Let’s implement it using CDK with Typescript!

Let’s start by creating our bucket and importing a topic to notify us of any errors

const userNotificationTopic = Topic.fromTopicArn(
this,
‘userNotificationTopic‘,
Fn.importValue(‘infrastructure::sns::user-notification-topic::arn‘),
)

const bucket = new Bucket(this, ‘bucket‘, {
bucketName: ‘automation-configuration‘,
removalPolicy: RemovalPolicy.DESTROY,
})

Now, let’s create our Lambdas and grant read permissions to the Lambda that will access the S3 object:

const etlLambda = new NodejsFunction(this, ‘etl-lambda‘, {
entry: ‘src/lambda/etl/etl.handler.ts‘,
functionName: ‘etl‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

bucket.grantReadWrite(
new Alias(this, id.concat(‘alias‘), {
aliasName: ‘current‘,
version: etlLambda.currentVersion,
}),
)

const categoriesLambda = new NodejsFunction(this, ‘insert-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const hospitalsLambda = new NodejsFunction(this, ‘hospitals-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

To call a Lambda within our Step Function, we need to create a Lambda Invoke for each of the Lambdas. So, I created a function to avoid repeating this for every Lambda:

private mountLambdaInvokes(
lambdasInvoke: Array<{
function: IFunction
name?: string
output?: string
}>,
) {
return lambdasInvoke.map(lambdaInvoke => {
return new LambdaInvoke(this, `${lambdaInvoke.name || lambdaInvoke.function.functionName}-task`, {
lambdaFunction: lambdaInvoke.function,
inputPath: ‘$‘,
outputPath: lambdaInvoke?.output || ‘$‘,
})
})
}

Using mountLambdaInvokes:

const [etlTask, categoriesTask, hospitalsTask] = this.mountLambdaInvokes([
{ function: etlLambda, output: ‘$.Payload‘ },
{ function: categoriesLambda },
{ function: hospitalsLambda },
])

We need to create our failure step and the SnsPublish to send failed events to the topic we imported earlier:

const errorTopicConfig = {
topic: userNotificationTopic,
subject: ‘Automation Config Failed 😥‘,
message: TaskInput.fromObject({
message: ‘Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘suporte@somemail.com‘] },
}),
}

const publishFailed = (publishFailedId: string) =>
new SnsPublish(this, `automation-config-sns-failed-${publishFailedId}`, errorTopicConfig)

const jobFailed = new Fail(this, ‘automation-config-job-failed‘, {
cause: ‘Unexpected Error‘,
})

Having done this, let’s set up our Parallel state. Each branch will be a Lambda processed simultaneously. We’ll also add a retry to attempt again if any issues arise and a catch to handle any failures in this parallel process:

const hospitalsCategoriesParallel = new Parallel(this, ‘auto-config-exams-parallel-map‘)
.branch(categoriesTask)
.branch(hospitalsTask)
.addRetry({ errors: [‘States.ALL‘], interval: Duration.seconds(5), maxAttempts: 1 })
.addCatch(publishFailed(‘exams‘).next(jobFailed), {
errors: [‘States.ALL‘],
})

At this point, we’ll create the definition of our tasks, a log group, and the State Machine, which is essentially our Step Function:

const definition = etlTask.next(hospitalsCategoriesParallel)

const logGroup = new LogGroup(this, ‘automation-configuration-log-group‘, {
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
})

const stateMachine = new StateMachine(this, `${id}-state-machine`, {
definition,
timeout: Duration.minutes(5),
stateMachineName: ‘automation-configuration‘,
stateMachineType: StateMachineType.EXPRESS,
logs: {
destination: logGroup,
includeExecutionData: true,
level: LogLevel.ALL,
},
})

After this, we need to create the rules to associate EventBridge with S3 and the execution of our Step Function:

const s3EventRule = new Rule(this, ‘automation-config-s3-event-rule‘, {
ruleName: ‘automation-config-s3-event-rule‘,
})

const eventRole = new Role(this, ‘eventRole‘, {
assumedBy: new ServicePrincipal(‘events.amazonaws.com‘),
})

stateMachine.grantStartExecution(eventRole)
s3EventRule.addTarget(
new SfnStateMachine(stateMachine, {
input: RuleTargetInput.fromObject({
detail: EventField.fromPath(‘$.detail‘),
}),
role: eventRole,
}),
)
s3EventRule.addEventPattern({
source: [‘aws.s3‘],
detailType: [‘Object Created‘],
detail: {
bucket: {
name: [bucket.bucketName],
},
object: {
key: [
{
wildcard: ‘csv/automation-configuration/*.csv‘,
},
],
},
},
})

Finally, let’s create a rule to listen to any unexpected failures that occur in our Step Function through EventBridge as well, thus maintaining the event-driven nature that we love so much:

const unexpectedFailRule = new Rule(this, ‘exam-automation-config-unexpected-fail-rule‘, {
ruleName: ‘exam-automation-config-unexpected-fail-rule‘,
})
unexpectedFailRule.addTarget(
new SnsTopic(userNotificationTopic, {
message: RuleTargetInput.fromObject({
subject: ‘Exam Automation Config Failed 😥‘,
message: ‘Exam Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘it@somemail.com‘] },
}),
}),
)

unexpectedFailRule.addEventPattern({
source: [‘aws.states‘],
detailType: [‘Step Functions Execution Status Change‘],
detail: {
stateMachineArn: [stateMachine.stateMachineArn],
status: [‘FAILED‘, ‘TIMED_OUT‘, ‘ABORTED‘],
},
})

Putting everything together in a class so you can understand the entire unified flow:

import { Duration, Fn, RemovalPolicy } from ‘aws-cdk-lib‘
import { Rule, RuleTargetInput, EventField } from ‘aws-cdk-lib/aws-events‘
import { SfnStateMachine, SnsTopic } from ‘aws-cdk-lib/aws-events-targets‘
import { Role, ServicePrincipal } from ‘aws-cdk-lib/aws-iam‘
import { Alias, Architecture, IFunction } from ‘aws-cdk-lib/aws-lambda‘
import { NodejsFunction } from ‘aws-cdk-lib/aws-lambda-nodejs‘
import { LogGroup, RetentionDays } from ‘aws-cdk-lib/aws-logs‘
import { Bucket } from ‘aws-cdk-lib/aws-s3‘
import { Topic } from ‘aws-cdk-lib/aws-sns‘
import { Fail, LogLevel, Parallel, StateMachine, StateMachineType, TaskInput } from ‘aws-cdk-lib/aws-stepfunctions‘
import { LambdaInvoke, SnsPublish } from ‘aws-cdk-lib/aws-stepfunctions-tasks‘
import { Construct } from ‘constructs‘

export default class AutomationConfiguration extends Construct {
constructor(scope: Construct, id: string) {
super(scope, id)

const userNotificationTopic = Topic.fromTopicArn(
this,
‘userNotificationTopic‘,
Fn.importValue(‘infrastructure::sns::user-notification-topic::arn‘),
)

const bucket = new Bucket(this, ‘bucket‘, {
bucketName: ‘automation-configuration‘,
removalPolicy: RemovalPolicy.DESTROY,
})

const etlLambda = new NodejsFunction(this, ‘etl-lambda‘, {
entry: ‘src/lambda/etl/etl.handler.ts‘,
functionName: ‘etl‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

bucket.grantRead(
new Alias(this, id.concat(‘alias‘), {
aliasName: ‘current‘,
version: etlLambda.currentVersion,
}),
)

const categoriesLambda = new NodejsFunction(this, ‘insert-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const hospitalsLambda = new NodejsFunction(this, ‘hospitals-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const [etlTask, categoriesTask, hospitalsTask] = this.mountLambdaInvokes([
{ function: etlLambda, output: ‘$.Payload‘ },
{ function: categoriesLambda },
{ function: hospitalsLambda },
])

const errorTopicConfig = {
topic: userNotificationTopic,
subject: ‘Automation Config Failed 😥‘,
message: TaskInput.fromObject({
message: ‘Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘suporte@somemail.com‘] },
}),
}

const publishFailed = (publishFailedId: string) =>
new SnsPublish(this, `automation-config-sns-failed-${publishFailedId}`, errorTopicConfig)

const jobFailed = new Fail(this, ‘automation-config-job-failed‘, {
cause: ‘Unexpected Error‘,
})

const hospitalsCategoriesParallel = new Parallel(this, ‘auto-config-exams-parallel-map‘)
.branch(categoriesTask)
.branch(hospitalsTask)
.addRetry({ errors: [‘States.ALL‘], interval: Duration.seconds(5), maxAttempts: 1 })
.addCatch(publishFailed(‘exams‘).next(jobFailed), {
errors: [‘States.ALL‘],
})
const definition = etlTask.next(hospitalsCategoriesParallel)

const logGroup = new LogGroup(this, ‘automation-configuration-log-group‘, {
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
})

const stateMachine = new StateMachine(this, `${id}-state-machine`, {
definition,
timeout: Duration.minutes(5),
stateMachineName: ‘automation-configuration‘,
stateMachineType: StateMachineType.EXPRESS,
logs: {
destination: logGroup,
includeExecutionData: true,
level: LogLevel.ALL,
},
})

const s3EventRule = new Rule(this, ‘automation-config-s3-event-rule‘, {
ruleName: ‘automation-config-s3-event-rule‘,
})

const eventRole = new Role(this, ‘eventRole‘, {
assumedBy: new ServicePrincipal(‘events.amazonaws.com‘),
})

stateMachine.grantStartExecution(eventRole)
s3EventRule.addTarget(
new SfnStateMachine(stateMachine, {
input: RuleTargetInput.fromObject({
detail: EventField.fromPath(‘$.detail‘),
}),
role: eventRole,
}),
)
s3EventRule.addEventPattern({
source: [‘aws.s3‘],
detailType: [‘Object Created‘],
detail: {
bucket: {
name: [bucket.bucketName],
},
object: {
key: [
{
wildcard: ‘csv/automation-configuration/*.csv‘,
},
],
},
},
})

const unexpectedFailRule = new Rule(this, ‘exam-automation-config-unexpected-fail-rule‘, {
ruleName: ‘exam-automation-config-unexpected-fail-rule‘,
})
unexpectedFailRule.addTarget(
new SnsTopic(userNotificationTopic, {
message: RuleTargetInput.fromObject({
subject: ‘Exam Automation Config Failed 😥‘,
message: ‘Exam Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘it@somemail.com‘] },
}),
}),
)

unexpectedFailRule.addEventPattern({
source: [‘aws.states‘],
detailType: [‘Step Functions Execution Status Change‘],
detail: {
stateMachineArn: [stateMachine.stateMachineArn],
status: [‘FAILED‘, ‘TIMED_OUT‘, ‘ABORTED‘],
},
})
}

private mountLambdaInvokes(
lambdasInvoke: Array<{
function: IFunction
name?: string
output?: string
}>,
) {
return lambdasInvoke.map(lambdaInvoke => {
return new LambdaInvoke(this, `${lambdaInvoke.name || lambdaInvoke.function.functionName}-task`, {
lambdaFunction: lambdaInvoke.function,
inputPath: ‘$‘,
outputPath: lambdaInvoke?.output || ‘$‘,
})
})
}
}

[🇧🇷 PT-BR]

Prefácio

Sempre que um usuário precisava subir uma automação, o nosso tech lead tinha que realizar um trabalho manual bem extensivo do qual levava cerca de dois dias para configurar tudo, usando um arquivo xls com mais de 6 abas e manipulando os dados com algumas funções do excel, ao ver esta situação percebi que podíamos sim automatizar este fluxo (inclusive foi uma discussão acalorada para chegarmos nesta conclusão), assim surgiu a arquitetura abaixo (temos algumas questões de código proprietário então esta arquitetura é uma visão simplificada e com items ficticios para demonstrar esta solução)

A Solução

Tinhamos que triggar um Step Function quando um objeto fosse criado no S3, infelizmente hoje não temos como chamar de forma direta um step function, então tive que criar um event bridge rule para ouvir a criação de um objeto no meu S3, e quando isso fosse criado chamar o step function e consumir o objeto em questão por um dos lambdas tasks, como eu precisava que fosse um processo mais rápido utilizei o tipo express, com esta decisão eu precisava paralelizar alguns steps (sim, minha automatização possibilitava isso) assim utilizei o state Parallel, por fim caso eu tivesse algum erro eu notifico utilizando um tópico para isso.

Express x Standard

Standard: Para processos longos, com execuções garantidas exatamente uma vez e recursos avançados de recuperação.

Express: Para processos rápidos e de alta taxa de transferência, com execuções garantidas pelo menos uma vez e cobrança baseada em tempo e memória.

Colocando a mão na massa

Bora implementar isto com CDK e Typescript!

Vamos começar criando nosso bucket e importanto um tópico para notificar nossos erros

const userNotificationTopic = Topic.fromTopicArn(
this,
‘userNotificationTopic‘,
Fn.importValue(‘infrastructure::sns::user-notification-topic::arn‘),
)

const bucket = new Bucket(this, ‘bucket‘, {
bucketName: ‘automation-configuration‘,
removalPolicy: RemovalPolicy.DESTROY,
})

Agora vamos criar nossos lambdas e dar a permissão de leitura para o lambda que irá acessar o objeto do S3:

const etlLambda = new NodejsFunction(this, ‘etl-lambda‘, {
entry: ‘src/lambda/etl/etl.handler.ts‘,
functionName: ‘etl‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

bucket.grantReadWrite(
new Alias(this, id.concat(‘alias‘), {
aliasName: ‘current‘,
version: etlLambda.currentVersion,
}),
)

const categoriesLambda = new NodejsFunction(this, ‘insert-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const hospitalsLambda = new NodejsFunction(this, ‘hospitals-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

Para que possamos chamar uma lambda dentro do nosso Step Function temos que criar um Lambda Invoke para cada um dos lambdas, assim criei uma função para não precisar ficar repetindo isto para cada lambda:

private mountLambdaInvokes(
lambdasInvoke: Array<{
function: IFunction
name?: string
output?: string
}>,
) {
return lambdasInvoke.map(lambdaInvoke => {
return new LambdaInvoke(this, `${lambdaInvoke.name || lambdaInvoke.function.functionName}-task`, {
lambdaFunction: lambdaInvoke.function,
inputPath: ‘$‘,
outputPath: lambdaInvoke?.output || ‘$‘,
})
})
}

Utilizando a função mountLambdaInvokes:

const [etlTask, categoriesTask, hospitalsTask] = this.mountLambdaInvokes([
{ function: etlLambda, output: ‘$.Payload‘ },
{ function: categoriesLambda },
{ function: hospitalsLambda },
])

Precisamos criar nosso step de falha e o SnsPublish para enviar os eventos que falharam para o tópico que importamos antes:

const errorTopicConfig = {
topic: userNotificationTopic,
subject: ‘Automation Config Failed 😥‘,
message: TaskInput.fromObject({
message: ‘Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘suporte@somemail.com‘] },
}),
}

const publishFailed = (publishFailedId: string) =>
new SnsPublish(this, `automation-config-sns-failed-${publishFailedId}`, errorTopicConfig)

const jobFailed = new Fail(this, ‘automation-config-job-failed‘, {
cause: ‘Unexpected Error‘,
})

Feito isto vamos montar nosso Parallel, onde cada branch é um lambda que será processado simultaneamente, adicionaremos também um retry para q seja feita a tentativa novamente caso ocorra algum problema e um catch para capturar as falhas desse processo paralelo:

const hospitalsCategoriesParallel = new Parallel(this, ‘auto-config-exams-parallel-map‘)
.branch(categoriesTask)
.branch(hospitalsTask)
.addRetry({ errors: [‘States.ALL‘], interval: Duration.seconds(5), maxAttempts: 1 })
.addCatch(publishFailed(‘exams‘).next(jobFailed), {
errors: [‘States.ALL‘],
})

Neste ponto vamos criar a definition das nossas tasks, um log group e a State Machine que nada mais é que nossa Step Function de fato

const definition = etlTask.next(hospitalsCategoriesParallel)

const logGroup = new LogGroup(this, ‘automation-configuration-log-group‘, {
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
})

const stateMachine = new StateMachine(this, `${id}-state-machine`, {
definition,
timeout: Duration.minutes(5),
stateMachineName: ‘automation-configuration‘,
stateMachineType: StateMachineType.EXPRESS,
logs: {
destination: logGroup,
includeExecutionData: true,
level: LogLevel.ALL,
},
})

Após isso precisamos criar as rules para associar o event bridge com S3 e com a execução do nosso step function:

const s3EventRule = new Rule(this, ‘automation-config-s3-event-rule‘, {
ruleName: ‘automation-config-s3-event-rule‘,
})

const eventRole = new Role(this, ‘eventRole‘, {
assumedBy: new ServicePrincipal(‘events.amazonaws.com‘),
})

stateMachine.grantStartExecution(eventRole)
s3EventRule.addTarget(
new SfnStateMachine(stateMachine, {
input: RuleTargetInput.fromObject({
detail: EventField.fromPath(‘$.detail‘),
}),
role: eventRole,
}),
)
s3EventRule.addEventPattern({
source: [‘aws.s3‘],
detailType: [‘Object Created‘],
detail: {
bucket: {
name: [bucket.bucketName],
},
object: {
key: [
{
wildcard: ‘csv/automation-configuration/*.csv‘,
},
],
},
},
})

E por fim vamos criar uma regra para ouvir as falhas inesperadas que ocorrerem em nosso step function através do event bridge também, assim mantendo o caráter de orientação a eventos que tanto adoramos:

const unexpectedFailRule = new Rule(this, ‘exam-automation-config-unexpected-fail-rule‘, {
ruleName: ‘exam-automation-config-unexpected-fail-rule‘,
})
unexpectedFailRule.addTarget(
new SnsTopic(userNotificationTopic, {
message: RuleTargetInput.fromObject({
subject: ‘Exam Automation Config Failed 😥‘,
message: ‘Exam Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘it@somemail.com‘] },
}),
}),
)

unexpectedFailRule.addEventPattern({
source: [‘aws.states‘],
detailType: [‘Step Functions Execution Status Change‘],
detail: {
stateMachineArn: [stateMachine.stateMachineArn],
status: [‘FAILED‘, ‘TIMED_OUT‘, ‘ABORTED‘],
},
})

Colocando tudo junto em uma classe para que você possa entender todo o fluxo unificado:

import { Duration, Fn, RemovalPolicy } from ‘aws-cdk-lib‘
import { Rule, RuleTargetInput, EventField } from ‘aws-cdk-lib/aws-events‘
import { SfnStateMachine, SnsTopic } from ‘aws-cdk-lib/aws-events-targets‘
import { Role, ServicePrincipal } from ‘aws-cdk-lib/aws-iam‘
import { Alias, Architecture, IFunction } from ‘aws-cdk-lib/aws-lambda‘
import { NodejsFunction } from ‘aws-cdk-lib/aws-lambda-nodejs‘
import { LogGroup, RetentionDays } from ‘aws-cdk-lib/aws-logs‘
import { Bucket } from ‘aws-cdk-lib/aws-s3‘
import { Topic } from ‘aws-cdk-lib/aws-sns‘
import { Fail, LogLevel, Parallel, StateMachine, StateMachineType, TaskInput } from ‘aws-cdk-lib/aws-stepfunctions‘
import { LambdaInvoke, SnsPublish } from ‘aws-cdk-lib/aws-stepfunctions-tasks‘
import { Construct } from ‘constructs‘

export default class AutomationConfiguration extends Construct {
constructor(scope: Construct, id: string) {
super(scope, id)

const userNotificationTopic = Topic.fromTopicArn(
this,
‘userNotificationTopic‘,
Fn.importValue(‘infrastructure::sns::user-notification-topic::arn‘),
)

const bucket = new Bucket(this, ‘bucket‘, {
bucketName: ‘automation-configuration‘,
removalPolicy: RemovalPolicy.DESTROY,
})

const etlLambda = new NodejsFunction(this, ‘etl-lambda‘, {
entry: ‘src/lambda/etl/etl.handler.ts‘,
functionName: ‘etl‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

bucket.grantRead(
new Alias(this, id.concat(‘alias‘), {
aliasName: ‘current‘,
version: etlLambda.currentVersion,
}),
)

const categoriesLambda = new NodejsFunction(this, ‘insert-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const hospitalsLambda = new NodejsFunction(this, ‘hospitals-categories-lambda‘, {
entry: ‘src/lambda/categories/categories.handler.ts‘,
functionName: ‘insert-categories‘,
timeout: Duration.seconds(30),
architecture: Architecture.ARM_64,
})

const [etlTask, categoriesTask, hospitalsTask] = this.mountLambdaInvokes([
{ function: etlLambda, output: ‘$.Payload‘ },
{ function: categoriesLambda },
{ function: hospitalsLambda },
])

const errorTopicConfig = {
topic: userNotificationTopic,
subject: ‘Automation Config Failed 😥‘,
message: TaskInput.fromObject({
message: ‘Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘suporte@somemail.com‘] },
}),
}

const publishFailed = (publishFailedId: string) =>
new SnsPublish(this, `automation-config-sns-failed-${publishFailedId}`, errorTopicConfig)

const jobFailed = new Fail(this, ‘automation-config-job-failed‘, {
cause: ‘Unexpected Error‘,
})

const hospitalsCategoriesParallel = new Parallel(this, ‘auto-config-exams-parallel-map‘)
.branch(categoriesTask)
.branch(hospitalsTask)
.addRetry({ errors: [‘States.ALL‘], interval: Duration.seconds(5), maxAttempts: 1 })
.addCatch(publishFailed(‘exams‘).next(jobFailed), {
errors: [‘States.ALL‘],
})
const definition = etlTask.next(hospitalsCategoriesParallel)

const logGroup = new LogGroup(this, ‘automation-configuration-log-group‘, {
retention: RetentionDays.ONE_WEEK,
removalPolicy: RemovalPolicy.DESTROY,
})

const stateMachine = new StateMachine(this, `${id}-state-machine`, {
definition,
timeout: Duration.minutes(5),
stateMachineName: ‘automation-configuration‘,
stateMachineType: StateMachineType.EXPRESS,
logs: {
destination: logGroup,
includeExecutionData: true,
level: LogLevel.ALL,
},
})

const s3EventRule = new Rule(this, ‘automation-config-s3-event-rule‘, {
ruleName: ‘automation-config-s3-event-rule‘,
})

const eventRole = new Role(this, ‘eventRole‘, {
assumedBy: new ServicePrincipal(‘events.amazonaws.com‘),
})

stateMachine.grantStartExecution(eventRole)
s3EventRule.addTarget(
new SfnStateMachine(stateMachine, {
input: RuleTargetInput.fromObject({
detail: EventField.fromPath(‘$.detail‘),
}),
role: eventRole,
}),
)
s3EventRule.addEventPattern({
source: [‘aws.s3‘],
detailType: [‘Object Created‘],
detail: {
bucket: {
name: [bucket.bucketName],
},
object: {
key: [
{
wildcard: ‘csv/automation-configuration/*.csv‘,
},
],
},
},
})

const unexpectedFailRule = new Rule(this, ‘exam-automation-config-unexpected-fail-rule‘, {
ruleName: ‘exam-automation-config-unexpected-fail-rule‘,
})
unexpectedFailRule.addTarget(
new SnsTopic(userNotificationTopic, {
message: RuleTargetInput.fromObject({
subject: ‘Exam Automation Config Failed 😥‘,
message: ‘Exam Automation Config Failed due to an unexpected error.‘,
cause: ‘Unexpected Error‘,
channel: ‘email‘,
destination: { ToAddresses: [‘it@somemail.com‘] },
}),
}),
)

unexpectedFailRule.addEventPattern({
source: [‘aws.states‘],
detailType: [‘Step Functions Execution Status Change‘],
detail: {
stateMachineArn: [stateMachine.stateMachineArn],
status: [‘FAILED‘, ‘TIMED_OUT‘, ‘ABORTED‘],
},
})
}

private mountLambdaInvokes(
lambdasInvoke: Array<{
function: IFunction
name?: string
output?: string
}>,
) {
return lambdasInvoke.map(lambdaInvoke => {
return new LambdaInvoke(this, `${lambdaInvoke.name || lambdaInvoke.function.functionName}-task`, {
lambdaFunction: lambdaInvoke.function,
inputPath: ‘$‘,
outputPath: lambdaInvoke?.output || ‘$‘,
})
})
}
}

Please follow and like us:
fb-share-icon
Tweet
Pin Share

Stiri similare

Python vs Java: A Deep Dive into the Best Programming Language for You

Python vs Java: A Deep Dive into the Best Programming Language for You

Claudio Ctin

Hey everyone! How’s your week going? 😊 Whether you’re in the middle of a coding marathon, enjoying a well-deserved break, or just here to explore new tech ideas, we’re happy to have you. Today, we’re diving into a hot topic: Python vs. Java. 🚀 These two programming giants are often at the center of debates,…

RMAG news

Using TypeORM with TSX: A Smoother Development Experience

Claudio Ctin

Working with TypeScript in Node.js applications can sometimes be challenging, especially when integrating with tools like TypeORM. While ts-node has been a popular choice for running TypeScript directly, it often comes with configuration headaches and performance bottlenecks. In this article, we’ll explore using tsx as an alternative to ts-node for TypeORM, providing a smoother and…

RMAG news

Next Js Localisation

Claudio Ctin

Hello All, I am new to this community and thank you for your support and time. I am working on a bilingual site (English and Japanese). I am using next-intl package to set the locale. I am trying to implement the following use cases Set the default language to user browser language Set the default…

RMAG news

Scale Deployments Based on HTTP Requests with Keda

Claudio Ctin

I’m using CloudWatch as a trigger, which means I have installed the CloudWatch agent responsible for sending metrics to CloudWatch. My KEDA fetches metrics based on the collection interval defined in the configuration of ScaledObject. You can follow the configuration below to set up the ScaledObject based on the number of HTTP requests. apiVersion: keda.sh/v1alpha1…

RMAG news

Human readable time in Rust

Claudio Ctin

When it comes to formatting time into the human-readable format in Rust, there could be various possible solutions including doing some math or using Instance. However, I demonstrate the millisecond crate, the dedicated and specialized crate for the purpose. This crate converts nanoseconds, microseconds, milliseconds, seconds, etc into short and long formats; suitable for human…

RMAG news

Introduction to the MERN Stack: Building Full-Stack Applications with Ease 🚀

Claudio Ctin

In the world of web development, the MERN stack has emerged as a powerful solution for creating dynamic and interactive applications. Comprising four key technologies—MongoDB, Express.js, React, and Node.js—this stack allows developers to build full-stack applications using JavaScript throughout. In this blog post, we’ll explore how these technologies work together, their pros and cons, and…

RMAG news

NEXT-JS PROJECT

Claudio Ctin

REVIEW In the course of two weeks I have gained skills needed to build a full-stack Next.js applications.On this project we were expected to build a full-stack web application which should have a public home page, login page, dashboard pages and setting up database. REQUIREMENTS . Operating system; MacOS, Windows or Linus . Node.js 18.17.0…

RMAG news

Special Types of Interviews: How to Identify and Succeed in Them..

Claudio Ctin

When preparing for interviews, it’s essential to recognize that not all interviews are alike. Some interviewers ask questions that follow specific patterns, and failing to identify these can make the process more challenging. Understanding these distinct types of interviews can help you better prepare and succeed. Let’s dive into the three main types: 1) Behavioral…

RMAG news

Essential Tips for Aspiring Web Developers

Claudio Ctin

Web development is an ever-evolving field that offers endless opportunities for creativity and innovation. Whether you’re just starting or looking to refine your skills, these tips will help you confidently navigate the world of web development. 1. Practice Regularly: Consistent practice is key to mastering web development. Set aside time each day to code, experiment…

RMAG news

Enhancing Healthcare Outcomes with Oracle Cloud Infrastructure: A Path to Better Results

Claudio Ctin

In today’s rapidly changing healthcare environment Leveraging advanced technology is critical to improving patient outcomes and operational efficiency. Oracle Cloud Infrastructure (OCI) offers a comprehensive suite of solutions designed to transform healthcare delivery by Improved data management Strong security guarantees and advanced analytics This article dives into how OCI drives better healthcare outcomes and why…

RMAG news

Mastering React: A Comprehensive Learning Guide

Claudio Ctin

Welcome to a thorough journey through React and its ecosystem! This guide will provide you with a structured path for mastering React, from understanding basic concepts to exploring advanced topics in React and associated technologies. Getting Started with React React is a powerful JavaScript library for building user interfaces, mainly for single-page applications where you…

RMAG news

IaC Security Analysis: Checkov vs. tfsec vs. Terrascan – A Comparative Evaluation

Claudio Ctin

Traditional, manual security processes can’t keep up with the speed of modern development, which leaves systems vulnerable to attacks. That’s where Security as Code (SaC) comes in. SaC automates security checks and policies, making them an integral part of the development pipeline. This ensures that security is built into every step without slowing down progress….

Terraform – Overview

Terraform – Overview

Claudio Ctin

What is Terraform? 🤔 Terraform is an open-source, cloud-agnostic, one of the most popular Infrastructure-as-code (IaC) tool developed by HashiCorp. It is used by DevOps teams to automate infrastructure tasks such as provisioning of your cloud resources. Terraform supported immutable infrastructure, a declarative language, a masterless and agentless architecture, and had a large community and…

RMAG news

Why is Nextjs so popular in startups?

Claudio Ctin

Nextjs is quite popular with startups. What are the main features of Next.js that make startups adopt it? Please follow and like us:

10 Essential Shadcn Components Every Developer Should Know About

10 Essential Shadcn Components Every Developer Should Know About

Claudio Ctin

Living in the web development world, it may just feel like digging for treasure until you find your tools. If you’re just starting now, well, you’re in luck because Shadcn is packed full of awesome components that can make life so much easier. If you are a professional or a beginner, knowing these 10 essential…

RMAG news

Execute Syntax on Background Process

Claudio Ctin

Pada postingan ini kita akan belajar cara remot ssh dan menjalankan perintah di balik layar atau background proses dengan 2 tools. Tools yg pertama akan kita bahas adalah tmux dan yg kedua adalah screen. Kedua alat tersebut adalah terminal multiplexer yang digunakan untuk multi tasking di terminal Linux. Okeh kita bahas cara penggunaanya. Saya asumsikan…

RMAG news

Scala is one of the best ways to learn Haskell

Claudio Ctin

First, a reminder: You DON’T NEED TO learn Haskell — that’s not what I want to talk about. I genuinely want to talk about learning Haskell, and Haskell specifically — not functional programming. The following might interest you if you want to get into Haskell for personal reasons (if you think it’s cool, well-paid, or…

What is the Difference Between Radix UI and ShadCN?

What is the Difference Between Radix UI and ShadCN?

Claudio Ctin

Be it a seasoned pro or a newcomer, UI library selection could make all the difference in any of your projects. Today, let’s take a deeper look at two of the top contenders: Radix UI and ShadCN. Both these libraries have something different to bring to the table, and by the end of this post,…

RMAG news

AuthorizationEndpoint vs TokenEndpoint

Claudio Ctin

In OAuth 2.0, the AuthorizationEndpoint and TokenEndpoint serve different roles in the process of obtaining access to resources on behalf of a user. Here’s a breakdown of the differences between them: 1. Authorization Endpoint Purpose: The AuthorizationEndpoint is responsible for obtaining authorization from the user to access their resources. This is where the user grants…

Custom Bootstrap 5 Breadcrumbs -Ver 2

Custom Bootstrap 5 Breadcrumbs -Ver 2

Claudio Ctin

Custom Breadcrumbs for Bootstrap 5 framework Abstract: We are presenting code (CSS) for custom Bootstrap 5 breadcrumbs. This is an improved version of the previously published article. 1 The need for better Breadcrumbs Bootstrap 5 framework is coming with very basic Breadcrumbs implementation. I needed something much better, both visually and more functional. Over time,…

RMAG news

A Comprehensive Guide to Appium 2.0 Migration

Claudio Ctin

In mobile app development, keeping up with the latest advancements in testing tools is crucial. A significant upgrade in automation testing is the release of Appium 2.0. With its myriad enhancements and new features, migrating to Appium 2 is essential for teams aiming to maintain a competitive edge. This Appium 2.0 Migration blog will guide…

FluxCD – A lightweight GitOps CD tool: Day 44 of 50 days DevOps Tools Series

FluxCD – A lightweight GitOps CD tool: Day 44 of 50 days DevOps Tools Series

Claudio Ctin

Welcome to Day 44 of our “50 DevOps Tools in 50 Days” series! Today, we are exploring FluxCD, one of the most popular tools in the GitOps ecosystem. FluxCD automates the process of deploying applications to Kubernetes, allowing for seamless continuous delivery and progressive deployments. It uses Git as the single source of truth, ensuring…

🌐 SSL Certificates and How to Implement Them in Your Website 🔐

🌐 SSL Certificates and How to Implement Them in Your Website 🔐

Claudio Ctin

In today’s web development world, ensuring security is critical. As developers, safeguarding user data and building trust with our audience are essential. One powerful way to achieve this is by implementing an SSL (Secure Sockets Layer) certificate on our websites. 🔒 In this post, we’ll dive into: What SSL is 🤔 Why it’s important 🌟…

Spring Boot RestTemplate getForEntity method

Spring Boot RestTemplate getForEntity method

Claudio Ctin

Producer application Dependencies Spring Web and Lombok pom.xml <?xml version=”1.0″ encoding=”UTF-8″?> <project xmlns=”http://maven.apache.org/POM/4.0.0″ xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd”> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.2.5</version> <relativePath/> <!– lookup parent from repository –> </parent> <groupId>com.example</groupId> <artifactId>ProducerApp</artifactId> <version>0.0.1-SNAPSHOT</version> <name>ProducerApp</name> <description>Demo project for Spring Boot</description> <properties> <java.version>17</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency>…

RMAG news

Mastering Terraform: How Loops and Conditionals Unlock Smarter Infrastructure Automation

Claudio Ctin

In the ever-growing world of cloud infrastructure, managing and automating deployments has become increasingly complex. This is where Terraform comes in — as one of the most powerful tools in Infrastructure as Code (IaC), Terraform allows engineers to declaratively define and provision infrastructure with ease. However, as configurations grow, keeping them concise and manageable is…

The Power of Machine Learning in Angular: A Developer’s Guide

The Power of Machine Learning in Angular: A Developer’s Guide

Claudio Ctin

The intersection of Machine Learning (ML) and Artificial Intelligence (AI) with Angular is a game-changer for modern web applications. This article delves into how you can integrate AI and ML into your Angular projects, breaking it down with practical examples, real-world use cases, and helpful tips to guide you along the way. please subscribe to…

RMAG news

Introducing HTPX: A Lightweight and Versatile HTTP Client for JavaScript and Node.js

Claudio Ctin

As developers, we often need a reliable and efficient HTTP client for our web applications, whether we’re building with JavaScript in the browser or Node.js on the server side. That’s why I created HTPX — a powerful, lightweight solution designed to simplify HTTP requests while offering a range of features for modern development. In this…

RMAG news

Advanced Terraform Module Usage: Versioning, Nesting, and Reuse Across Environments

Claudio Ctin

Terraform, HashiCorp’s popular Infrastructure as Code (IaC) tool, has transformed how infrastructure is provisioned and managed. One of its most powerful features is the ability to create and use modules — reusable configurations that encapsulate sets of resources. For teams looking to scale and manage their infrastructure efficiently, mastering advanced Terraform module usage — including…

10 sec for UX Design Principles.

10 sec for UX Design Principles.

Claudio Ctin

DFFH RULE 1.D- Doherty Threshold : Users somehow replicate the computer’s working speed to navigate the components in the design. Tend to keep the waiting time for users at less than 400 ms. The loading time can be utilized to show visual animations of loading or task-processing phenomena to build trust. 2.F- Fitt’s Law :…