Name is required.
Email address is required.
Invalid email address
Answer is required.
Exceeding max length of 5KB

Matillion not listening to SQS queue

Hi!

I have an SQS FIFO queue that I want to use a trigger an orchestration job. I already have everything set up and the messages land in the queue, I set up the listener in Matillion, but every time the message is received nothing gets triggered in Matillion.

The message does have the correct format:

sqs_msg = {
"group": "BookSupportBI"
, "project": "ProjectInk"
, "version": "default"
, "environment": "BookSupportBICluster"
, "job": "ContactTraceRecords"
, "variables": {
"file_to_load": new_key
}
}

How can I check what's missing? I'm following this:

https://redshiftsupport.matillion.com/customer/en/portal/articles/2243961-triggering-etl-from-an-s3-event-via-aws-lambda

So if the messages land, why are they not getting picked up?

5 Community Answers

Matillion Agent  

Paul Johnson —

Hi Rodrigo,
Can you try with your variable value new-key quoted please so it reads “file_to_load”: “new_key”

Let me know if this resolves the issue please so i can update the documentation.

regards,
Paul


Rodrigo Carvajal Fernandez —

Hi!

New key is a string variable being passed in the lambda function to the message that will be sent to SQS, tha final SQS message does have a quoted string there.


Matillion Agent  

Kalyan Arangam —

Hi Rodrigo,

Thanks for the update.

We usually log any message we receive via SQS. Please download your logs from Admin menu and look for any occurrence JSON with “BookSupportBI” which might be the message you are sending.

Just to be sure, please visit Project→ManageSQSConfiguration and see if you’ve got the correct SQS Queue selected under Listen.

Another useful test would be to create a new orchestration job and add a SQS component to it. Then Copy/Paste the JSON from lambda replacing new_key with the appropriate filename. Check if running this job will trigger job – ContactTraceRecords.

Hope that helps.

Best
Kalyan


Rodrigo Carvajal Fernandez —

Thanks. The server logs pointed me to the right answer, the role attached to the Matillion instance did not have access to the KMS key that the queue is encrypted with.


Matillion Agent  

Kalyan Arangam —

Hi Rodrigo,

Glad you managed to resolve this. Are you okay for me to close this ticket?

Best
Kalyan

Post Your Community Answer

To add an answer please login