Amazon Simple Queue Service (SQS) is a distributed message queuing service that supports asynchronous tasks. It enables developers to manage asynchronous communication between applications or microservices through a polling (pull-based) model. Amazon SQS supports two types of message queues: Standard queues and FIFO (first-in-first-out) queues.
SQS Standard queues (default) offer unlimited throughput, at-least-once delivery, and best-effort ordering. SQS FIFO queues complement the Standard Queue by supporting high throughput, exactly-once processing, and first-in-first-out delivery.
With SQS, developers can build distributed applications using decoupled components and transmit any volume of messages/data between software components, without losing messages or requiring other services to be available.
How did Amazon SQS come about?
Amazon Simple Queue Service (SQS) is a messaging queuing service that was launched for public usage by Amazon in November 2004. It was developed to provide a highly scalable hosted message queue that can handle issues caused by the common bounded-buffer problem. However, SQS has much wider messaging middleware applicability.
Amazon SQS offers Software Development Kits (SDKs) for developers in several programming languages such as C++, Go, Node.js, Java, Ruby, Python, .NET, PHP and JavaScript. In December 2014, Amazon released a Java Message Service (JMS), 1.1 client, for Amazon SQS that provides generic messaging models capable of addressing the producer-consumer problem. Developers can use JMS to send and receive messages between software systems.
Today, Amazon SQS is used by Lyft, Stack, Medium, Coursera, Accenture, Typeform, Barogo, among others.
How does Amazon SQS work?
As with most messaging middleware, Amazon SQS consists of three major components: producers (components that send messages to the queue), the queue (which stores messages across multiple Amazon SQS servers), and consumers (other components that receive messages from the queue).
With SQS, a producer sends a message to a queue for redundant distribution across the Amazon SQS servers. A consumer that’s ready to process messages receives the message from the queue. The message that’s being processed remains in the queue and it isn’t returned to subsequent receive requests until the message visibility timeout lapses. The consumer deletes the message from the queue after processing to prevent the message from being visible again in the queue after the visibility timeout.
Note: With Amazon SQS, multiple consumers can’t receive the same message simultaneously. A message can only be received from a queue by one consumer that’s ready to process and then delete the message received. A message can be retained in queues for 14 days maximum.
To better understand how Amazon SQS works, you have to be familiar with the following important terms:
Amazon SQS Visibility Timeout: This is the period of time that a message received from a queue by one consumer won’t be visible to the other consumer. Visibility timeout for a message ranges between 0 seconds to 12 hours, with the default set at 30 seconds. You can check Configuring queue parameters (console) to learn about how to set up a message visibility timeout using the console.
Dead-Letter Queue: This is a queue that other source queues can target for messages that fail to process correctly . Whether it’s client or server errors, failed messages are moved to a dead-letter queue for further checks, analysis, and reprocessing. You can check Configuring a dead-letter queue (console) to learn how to use the Amazon SQS console to set up a dead-letter queue.
Note: You need to first create the queue before you can use it as a dead-letter queue because Amazon SQS doesn’t create the dead-letter queue automatically.
Amazon SQS Short and Long Polling: These are the two ways of message processing in Amazon SQS. Short polling is used by default and it occurs when the waiting time equals 0 (zero). With the short polling, you get an immediate response (even with no message found) because the ReceiveMessage request queries only a subset of the servers (where the queue stores and processes messages) to find available messages to include in the response.
However, it changes to long polling when the waiting time for the ReceiveMessage API action is greater than 0 (zero). With long polling, it takes more time to return a response because the ReceiveMessage request queries all of the servers for messages and waits until a message arrives in the queue or the message long poll times out. A long polling frequency can be set between 1 to 20 seconds.
Technical Overview of Amazon SQS
From a technical standpoint, Amazon SQS features five basic APIs for developers to get started with. These include: CreateQueue, SendMessage, ReceiveMessage, ChangeMessageVisibility, and DeleteMessage. There are also extra APIs for advanced functionality. With SQS APIs, serverless developers can directly implement an SQS queue with any other AWS service, including a third-party service, allowing them to work in most any programming language of their choice.
Amazon SQS supports message payloads comprising close to 256KB of text in any format. However, you can manage Amazon SQS messages content larger than 256KB (up to 2 GB) using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB.
Amazon SQS supports server-side encryption (SSE) for queues to allow the exchange of sensitive data between applications. SQS SSE uses the AES-256 GCM algorithm which is the 256-bit Advanced Encryption Standard to keep sensitive data secure using a unique encryption key. The encryption key is managed within the AWS Key Management Service (KMS). In addition, AWS KMS logs all your encryption key usage to AWS CloudTrail for regulatory and compliance requirements.
Amazon SQS is subscribed to HIPAA Compliance Program to allow developers to build HIPAA-compliant applications. Developers can use SQS to store and transmit messages between healthcare systems (including PHI-messages).
Amazon SQS integrates well with other AWS infrastructure web services, including Redshift, DynamoDB, RDS, EC2, ECS, Lambda, and S3 to increase the flexibility and scalability of your distributed application.
Amazon SQS Pros:
Fully-managed message queuing service
Provides good support for asynchronous processes
Options for standard and FIFO queues
A dedicated queue for unprocessed messages (dead-letter queue)
Integration with other AWS infrastructure web services
JMS compliant
Amazon SQS Cons:
Read More…
Amazon SQS Reviews & Ratings 2020
Amazon SQS – Reviews, Pros & Cons
What is Amazon Simple Queue Service?
Amazon Simple Queue Service (SQS) FAQS
Recommended Articles
Scalable, dependable chat applications with Apache Kafka and Ably
Learn how Ably complements and extends Kafka to end-users at the edge, enabling you to architect scalable, dependable chat apps.
Google Cloud Pub/Sub
Google Cloud Pub/Sub is a pub/sub-based messaging service.
Apache Kafka vs RabbitMQ vs AWS SNS/SQS: Which message broker is best?
Discover the key considerations for choosing a message broker and how to choose between Apache Kafka, RabbitMQ, and AWS SNS/SQS.