AWS SQS: The Ultimate Decoupling Solution
Overview
From the time since cloud computing came into existence, it has been revolutionizing and coming up with new services and features for the ease and satisfaction of its clients
Message Queues are at the heart of modern software engineering and one of the most vital middle-ware components. As startups and enterprise needs to focus on their business instead of setting up the infrastructure and various other software components, cloud providers such as AWS has always been the rescue for them.
There is no denying to the fact that in today’s modern serverless and microservices architecture, message queues plays a vital role and in fact, are necessary for the working of application following these architectures.
That’s why cloud providers such as AWS has come up with their own managed solution for a message queue. AWS Sqs is amazon’s managed and hosted queue solution for their customers.
So let’s find out what a message queue is first and then we will see what Sqs can do for us.
What is a message queue?
As the name suggests this thing looks like a queue or a container for some kind of message. But what is it in terms of software engineering and what is its purpose?
Message queues and mailboxes are software-engineering components typically used for inter-process communication (IPC), or for inter-thread communication within the same process. They use a queue for messaging — the passing of control or of content
Messages are stored on the queue until they are processed and deleted. Each message is processed only once, by a single consumer. Message queues can be used to decouple heavyweight processing, buffer, or batch work, and to smooth spiky workloads.
Now let’s take a closer look at their work and specifically the ‘decoupling’ part.
In modern architecture, applications are decoupled into smaller, independent building blocks that are easier to develop, deploy and maintain. Message queues provide communication and coordination for these distributed applications. Message queues can significantly simplify the coding of decoupled applications while improving performance, reliability, and scalability.
Message queues allow different parts of a system to communicate and process operations asynchronously. A message queue provides a lightweight buffer that temporarily stores messages, and endpoints that allow software components to connect to the queue in order to send and receive messages. The messages are usually small and can be things like requests, replies, error messages, or just plain information. To send a message, a component called a producer adds a message to the queue. The message is stored on the queue until another component called a consumer retrieves the message and does something with it. Many producers and consumers can use the queue, but each message is processed only once, by a single consumer.
AWS SQS: An Introduction
After seeing the what and why of a message queue and probably understanding how important its role is in the modern application architecture, let’s take a look at AWS Sqs.
Amazon Simple Queue Service (Sqs) is a managed message queue service offered by Amazon Web Services (AWS). It provides an HTTP API over which applications can submit items into and read items out of a queue. The queue itself is fully managed by AWS, which makes Sqs an easy solution for passing messages between different parts of software systems that run in the cloud.
It offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon Sqs offers common constructs such as dead-letter queues and cost allocation tags. It provides a generic web services API that you can access using any programming language that the AWS SDK supports.
The messages that Sqs handles can be unformatted strings, XML or JSON. Because Sqs guarantees “exactly once” delivery, and because you can concurrently submit messages to and read messages from a given queue, Sqs is a good option for integrating multiple independent systems.
SQS and Serverless architecture
In an optimal Serverless architecture, when using multiple microservices, you want your services to be independent of each other. This means that they use separate databases and even run in different network segments but can still communicate with each other. The distributed systems term for this kind of independence is a decoupled system. When services are decoupled, you can’t pass messages, for example, by having them both write to a shared database. The available options under this model are: implement an API in each service or use message queues to pass information between services.
Message queues tend to be a more lightweight approach and conducive to faster iteration and aside from being a queue system that helps you iterate faster, Sqs is also fully managed. This makes it a great choice for Serverless applications running in AWS
Benefits of SQS
Scalability Your Sqs queues scale to the volume of messages you’re writing and reading. You don’t need to scale the queues; all the scaling and performance-at-scale aspects are taken care of by AWS.
Pay for what you use When using Sqs, you only get charged for the messages you read and write (see the details in the Pricing section). There aren’t any recurring or base fees.
Ease of setup Since Sqs is a managed service, so you don’t need to set up any infrastructure to start using Sqs. You can simply use the API to read and write messages or use the Sqs <-> Lambda integration.
Security You control who can send messages to and receive messages from an Amazon Sqs queue. Server-side encryption (SSE) lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
Durability For the safety of your messages, Amazon Sqs stores them on multiple servers. Standard queues support at least one message delivery, and FIFO queues support exactly one message processing.
Availability Amazon Sqs uses redundant infrastructure to provide highly concurrent access to messages and high availability for producing and consuming messages.
AWS SQS and Oyster: A Case Study
Finally, let’s take a look at how a real company is leveraging the power of AWS Sqs for the benefit of their business.
What is an Oyster?
New York-based Oyster.com shares unvarnished reviews of hotels in nearly 200 destinations worldwide. The company’s own investigators visit each location to assess cleanliness, amenities, service, and overall quality. What sets Oyster apart from similar sites is its extensive collection of photographs. Oyster takes hundreds of photos for each property, and every review includes dozens of untouched images (submitted by guests as well as investigators) that allow potential visitors to compare a hotel’s marketing material with reality.
The huge amount of photos that they need to capture and process soon became a huge problem for the company.
The Problem
Since its 2009 launch, Oyster has published more than one million high-quality digital images. When this massive volume of images became too cumbersome to handle in-house, the company decided to offload the content to a central repository on Amazon Simple Storage Service (Amazon S3). “We migrated to Amazon S3 in 2010,” says Eytan Seidman, Co-Founder and Vice President of Product. “We chose moving to the cloud and Amazon S3 because storing images in our data center would have been too costly. Amazon S3 was a more economical solution.”
Oyster reprocesses its entire collection of photographic images a few times each year to update the copyright year and, if necessary, to change the watermarks. Using their previous solution, reprocessing the entire collection of photographs required about 800 hours to complete. In addition, Oyster often recreated existing images in new formats and sizes for mobile and tablet devices. Resizing existing images and adding new ones was slowing down the rate at which the company was able to process the collection. “Our processes were slowing down,” says Seidman. “When the iPad with Retina display came out, for example, it took us more than a week to create new sizes specifically for that resolution.” Oyster considered purchasing additional hardware but found the cost of new hardware and routine maintenance was too high, especially when the machines would sit idle most of the time.
Moreover, there were numerous software bugs in the multiprocessing solution that the company used, but since the solution didn’t scale, Oyster didn’t bother to fix them
AWS Sqs and EC2 the saviors
While the company is still running one local server, the bulk of the processing work now takes place on the AWS Cloud. Oyster is using a customized Amazon Linux AMI within Amazon EC2. Within this new environment, the company connects to Amazon S3 and Amazon Simple Queue Service (Amazon Sqs) using boto, a Python interface to AWS. The images themselves are processed with the ImageMagick software available in the AMI package.
Oyster uses Amazon EC2 instances and Amazon Sqs in an integrated workflow to generate the sizes they need for each photo. The team processes a few thousand photos each night, using Amazon EC2 Spot Instances. When Oyster processes the entire collection, it can use up to 100 Amazon EC2 instances. The team uses Amazon Sqs to communicate the photos that need to be processed and the status of the jobs.
The Benefits
Oyster’s old system needed approximately 400 hours to process one million photos. By using AWS, the company can process the same number of photos in about 20 hours — a 95 percent improvement. “It took less time to rewrite the code and do a full processing job with AWS than it took to do a single run with the old method,” says Seidman. “It used to take close to a week to produce photos specifically for the iPad. With AWS, we can create the photos in just a few hours. The documentation is straightforward and the dashboards are incredibly helpful.”
So in brief the benefits are:-
- Reduced time to process photos by 95%
- Reduced in-house hardware expenses
- Saves $10,000 in capital expenditure and reduced operating expenses by an additional $10,000
Conclusion
Message Queues are an important component for the modern serverless and microservices architecture to function smoothly. AWS Sqs provides a managed queue service that provides lots of cool features which are leveraged by enterprises for their solutions.
