Solution for Python Boto3 not receiving messages but SQS shows in flight
is Given Below:
I’ve a docker which fetches messages from a standard SQS. But most of the times, the code shows it received zero messages and exits. While the SQS console shows the messages under “Messages in flight”, so the messages were received by some consumer.
This is my docker entry point
ENV PYTHONPATH="$PYTHONPATH:/app" ENTRYPOINT [ "python3" ] CMD ["multi.py"]
This is multi.py code
import multiprocessing as mp import subprocess def s(): subprocess.call(['python3', 'script.py']) n_process = min(mp.cpu_count(), 8) process =  for i in range(n_process): p = mp.Process(target=s) process.append(p) p.start() for p in process: p.join()
This is script.py part of the code which calls receive_messages
sqs = boto3.resource('sqs', region_name=REGION, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY) queue = sqs.get_queue_by_name(QueueName=QUEUE_NAME) def main(): while True: m = queue.receive_messages() for message in m: process_message(message) message.delete()
Also, the docker works like 60% of the time. But I’m trying to figure out why it fails.
This is from the boto3 docs
Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. Thus, only the messages on the sampled machines are returned. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response. If this happens, repeat the request.
m = queue.receive_messages(WaitTimeSeconds=5)
This will resolve the issue because in cases where there are very less amount of messages in SQS, polling for messages will be very likely to fail.
You can read about short-polling on boto3 docs here.