Team update: A lot has happened since this blog - all great things! We’re now in the United States and our new product launched in November 2021, helping teams in fast-growing organizations find and hire their best-fit junior and mid-level talent in Sales, Marketing, Operations, and Customer Success. Try it here for free. This means some of our articles before this date may have product shots that look a little different. That’s all from us, enjoy the blog.

‍

The team at Hatch spun up the Labour Exchange in a few days re-purposing our tech to help stood down workers find employment during the Covid-19 crisis.

In order to get the system up and running in such a short time frame we decided to use S3 as a flat-file data store to maintain our serverless batch job states and caches. After a very cursory search to satisfy ourselves S3 would guarantee read-after-write consistency, we flew on.

From the documentation

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all Regions with one caveat. The caveat is that if you make a HEAD or GET request to a key name before the object is created, then create the object shortly after that, a subsequent GET might not return the object due to eventual consistency.

Unfortunately, we missed the bolded section and the part further down the page that explicitly states our use case:

A process replaces an existing object and immediately tries to read it. Until the change is fully propagated, Amazon S3 might return the previous data.

As these processes run on a schedule (not as part of a user facing API) we could afford to spend some extra calls to S3 to roll our own read-after-update consistency. Knowing that S3 guarantees read-after-firstWrite, we can write a new file for every change, read the latest file and make sure we cleanup.

So every time we write a file we:

• Append a timestamp to the filename
• Remove older files

When we read a file we:

• List all files with the key prefix (S3 guarantees listing files will be ordered by ascending UTF-8 binary order)
• Get the newest file in the list

‍

‍