Consistent Messaging in the Cloud

Consistent Messaging in the Cloud

Speakers: Szymon Pobiega & Tomasz Masternak

 

Abstract:

How to build fault-tolerant distributed systems when throwing away consistency is not an option

The workshop focuses on building line-of-business fault-tolerant cloud-based distributed systems. Such systems cannot afford to lose messages (nobody wants their order for Christmas gifts to be lost) nor to get them duplicated (that second Porsche in the drive way — who ordered that?). Such systems were, in the past, built based on the firm ground established by either distributed transactions or large database instances that served also as messaging brokers.

These technologies are either too expensive, too cumbersome or simply not available in the cloud. In this workshop we will show how one can deal with the consistent messaging problem by de-duplicating messages.

We’ll start by asking ourselves a question why the systems we build need to be distributed. We’ll see how duplicating messages is the only way to get components to reliably exchange information. Finally, we’ll spend most of our time identifying subtle issues inherent to message processing, devising solutions to these issues and encoding these solutions in reusable patterns.

Join me in the series of ten hands-on exercises interleaved with short lectures after which you’ll have good understanding of most of the things that can go wrong when processing messages and enough knowledge to either build build a bullet-and-duplicate-proof message processor or (even better) find a framework that implements one for you.


Objectives:

After the workshop you will know that

  • Idempotence is not a medical condition
  • Idempotence by itself does not guarantee the correctness of message processing
  • Correct message processing requires consistency not only with regards to data changes but also to generation of outgoing messages
  • In the Cloud you need to take care of consistent messaging yourself. There is no infrastructure such as DTC that can help you
  • Consistent messaging can be achieved by storing each version of state or by storing outgoing messages
  • Outbox pattern is a simple way of implementing consistent messaging
  • Outbox has significant limitations
  • Outbox can be extended with Inbox to solve the problem of storing an infinite amount of information about messages already processed
  • Inbox can be turned into a Token Store to remove the need to store information about messages already processed
  • Transactionaly consistent RDBMS-based transport can be used instead of de-duplication
  • Blob storage techniques (S3, Blog Storage) have different consistency guarantees

Topics covered:
  • Message de-duplication
  • Outbox Pattern
  • Inbox Pattern
  • Efficient usage of RDBMS as a queue
  • Consistency guarantees

Intended audience:

Senior developers, technical leads and software architects


Required equipment:

Laptop


Required software:
  • VisualStudio 2019 (any edition, including Community)
  • SQL Server (any edition, can be Azure SQL)

Workshop type:

60% hands-on exercises, 40% lectures