« Coupling in a RESTful Way | Main | How Not to Plan a Career in Software »

Thursday, December 28, 2006



Dan: If you assume the existance of a reliable and fault-tolerant message queuing infrastructure, why doesn't your design simply inject transactions into the queue and have queue consumers pull requests off, handle the cross-database work and on failure handle rollback and requeueing of a message?

Oh, we've ended up with a flavor of 2PC again (queue message, commit transaction). I think saying "we're avoiding 2PC" on one hand and saying "yes, we'll use a reliable, fault-tolerant message queue" on the other is kind of cheating. Maybe I'm wrong?

What happens when we boil down the problem to only having three basic operations: read, write and move (rename)? These are your typical atomic I/O operations, right? What if we had "unstoppable transactions"--think about your typical journal. The system state is determined at read-time, using either first-in-wins (FIW) or last-in-wins (LIW)--or, some other arbitrary but unambiguous means. This uses more storage (as every transaction is stored) but avoids 2PC, assumes your persistent store IS your message queue (in essence), messages are either lost (!) but never requeued (a message lost is the same as a failure failing to be queued, anyhow), and assuming a sane journaling implementation should be O(1) to determine state at read-time in JIT fashion.

Hmm, I think this won't work because it assumes (relies upon?) having robust synchronous writes across the distributed systems. Oops. :-) But, it sure was fun to think about!

Thanks for the brain food.

Dan Pritchett

The message queue can be made reliable via horizontal scaling on the queuing side. I can avoid 2PC on the dequeue by relying upon idempotent handling of the events. I refer to that as at least once delivery of the message.

You did hit upon something I didn't state but have employed. The message queue IS the persistent store. A table and a queue are not very different!


Dan, if I understand correctly, what you're saying is:

1) Identify atomic operations (primitives) on each of the distributed systems.

2) For bonus points, design the system such that those primitives are idempotent.

3) Design your higher-order transactions to only be a composite of those primitives.

4) Lather, rinse, repeat.

Dan Pritchett

Wow, you summed up two articles in 3 bullets. Well done!

Jason Watkins

It's not clear to me how the reservations and item counts are reconciled. For example, if we have only one count left in the items database, and two in flight transactions both log 1 item reservations, how does this get resolved? Does it depend on who gets to the item count updating first, and then the loser removes it's previous work? Or is one of the reservations some how prevented when it's insert is attempted?

Dan Pritchett

Sorry I didn't explain that very well. The idea is that first you retrieve the item count and all the current reservations. If count - sum(reservations) is positive, you're free to insert a reservation for your quantity.

Once you've done that, you have to immediately retrieve count and sum(reservation) again. If the result is now negative, the newest reservations have to be removed until the count is again positive. Ideally each instance only removes their reservation.

Jason Watkins

Thanks, that makes it clear.

So would it be fair to say, you have to design your system as a series of communicating state machines, where the actions undertaken on transitions are idempotent or reversible? That way rather than locking you can use an optimistic approach and deal with interference explicitly rather than requiring blocking to serialize for you (similar to the TimeWarp algorithm).

Dan Pritchett

Yes, is fair to say.

Noah Campbell

TLA warning: Idempotent Orientated Architecture or IOA for scalable architecture. :)

Dossy: You've described the fundamental of a transaction in three bullet points. The only additional item I'd add is to identify which operations are non-essential in case of failure. In this case, the writing of the user preferences.

The scenario also describes a best effort attempt to accomplish the transaction in memory and if there's a failure, a reaper goes into effect and tries to reconcile. This is definitively a compensating transaction.

Is it possible to accomplish this same task without a reaper or a reservation table?

ped ant

I think you meant to say "piquing the interest", not "peaking".


John Pailing

Sorry I don't understand the final preferences message queue. Where is this placed? Is this on the same system as the transaction database, or on the reservations system?

As far as I can tell that this must be part of or the same as the message queue for the transaction recording.

So the process that is checking the completed transaction queue must be both checking the reservations have been removed and the preferences have been saved?

Or are there 2 queues?

The comments to this entry are closed.