Locks and Leaders with Spring Integration

Dave Syer, 2016
Twitter: @david_syer (see also @artem_bilan, @gprussell)
Email: dsyer@pivotal.io

Agenda

Warning: Here be Dragons!

A Simple Distributed Operation

basic-sequence-0

Two Clients

basic-sequence-1

Shuffle!

basic-sequence-2

Shuffle!

basic-sequence-3

Shuffle!

basic-sequence-4

Shuffle!

basic-sequence-4

Shuffle!

basic-sequence-4

Shuffle!

basic-sequence-4

Scary, Eh?

basic-sequence-5

Locks

Example code using java.util.concurrent.locks.Lock:

boolean acquired = false;
try {
  acquired = lock.tryLock(10, TimeUnit.SECONDS);
  if (acquired) {
    // Do something unique!
  }
} catch (InterruptedException e) {
  Thread.currentThread().interrupt();
  throw new RuntimeException("Interrupted");
} finally {
  if (acquired) {
    lock.unlock();
  }
}

Spring Integration: LockRegistry

public interface LockRegistry {
    Lock obtain(Object lockKey);
}

Locks with Spring Integration

Example code using LockRegistry:

boolean acquired = false;
try {
  acquired = lock.tryLock(10, TimeUnit.SECONDS);
  if (acquired) {
    // Do something unique!
  }
} catch (InterruptedException e) {
  Thread.currentThread().interrupt();
  throw new RuntimeException("Interrupted");
} finally {
  if (acquired) {
    lock.unlock();
  }
}

(same code)

Locks and Leases

A distributed lock nearly always has a shelf life (it expires).

Technically, that makes it a "lease".

Without expiry system can't make progress when a lock holder dies.

Dragons

(At least) two problems are lurking:

  1. Acquiring a lock requires consensus.
  2. Leases expire and holder can never be sure when that happens.

Read this: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

TL;DR If using a lock for "correctness" not "efficiency" be very careful.

Public Service Announcement

Important: you can tune the system to adjust the probability, or how long it lasts, but fundamentally you cannot prevent the system from ever allowing more than one holder of a lock.

Leader Elections

Simple idea: if you hold a lock you are the leader.

What can you do with it?

Highly available globally unique things, often with messages

Spring Integration: Leader Initiator

Implementations of leader election need to be able to start an election and fire events on granted and revoked.

For a user it looks like this (create a new bean which is a SmartLifecycle):

@Bean
public LeaderInitiator leaderInitiator(CuratorFramework client,
            Candidate candidate) {
  return new LeaderInitiator(client, candidate);
}

(*) No support for etcd v3. Probably dead.

Spring Integration: Callbacks

Callbacks on leadership events:

public interface Candidate {
    void onGranted(Context ctx) throws InterruptedException;
    void onRevoked(Context ctx);
    ...
}

See also:

@EventListener(OnGrantedEvent.class)
public void start() {

}

@EventListener(OnRevokedEvent.class)
public void stop() {

}

Closing

#