Dragonmark

Distributed core.async

Clojure eXchange 2014
David Pollak / @dpp

About @dpp

  • Founded Lift web framework
  • Wrote a bunch of commercial spreadsheets
  • Crazy passionate lawyer-trained tech dude

Dragonmark

  • Distributed core.async and associated libraries
  • Clojure and ClojureScript (nearly) unified API
  • An opinionated, philosophically based pile of parens

Dragonmark: the name

Not Spike from My Little Pony

Dragonmark: the name

from How to Train Your Dragon (the book).
The mark of those rebelling against Alvin the Treacherous.

Anti-Diaspora

Yes, this talk will contain a lot of seemingly random threads... I hope to pull it together in the end (and in under 50 minutes)

Back to Ponyville

->

Spike sends messages from Ponyville to Princess Celstia in Equestria... across address spaces...

Serialization

  • Conversion of data to stream of bytes, then...
  • Reify stream of bytes into data, such that...
  • The data has the same meaning

Objects

  • Objects have named "slots"
  • Some slots contain "data" and some "code" (methods)
  • How do you send an "object" from Ponyville to Equestria?

Huge Freakin' Problem

Mutable Objects

  • Dealing with a cyclical graph
  • Especially hard when graph can be mutated by another thread during serialization

Yay Clojure

  • Separation of data and schema (type) and behavior
  • Immutable data structures: not cyclical
  • Transit: Richer than JSON, cross platform serialization


We can serialize Clojure data and send it to another address space such that it will have the same meaning when it's reified!

And now for
Something completely different...

Thinking asynchronously...

Well...

Nothing is synchronous...

Exception some 'weird' quantum effects...

But...

We humans mostly gloss over asynchronousness.

Actors & Channels

  • Any => Unit: I'm going to the pub
  • X => LAFuture[Y]: Bruce, I'd like a hug!
  • Channels > Actor
    'cause backpressure and separation of concerns.
    Actors can be modeled with Channels,
    but not vice versa.

Linearizing the non-linear

core.async: linear instructions for non-linear execution

Spike: find the book
Pinky: Find Applejack
S&P: Look for gems

When S&P are done, Twightlight does stuff with gems, Applejack , and the book

aka the gofor macro

Yay! Finally! Some Dragonmark

gofor: for comprehension, go-style


(gofor
  [a (foo channel {:thing value :other_thing other_value})]
  :let [b (+ a 1)]
  [c (baz other_channel)
   d (moose third_channel {:p b}
   :timeout 45000]
  (println a b c)
  :error (println &err &at))
            
[a (foo channel {:thing value :other_thing other_value})]

sends a message to channel: 


{:_cmd "foo"
 :thing value
 :other_thing other_value
 :_return a_created_channel}

Simultaneous dispatch & timeout

Namespace -> Service

(def service-channel (dc/build-service ns))

In the namespace, we define functions:


(sc/defn ^:service get-42 :- sc/Num
    ([] 42)
    ([x :- sc/Num] (+ x 42)))

(sc/defn ^:service plus-one :- sc/Num
    "Hello"
    [x :- sc/Num]
    (+ 1 x))

Asking a service for documentation


docs (_commands service)]
:let [_ (println "Commands:\n" docs)


Commands:
{get-42 Inputs: ([] [x :- sc/Num])
Returns: sc/Num

Hello, plus-one Inputs: [x :- sc/Num]
Returns: sc/Num}

Recap (so far...)

  • Serialization mostly addressed in Clojure
  • Linearizing asynchronous execution easy with core.async
  • Dragonmark utilities:
    • turn a namespace into services
    • orchestrate complex message sends/receives

I like to be proactive

Dragonmark:
semantics of core.async

  • But across address spaces
  • Non-obvious to app
  • Transport independent

Boring Lift Comet rant

Plumbing

  • The best plumbing "just works"
  • Focus on taking a shower rather than where the water is coming from or going to
  • Pipes accessible but not in your face, so you can fix them when you need to or call a plumber do the fixing

Dragonmark: Why?

  • Isolate logic from transport for testing
  • faster dev cycles
  • Fewer developer context switches

Dragonmark: How?

  • Discover service
  • send messages

Demo

Setting up the Chat


(circ/gofor
 :let [other-root (circ/remote-root transport)]
 [the-chat-server (locate-service other-root {:service "chat"})]
 [_ (add-listener the-chat-server {:the-chan chat-listener})]
 (reset! chat-server the-chat-server)
 :error (.log js/console "Got error " &err " var " &var)
 )
  • get the remote root
  • locate the "chat" service
  • register as a listener

Boom!

Counting


(defn make-remote-calls
  []

  (circ/gofor
   [answer (inc (circ/remote-root transport))]
   (do
     (swap! app-state assoc :text (str "Remote count: " answer))
     (js/setTimeout make-remote-calls 1000)
     )
   ))


root (circ/build-root-channel
     {"get" (fn [msg env] @count-atom)
      "inc" (fn [msg env]
            (swap! count-atom inc))}

One small bit of magic


  • Special Transit serializer to create proxy for channels
  • thus channels can be sent across address spaces
  • and they have the same semantics

{:_cmd "foo"
 :thing value
 :other_thing other_value
 :_return a_created_channel}

Pluggable Transport


(defprotocol Transport
  "A transport to another address space"
  (remote-root
    "Returns a channel that is a way to send messages to
     the remote root"
    [this])
  (close!
    "Close the transport"
    [this])

  (proxy-info "GUID to Proxy functions" [this])

  (serialize "Serialize a msg" [this info])

  (deserialize "reify a msg" [this info]))

Other stuff

  • Backpressure maintained across address spaces
  • transport currently websockets with auto-reconnect
  • Rudimentary service discovery
  • "Service" functions can be marked remote or local (security)

Questions?

Dragonmark Repo: https://github.com/dragonmark