Generalities & Details: Adventures in the High-tech Underbelly

On (Haskell) Type Classes and (C#) Interfaces

Mon, 01 Mar 2010 04:52:59 GMT

Simon Peyton Jones was in town a couple weeks back to deliver a repeat of his ECOOP’09 keynote, “Classes, Jim, but not as we know them. Type classes in Haskell: what, why, and whither”, to a group of internal Microsoft language folks. It was a fantastic talk, and pulled together multiple strains of thought that I’ve been pondering lately, most notably the common thread amongst them.

In the talk, he compared polymorphism in Java-like languages (including C# which I will switch to referring to over Java hereforth) with ML and Haskell. In other words, how does a programmer commonly write code in each language that is maximally reusable? Of course, C# programmers primarily achieve this through subclassing, whereas functional programmers rely on type parameterization. Over the years, however, the former group has begun to borrow a great deal from the latter; as evidence, witness the growingly-pervasive use of generics in both Java and C# over the past decade. The talk was given mainly through the lens of this evolution, which appears to approach an interesting limit if projected far enough into the future.

Type classes came on the scene towards the end of the 1980’s, and immediately became a fertile seed for research and exploration in the relationship between subclass and parametric polymorphism. Type classes are much closer to subclass polymorphism than Haskell’s borrowed ML-style, which is to say parametric polymorphism. This is most intriguing because Haskell does not rely on subclassing, and so the mixture of two breeds new patterns.

I thought that it might be interesting to compare the mixture of subclass and parametric polymorphism in Haskell vis-à-vis type classes with the same in C# vis-à-vis a mixture of interfaces, generics, and generic constraints. Hence this post. We shall proceed by examining some basic type classes in Haskell with their equals in C#. Though similar, the dissimilarities are as stark as the similarities. And the lack of higher kinds -- particularly when combined with type classes -- means that some Haskell patterns simply are not expressible in C#.

The Simple Case: Equality (or Lack Thereof)

The most basic type class of all is Eq, which allows the comparison of two like-typed pieces of data. This may seem like a commodity if you ordinarily write code in languages like Java and C# which have a strong notion of object identity. In Haskell, however, equality is value equality over algebraic data types rather than objects, so polymorphism over equality operators is quite a bit more important. Indeed, as we shall see, Haskell’s approach is more powerful than == in Java-like languages. (Witness the neverending dichotomy between reference and value equality vis-à-vis Object.Equals in C#.) But alas, let us proceed by crawling in a series of logical steps, rather than leaping to the conclusions.

Haskell’s Eq type class is defined as such:


    class Eq a where 


    (==), (/=) :: a -> a -> Bool 


   
   
      
   
   


    x /= y = not (x == y) 


    x == y = not (x /= y)

As you see, Eq provides two operators: == and /=. Default implementations of each define == as the inverse of /= and /= as the inverse of ==. Not only is this a convenience, but it also specifies the desired contract implementations ought to abide by. Other types may become members of the Eq class by mapping the one or both operators to type-specific functionality. You will immediately recognize the similarity to virtual methods in OOP languages, where the operators can be overridden by subclasses.

Of course all of the primitive data types already implement Eq, so you get value equality over numbers, strings, etc. Imagine we declared a new Coords type – comprised of two integers – and want to make it a member of Eq also – wherein equality is determined by a pairwise comparison of each’s members:


    data Coords = Coords { fst
   :: Integer, snd :: Integer }

We make Coords a member of the Eq type class, and thereby define equality over instance, through the ‘instance Eq Coords where’ construct. This maps type class functions to real implementation functions. The example here defines them inline, though you may of course refer to existing functions instead:


    instance Eq Coords where 


    (Coords fst1 snd1) == (Coords
   fst2 snd2) = (fst1 == fs2) && (snd1 == snd2)

Now we can take a ‘[Coords]’ and ask whether a particular ‘Coords’ exists within it.

A function may constrain a type variable to a certain class, and thereby access members of that class. For example, the following ‘isin’ function tests whether an instance of some type ‘a’ is contained within a list of type ‘[a]’. To do this, it demands that ‘a’ is a member of Eq using the syntax “Eq a =>”:


    isin :: Eq a => a -> [a]
   -> Bool 


    x `isin` [] = False 


    x `isin` (y:ys) = x == y
   || (x `isin` ys)

The moral equivalent to the Eq type class in C# is not so easy to decide. The most obvious first guess is
the built-in == and != operators. However, we will quickly find that this is not quite right, because these operators are not polymorphic in C#. To illustrate this point, let’s try to write the ‘isin’ method in C#, using generics and the == operator, for example:


    bool
   IsIn<T>(T x, T[] ys) 


    { 


    foreach
   (T y in ys) { 


    if
   (x == y) 


    return
   true; 


    } 


    }

This function will not compile. The reason is that == and != in C# are not defined over all types (specifically not for value types). You can get IsIn to compile by restricting the T to a reference type:


    bool
   IsIn<T>(T x, T[] ys) where T : class 


    { 


    …
   same as above … 


    }

Although this code is deceptively similar to the Haskell example, it is actually quite different. The == used to compare two instances compiles into the MSIL CEQ operator, effectively hard-coding an object identity comparison. Even if an overloaded == operator for a particular instantiated T is available, the compiler will not bind to it. Why? Because it is overloading and specifically *not* overriding. For example, say we had a MyData type and an overloaded == operator for comparing two instances:


    class
   MyClass 


    { 


    public
   static bool operator ==(MyClass a, MyClass b) { return true; } 


    public
   static bool operator !=(MyClass a, MyClass b) { return false; } 


    }

According to this, all MyClass objects are equal. However, the following call yields the answer ‘false’:


    IsIn<MyClass>(new
   MyClass(), new MyClass[] { new MyClass() });

The same problem arises should instances of MyClass get referred to by Object references. == and != do not perform any kind of virtual dispatch; the selection of implementation is chosen statically.

Perhaps it is the Equals method inherited from System.Object, then? This, at least, is virtual. And indeed, this gets much closer to Eq. Any type may override Equals, and a generic definition defined in terms of it dispatches virtually and allows subclasses to change behavior on a type-by-type basis:


    bool
   IsIn<T>(T x, T[] ys) 


    { 


    foreach
   (T y in ys) { 


    if
   (x == y || (x != null && x.Equals(y))) 


    return
   true; 


    }
   


    return
   false; 


    }

(Even this is slightly different, because it assumes a certain type-agnostic behavior about nulls.)

This is cheating, however. We’ve taken advantage of the fact that someone thought to put an Equals method on System.Object, thereby giving all Ts such a method. There are clearly limits to how many crosscutting things can be added to System.Object before it becomes overwhelmed with concepts, not to mention the size (e.g. v-tables). Moreover, Equals on Object is weakly typed; a better solution is to use interfaces, like the IEquatable<T> interface that introduces a strongly typed Equals method:


    public
   interface IEquatable<T> 


    { 


    bool
   Equals(T other);

    }

And to use a generic type constraint on IsIn’s T, much more akin to what ‘isin’ in Haskell above did:


    bool
   IsIn<T>(T x, T[] ys) 


    where
   T : IEquatable<T> 


    { 


    foreach
   (T y in ys) { 


    if
   (x == y || (x != null && x.Equals(y))) 


    return
   true; 


    }


    


    return false; 


    }

This is cheating a little less, because we can implement an interface after-the-fact without impacting a class’s type hierarchy. This, in fact, looks remarkably similar to the Haskell ‘isin’ shown earlier, using type classes and parametric polymorphism, where here we have used interfaces in place of type classes.

We might be tempted to define a default NotEquals method over all IEquatable<T> instances, just like Haskell does by implementing the defaults for == and /= as the inverse of each other:


    public
   static class Equatable 


    { 


    public
   static bool NotEquals<T>(this IEquatable<T> @this, IEquatable<T>
   other) 


     { 


    return
   !this.Equals(other); 


    } 


    }

This is not perfect. It is not polymorphic; see my previous post for an extensive discussion of this and related points. And what about nulls? If '@this' is null, the default implementation is going to AV. We’d need to bake in type-agnostic knowledge of null again. Sigh!

Sadly, it turns out this whole approach in general isn’t quite right anyway. For two reasons:

First, we still infect the type in question with the interface being implemented; it cannot be done completely outside of the type’s definition, as with type classes.
Second, type classes in Haskell do not actually require a value of the type in question to dispatch against the class’s functions, whereas we clearly do in the above example: we need to virtually dispatch against the object, and rely on this virtual dispatch to execute different code for each type. This will come up as we look at the numeric classes, but it is a critical difference.

A closer analogy is to use IEqualityComparer<T>:


    public interface IEqualityComparer<T> 


    { 


    bool Equals(T x, T y); 


    }

(IEqualityComparer<T> in .NET also has a GetHashCode method on it. Let’s ignore that for now.)

Unfortunately, if our IsIn method were to use IEqualityComparer<T> to do its job, callers would be required to pass an instance explicitly; we cannot infer a “default” comparer based solely on the T:


    bool
   IsIn<T>(T x, T[] ys, IEqualityComparer<T> eq) 


    { 


    foreach
   (T y in ys) { 


    if
   (eq.Equals(x, y)) 


    return
   true; 


    }
   


    return
   false; 


    }

Type classes actually function rather similarly, with two major differences:

The interface object – called a dictionary – is passed and used implicitly.
The mapping from types to dictionaries is done implicitly, whereas in .NET you’ll need to find an instance of the interface in question through other means.

This second difference is solved by a little hack in .NET. If you take a look at the EqualityComparer<T>.Default property, you shall see a lot of slightly gross reflection code to return an instance of IEqualityComparer<T> for any arbitrary T. The code checks some well-known types and conditions, and ultimately falls back to the aforementioned interfaces and default Equals method for the most general case. It’s not pretty, but it’s a beautiful hack given the tools at our disposal in C#.

A Harder Case: Polymorphic Numbers, on Output Parameters

The Eq type class is easy. The functions it defines are polymorphic on their inputs, but not on their outputs; both == and /= return Bool values. Once we transition to polymorphic output parameters or return values, we encounter a pattern quite different from that which is found in most .NET interfaces.

Let’s illustrate these differences by looking at Haskell’s Num type class:


    class (Eq
   a, Show a) => Num a where 


      (+),
   (-), (*) :: a -> a -> a 


      negate ::
   a -> a 


      abs,
   signum :: a -> a 


     fromInteger ::
   Integer -> a

Here we see another feature of Haskell type classes: inheritance. Num derives from both Eq and Show – indicated by “(Eq a, Show a) => Num a” – the latter class of which we have not yet shown but is the moral equivalent to .NET’s Object.ToString method. It enables pretty printing of values, clearly something that would be expected to be common among all numeric data types. Haskell’s numeric class hierarchy is quite elegant, enabling highly polymorphic computations. A nice little tutorial of can be found here: http://www.haskell.org/tutorial/numbers.html.

But the question at hand is what the C# equivalent would be.

Our first approach would be to mimic the IEquatable<T> solution above:


    interface
   INumeric<T> 


    { 


      T
   Add(T d); 


      T
   Subtract(T d); 


      T
   Multiply(T d); 


      T
   Absolute(); 


      T
   FromInteger(int x); 


    }

This works fine, and primitive types in .NET could presumably implement it:


    struct
   int : INumeric
      
         { .. }
    


    struct
   float : INumeric
      
         { .. }
    


    struct
   double : INumeric
      
         { .. }
    


    …

This enables polymorphic code, like a Sum method, through the use of generic type constraints:


    public
   static T Sum<T>(params T[] values) 


    where
   T : INumeric<T> 


    { 


    T
   accum = default(T); 


    foreach
   (T v in values) 


    accum
   = v.Add(accum); 


    return
   accum; 


    }

This example works great. Why then, you might wonder, doesn’t LINQ use this instead of providing special-case overloads of Average, Min, Max, Sum, etc. for all well-known primitive data types?

The primary reason is the performance hit taken to perform addition through O(N) interface calls versus O(N) MSIL ADD instructions. It is just a basic fact of life that today’s leading edge separate compilation techniques will not achieve parity with the hand specialized variants. While it is true that the JIT compiler *could* specialize the code for specific Ts and specific interfaces to emit more efficient instructions, like int, float, etc. over INumeric<T> calls, it will not do so today. This reduces the ability to share code – which admittedly is what we want here – and is tangled up in a judgment call based on heuristics. But I digress.

There is a larger problem that arises with other examples, at least from a language expressiveness point-of-view: the need to have an instance in hand to invoke interface methods. FromInteger, for example, is rather awkward to write. In fact, we cannot write a method with INumeric<T> like we could in Haskell:


    public
   static T MakeT<T>(int value) 


    where
   T : INumeric<T> 


    { 


    …
   ? … 


    }

How do we invoke FromInteger, given that no T is available at the time of MakeT’s invocation? You can’t; you need to arrange for an instance to be available. There are ways out of this corner. One solution is to mandate that T has a default constructor:


    public
   static T MakeT<T>(int value) 


    where
   T : INumeric<T>, new() 


    { 


    return
   new T(value).FromInteger(value); 


    }

That is always acceptable for structs, since they always have such a constructor; but this practice requires that classes be designed to possibly not hold invariants at all times, and so is not always acceptable or at the very least requires design accommodation.

The alternative is probably obvious. Use a similar approach to IEqualityComparer<T>:


    interface
   INumericProvider<T> 


    { 


      T
   Add(T x, T y); 


      T
   Subtract(T x, T y); 


      T
   Multiply(T x, T y); 


      T
   Absolute(T x); 


      T
   FromInteger(int x); 


    }

And now, of course, each method that does polymorphic number crunching must accept an instance of INumericProvider<T>. That’s particularly cumbersome, so it’s more likely that .NET developers would prefer the aforementioned approach, where the type must provide a default constructor.

Admittedly, I seldom run into this particular problem in practice; but when I do, I really wish I had something like Haskell type classes to help me out.

Before moving on, it is worth pointing out one Haskell type class problem that explicit interface object passing in .NET helps to avoid. Should you need multiple implementations of a given class for the same type, as is relatively common with equality comparisons, you must disambiguate in Haskell by separation by module and being careful about what you import. This is similar to C#’s extension methods. With explicitly passed interface objects, however, it is trivial to manage and pass separate objects if you’d like.

Close, but No Cigar: Higher Kinds

There is one last feature that Haskell provides – a pretty big one, I might add – that C# simply cannot do: higher kinded types, or polymorphism over constructed types. This feature is orthogonal to type classes, but gets used pervasively in conjunction with them. An example will make this stunningly clear:


    class Monad
   m where 


    (>>=) ::
   m a -> (a -> m b) -> m b 


    (>>) ::
   m a -> m b -> m b 


    return ::
   a -> m a 


    fail ::
   String -> m a 


   
   
      
   
   


    m
   >> k = m
   >>= \_ -> k 


    fail
   s = error s

Let’s try to transcribe the core of this class in C#, renaming >>= to Bind, and omitting the >> and ‘fail s’ operators because they have default implementations:


    public
   interface IMonad<M, A> 


    { 


    M<B>
   Bind<B>(M<A> m, Func k); 


     M<A>
   Return(A a); 


    M<A>
   Fail(string s); 


    }

This approach is tantalizingly close. It suffers from the already-admitted problem that, for any M<A> instance, you will need to pass the appropriate IMonad<A> provider object – just as with the IEqualityComparer<T> and INumericProvider<T> examples above.

But the code of course won’t *actually* compile, because the type variable M cannot be constructed as shown here. We find references to M<A> and M, which are complete nonsense to C#. M is just a plain type variable. M is required to be what Haskell calls a type constructor (* -> *), which is a generic type that must be instantiated before it is a terminal type. I’ve written about this before. Although it seems like a trivial omission in C#’s language definition, it strikes at the heart of the type system.

A fictitious syntax for expressing this in C# might be:


    public
   interface IMonad 


    <>where
   M : 
    


    {


    ... 


    }

And if, say, M were expected to be a two- or three-parametered type, we would find, respectively:


    ... <,>where
   M :  


    ... <,,>where
   M :

And so on.

This could in theory work. But C# -- and more worrisome .NET and the CLR – do not support this presently, and, to be quite honest, likely never will. It is immensely powerful, however. Life without monads is a life destined to continuous repetition. The “LINQ Pattern”, for example, is one example case in .NET where, for each ‘source’ type, we must create a “copy” of the original System.Linq.Enumerable variant. And shame on those who wish to write polymorphic code that will work for any LINQ provider.

Winding Down

Let’s wind down. I need to go grab dinner at Mama's Fish House on Maui right now.

I hope to have shown some of the similarities and dissimilarities between type classes and interfaces, and some patterns that arise when these things are mixed with parametric polymorphism. The mix of inheritance for type classes, but not for implementation types, in Haskell is unique. C#, of course, allows inheritance both amongst interfaces and implementations which is both a blessing and a curse.

I do think both camps have something to teach one another. For example, having a default interface lookup mechanism for arbitrary types in C# would be wonderful, and indeed might provide a replacement for extension methods that has more longevity. I’m sure much of this will happen with time; either “in place” as the respective languages evolve, or as new languages are created with time.

But most importantly, I hope that the blog post was educational and fun. Enjoy.

Extension methods as default interface method implementations

Wed, 10 Feb 2010 02:21:49 GMT

One of my comments in the 2nd edition of the .NET Framework design guidelines (on page 164) was that you can use extension methods as a way of getting default implementations for interface methods. We've actually begun using these techniques here on my team. To illustrate this trick, let's rewind the clock and imagine we were designing new collections APIs from day one.

Let's say we gave the core interfaces the most general methods possible. These may neither be the most user friendly overloads nor the ones that most people use all the time. They would, however, be those from which all the other convenience methods could be implemented. An INewList<T> interface that was designed with these principles in mind may look like this:

public interface INewList<T> : IEnumerable<T>

{

 int Count { get; }

 T this[int index] { get; set; }

 void InsertAt(int index, T item);

 void RemoveAt(int index);

}

This interface is missing all the nice convenience methods you will find on .NET's IList<T>, like Add, Clear, Contains, CopyTo, IndexOf, and Remove. So it's not really as nice to use. You can't write an API that takes in an INewList<T> and performs an Add against it, for example, like you can with IList<T>.

One approach to solving this might be to write a concrete class -- much like .NET's System.Collections.ObjectModel.Collection<T> -- that provides concrete implementations of all of these methods, and then other lists can simply subclass that. But we can do better.

Instead, let's give INewList<T> default implementations of all of these methods. How do we do this? That's right: with extension methods. Voila!

public static class NewListExtensions

{

 public static void Add<T>(this INewList<T> lst, T item)

 {

 lst.InsertAt(lst.Count, item);

 }

 public static void Clear<T>(this INewList<T> lst)

 {

 int count;

 while ((count = lst.Count) > 0) {

 lst.RemoveAt(count - 1);

 }

 }

 public static bool Contains<T>(this INewList<T> lst, T item)

 {

 return lst.IndexOf(item) != -1;

 }

 public static void CopyTo<T>(this INewList<T> lst, T[] array, int arrayIndex)

 {

 for (int i = 0; i < lst.Count; i++) {

 array[arrayIndex + i] = lst[i];

 }

 }

 public static int IndexOf<T>(this INewList<T> lst, T item)

 {

 var eq = EqualityComparer<T>.Default;

 for (int i = 0; i < lst.Count; i++) {

 if (eq.Equals(item, lst[i])) {

 return i;

 }

 }

 return -1;

 }

 public static bool Remove<T>(this INewList<T> lst, T item)

 {

 int index = lst.IndexOf(item);

 if (index == -1) {

 return false;

 }

 lst.RemoveAt(index);

 return true;

 }

}

Well isn't that neat. We've now given any INewList<T> implementations all these common methods without dirtying their class hierarchies, built atop a tiny core of extensibility. This is much like .NET's Collection<T> which exposes the core as abstract methods. Indeed, we can go even further. Any convenience overloads, like the multitude of CopyTos on List<T> in .NET, can be given to all INewList<T>'s also. And yet implementing INewList<T> remains as braindead simple as it was before: two properties and two methods. In fact, it's simpler than doing a more feature-rich IList<T>, because the convenience methods come for free.

It would be even niftier if you could add these methods straight onto INewList<T>, and have the C# compiler emit the extension methods silently for you. In other words:

public interface INewList<T> : IEnumerable<T>

{

 ... interface methods (as above) ...

 void Add(T item)

 {

 InsertAt(Count, item);

 }

 void Clear()

 {

 int count;

 while ((count = Count) > 0) {

 RemoveAt(count - 1);

 }

 }

 ... and so on ...
}

Although this would just be sugar for the NewListExtensions class shown earlier, it sure saves some typing and makes it the pattern more apparent and first class.

Though cool, this whole idea is certainly not perfect.

For one, there are no extension properties. So you can't use this trick for properties.

But the more obvious and severe downside to this approach that these methods are not specialized for the given concrete type. For example, the Clear method is potentially far less efficient than a hand-rolled List<T>, because it does O(N) RemoveAts rather than a single O(1) fixup of the count.

Recall now that the compiler binds more tightly to instance methods than extension methods. So we could implement our own little list class with a faster Clear method if we'd like:

class MyList : INewList<T>

{

 ... the two properties and two methods from INewList<T> ...

 public void Clear()

 {

 .. efficient! ...
 }

}

Now when someone calls Clear on a MyList<T> directly, the compiler will bind to the efficient Clear.

This is still not perfect. If you pass the MyList<T> to an API that takes in an INewList<T>, any calls to Clear will fall back to the extension method. Extension methods are not virtual in any way. You can try to simulate virtual dispatch, but it gets messy quick. For example, say we defined an IFasterList<T> that includes all those convenience methods that lists frequently want to make faster; we can then do a typecheck plus virtual dispatch in the extension method.

For now, let's pretend that's just the Clear method:

public interface IFasterList<T> : INewList<T>

{

void Clear();

}

Of course, MyList<T> above would now implement IFasterList<T>. Invocations through IFasterList<T> will automatically bind to the faster variant; but if objects that implement IFasterList<T> get passed around as IList<T>s, you lose this ability. So the Clear extension method can now do a typecheck:

public static void Clear<T>(this INewList<T> lst)

 {

 IFasterList<T> fstLst = lst as IFasterList<T>;

 if (fstLst != null) {

 fstLst.Clear();

 return;

 }

 int count;

 while ((count = lst.Count) > 0) {

 lst.RemoveAt(count - 1);

 }

 }

This works but is obviously a tedious and hard-to-maintain solution. It would be neat if someday C# figured out a way to "magically" reconcile virtual dispatch and extension methods. I don't know if there is a clever solution out there. I am skeptical. Nevertheless, despite this flaw, the above techniques are certainly thought provoking and interesting enough to play around with and consider for your own projects. And at the very least, it's fun. Enjoy.

Musing on messages and blocking

Fri, 08 Jan 2010 08:11:39 GMT

Sometimes you need to wait for something before proceeding with a computation.

Perhaps you need to know the value of some integer that is being computed concurrently. Maybe you need to wait for the bytes to flush to disk before telling another process the file is consistent and ready to read. Or you need to get that next row back from the database before painting it on the UI. It could be that you need to wait for the missile to leave the bay before closing the bay door. And so on.

And sometimes there’s simply nothing better to do while waiting for these things to happen other than to let the CPU halt (or let other processes on the machine run). You need to twiddle your thumbs a bit, and exhibit a little patience. Or at least your program does. This is simply an unfortunate fact of life.

This manifests numerous ways in our programming models:

        1) Waiting on an event.
        2) Waiting to acquire an already-held lock.
        3) Finding that the GUI message queue is empty and doing a MsgWaitForMultipleObjectsEx.
        4) Finding that the COM RPC queue is empty and doing a CoWaitForMultipleHandles.
        5) Issuing an Ada rendezvous ‘accept’ and finding that no messages await you, thus blocking.
        6) Issuing an Erlang ‘receive’ and finding that no messages await you, thus blocking.
        7) Waiting on a .NET 4.0 task.
        8) Issuing a ContinueWith on a .NET 4.0 task.
        9) And so on.

There are three big distinctions to make about the characteristic nature of this waiting: namely, (1) what condition's establishment is being sought -- i.e. the reason for the wait, (2) whether multiple such conditions of interest may be waited on simultaneously, and, related, (3) whether waiting for said condition(s) necessarily means that the processing of some other conditions that may arise elsewhere, but require the blocked context to run, cannot occur.

I will be the first to admit that this statement is rather abstract. But it really does matter.

For example, MsgWaitForMultipleObjectsEx is a pumping wait. Not only do you wait for the occurrence of one of several events to get set, but the arrival of a new top-level message at the message queue (either GUI or COM RPC-related) causes immediate processing of that message, presuming the thread is blocked at that call at the time. Although you can be deeply nested in some complicated code, you “jump” to the event loop to run the message handling code. Vanilla WaitForMultipleObjectsEx works in a similar way vis-à-vis APCs, provided the wait is alertable. This is quite different from a fully blocking non-pumping wait, which only waits for one or more very specific events, but does not dispatch messages simultaneously.

Win32 esoterica aside, the concepts appear elsewhere. The moral equivalent in Ada or Erlang is to do a selective-accept or -receive, intentionally not dispatching certain messages that might arrive in the meantime. (To be fair, you can also do this in COM with message filters.) This often happens when you nest accepts and receives. You may be capable of processing messages A-Z at the top-level tail recursive loop; but if that nested accept only knows about message kinds M and N, then there are 24 other kinds that will not be picked up in the meantime.

Not pumping for messages is dangerous. And it can lead to deadlock if you pump for the wrong ones. Like if you’re accepting M or N, yet the triggering of M or N depends on first processing some message K waiting in the queue. COM RPCs with cycles run face first into this. And/or not pumping can lead to responsiveness and scalability problems. Perhaps M or N eventually does arrive, yet little old K needs to wait an indeterminate amount of time before it is seen. Whereas we could have overlapped its processing. This is why most STAs pump while waiting, and, similarly, why many Erlang processes consist of a main loop that is prepared to handle any message the process accepts at that top level loop. They may seem very different but they are strikingly not.

Yet paradoxically pumping for messages is also dangerous. You must predict all the kinds of messages that may reentrantly get executed, and your state at the point of the blocking call must be consistent enough to tolerate them. (At least those that will actually happen.) In COM STAs, this can be wholly unpredictable and indeed because the CLR auto-pumps on STAs the blocking points can be hidden. Overly aggressively admitting messages may seem like the right thing to do, until you’ve wedged yourself into some unforeseen inconsistent state. You can avoid this by making each message handler atomic; see Argus. But if you can't or don't have the discipline to do that, or aren't quite sure, you must not pump. You either avoid pumping altogether or you selectively pump messages that do not touch the state encapsulated by the pump. Or you lock access to state with a non-recursive lock and run the risk of deadlock.

I have found it clarifying to think about blocking in event loop concurrency and state machine terms, advancing from one state to the next in between waits. It’s a slippery model, but particularly when working in message passing systems that employ event loops, it can help to identify all the familiar problems with shared memory, blocking, and consistency.

Indeed it is interesting how blocking and non-blocking systems can rapidly approach each other. Starting from either extreme tempts you to tiptoe closer and closer to the middle. The familiarity of the other extreme tempts you. Until, alas, you just might meet in the middle.

A (brief) retrospective on transactional memory

Sun, 03 Jan 2010 19:05:12 GMT

Rewind the clock to mid-2004. Around this time awareness about the looming “concurrency sea change” was rapidly growing industry-wide, and indeed within Microsoft an increasing number of people – myself included – began to focus on how the .NET Framework, CLR, and Visual C++ environments could better accommodate first class concurrent programming. Of course, our colleagues in MSR and researchers in the industry more generally had many years’ head start on us, in some cases dating back 3+decades. It is safe to say that there was no shortage of prior art to understand and learn from.

One piece of prior art was particularly influential on our thoughts: software transactional memory. (STM, or, in short just TM.) In fact, right around that time, Tim Harris’s TM work grew in notoriety (my first exposure arriving by way of OOPSLA’03’s proceedings, which contained the “Language Support for Lightweight Transactions” paper). TM was immediately fascinating, and simultaneously promising. For a number of reasons:

TM hid sophisticated synchronization mechanisms under a simple veil.
It could be implemented using sophisticated (and scalable) techniques, again under a simple veil.
It built on decades of experience in building scalable and parallel transactional databases.
Among others. But most of all, it was a bright shiny light in a sea of complexity.
And how fortunate: Tim was a colleague in our neighboring MSR Cambridge offices (and still is).

In a nutshell, TM offered declarative concurrency-safety. You declare what you’d like in as few simple words as possible, and you get what you want. In this case, those simple words are ‘atomic { S; }’.

Many people latched onto TM rapidly and simultaneously, both inside and outside of Microsoft. I hacked together a little prototype built atop SSCLI (“Rotor”), and another architect on our team built an even more feature-rich prototype using MSIL rewriting. We compared notes, began jointly exploring the design space, and talking more regularly with other colleagues like Tim in MSR. Soon thereafter we kicked off a small working group with about a dozen architects and researchers from around the company, aiming to articulate what a real productized TM might look like. Fun times.

We were eventually given the OK for an official “incubation” project, and multiple years’ of exploration and hard work ensued. In fact, the fruits of a team of many’s labor recently got released in the form of a Community Technology Preview -- a good conduit for experimentation, but with no commitment to add it to any of Microsoft’s products. To be clear, I had only a small part to play in this ambitious project, and mostly towards the start. Partway through, I stepped away to do PLINQ and Parallel Extensions to .NET, both of which are now part of the .NET Framework 4.0. Dozens of amazing people played a significant role in the project over the years. But I am getting way ahead of myself…

I’ve been away from the nitty-gritty day-to-day details of TM for about 3 years now, which feels sufficiently long to develop a healthy perspective on the project. So here it is. What follows is of course in no way Microsoft’s “official position” on the technology, but rather my own personal one. I’ve interspersed generalizations with specific details because that’s just how my brain thinks about TM.

Towards the North Star

A wondrous property of concurrent programming is the sheer number and diversity of programming models developed over the years. Actors, message-passing, data parallel, auto-vectorization, ...; the titles roll off the tongue, and yet none dominates and pervades. In fact, concurrent programming is a multi-dimensional space with a vast number of worthy points along its many axes.

This rich history is simultaneously a blessing and a daunting curse. But in any case can make for some very interesting multi-year-long immersion. My UW talk from 1 1/2 years ago just barely touches on the sheer breadth.

TM’s greatest virtue is the first word in its name: transactional. It turns out that, no matter your concurrent programming model du jour, three fundamental concepts crop up again and again: isolation (of state), atomicity (of state transitions), and consistency (of those atomic transitions). We use locks in shared-memory programming, coarse grained messages in message-passing, and functional programming to achieve all of these things in different ways. Transactions are another such mechanism, sure, but more than that, transactions are an all-encompassing way of thinking about how programs behave at their most fundamental core. Transaction is a religion.

Not everybody believes this, and of course why would they: it is an immensely subjective and qualitative statement. Some will claim that models like message passing entirely avoid the likes of “race conditions,” and such, but this is clearly false: state transitions are made, complicated state invariants are erected amongst a sea of smaller isolated states, and care must be taken, just as in shared memory. Even Argus, a beautiful early incarnation of message-passing (via promises) demands that messages are atomic in nature. This property is not checked and, if done improperly, leads to “races in the large.” Even Argus introduced the notion of transactions and persistence in the form of guardians.

Of course, message passing helps push you in the right direction. It is not, however, a panacea.

I was reading my ICFP proceedings recently and was reminded of research done in the context of Erlang that supports this assertion. In it, they apply CHESS-like techniques (with clever search space culling) to find race conditions. Indeed we use similar techniques very successfully for our message-passing programming models on my team here at Microsoft.

Transactions are terrific because they are “automatic”. You declare the boundaries, and the transactional machinery takes care of the rest. This is true of databases and also TM. Countless developers in the wild write massively concurrent programs by issuing operations against databases: they can do this so easily because they grok the simple façade that transactions provide. Numerous server-side state-based applications use transactions to shield programmers from the pitfalls of concurrency. Behold MSDTC. The bet we were making is that similar models would scale down just as well “in the small”.

The canonical syntactic footprint of TM is also beautiful and simple. You say:

atomic {

… concurrency-safe code goes here …

}

And everything in that block is magically concurrency-safe. (Well, you still need to ensure the consistency part, but isolation and atomicity are built-in. Mix this with Eiffel- or Spec#-style contracts and assertions like those in .NET 4.0, run at the end of each transaction, and you’re well on your way to verified consistency also. The ‘check E’ work in Haskell was right along these lines.) You can read and write memory locations, call other methods, all without worrying about whether concurrency-safety will be at risk.

For example, consider three transactions running concurrently:

int x = 0, y = 0, z = 0;

atomic { atomic { atomic {

x++; y++; z++;

} x++; y++;

} x++;

}

No matter the order in which these run, the end result will be x == 3, y == 2, z == 1.

Contrast this elegant simplicity with the many pitfalls of locks:

Data races. Like forgetting to hold a lock when accessing a certain piece of data. And other flavors of data races, such as holding the wrong lock when accessing a certain piece of data. Not only do these issues not exist, but the solution is not to add countless annotations associating locks with the data they protect; instead, you declare the scope of atomicity, and the rest is automatic.
Reentrancy. Locks don’t compose. Reentrancy and true recursive acquires are blurred together. If a locked region expects reentrancy, usually due to planned recursion, life is good; if it doesn’t, life is bad. This often manifests as virtual calls that reenter the calling subsystem while invariants remain broken due to a partial state transition. At that point, you’re hosed.
Performance. The tension between fine-grained locking (better scalability) versus coarse-grained locking (simplicity and superior performance due to fewer lock acquire/release calls) is ever-present. This tension tugs on the cords of correctness, because if a lock is not held for long enough, other threads may be able to access data while invariants are still broken. Scalability pulls you to engage in a delicate tip-toe right up to the edge of the cliff.
Deadlocks. This one needs no explanation.

In a nutshell, locks are not declarative. Not even close. They are not associated with the data protected by those locks, but rather the code that accesses said data. (For example: in the above code snippet, do we need three locks? Or one? Or …? Imagine we choose three: one for each variable, x, y, and z. What if we increment z, release its associated lock, and some other thread can now see the newly incremented z before the y and x get incremented. Whether this is acceptable depends on the program.) Sure, you can achieve atomicity and isolation, but only by intimately reasoning about your code by understanding the way they are implemented. And if you care about performance, you are also going to need to think about hardware esoterica such as CMPXCHG, spin waiting, cache contention, optimistic techniques with version counters and memory models, ABA, and so on.

The contrast is stark. Atomic-block-style transactions provide automatic serializability of whole regions of code, no matter what that code does, and the TM infrastructure does the rest, choosing between: optimistic, pessimistic, coarse, fine, etc. The linearization point of a transaction is clear: the end of the atomic block. TM can even adjust strategies based on the surrounding environment: hardware, dynamic program behavior, etc. (“Policy”.) In comparison to locks, TM is an order of magnitude simpler. There have even been studies whose conclusions support this assertion.

(Transactions unfortunately do not address one other issue, which turns out to be the most fundamental of all: sharing. Indeed, TM is insufficient – indeed, even dangerous – on its own is because it makes it very easy to share data and access it from multiple threads; first class isolation is far more important to achieve than faux isolation. This is perhaps one major difference between client and server transactions. Most server sessions are naturally isolated, and so transactions only interfere around the edges. I’m showing my cards too early, and will return to this point much, much later in this essay.)

TM also has the attractive quality of automatic rollback of partial state updates. (How did I get this far without discussing rollback?) Concurrency aside, this avoids needing to write backout code to run in the face unhandled exceptions. In retrospect this capability alone is almost enough to justify TM in limited quantities. Reams of code “out there” contain brittle, untested, and, therefore, incorrect error handling code. We have seen such code lead to problems ranging in severity: reliability issues leading to data loss, security exploits, etc. Were we to replace all those try/catch/rethrow blocks of code with transactions, we could do away with this error prone spaghetti. We’d also eliminate try/filter exploits thanks to Windows/Win32 2-pass SEH. Sometimes I wish we focused on this simple step forward, forgot about concurrency-safety, and baby stepped our way forward. Likely it wouldn’t have been enough, but I still wonder to this day.

We also toyed with the ability to replace reliability-oriented CER blocks with transactions. As you go through a transaction, there is a log of forward progress and how to undo it. So no matter the kind of failure, including OOM, you can rollback the partial state updates with zero allocation required.

At some point we began describing an ‘atomic’ block as though the program used a single global lock for all its concurrency operations. This would be grossly inefficient, of course, and fails to capture the precise isolation and rollback properties, but nevertheless conveys the basic idea. It also, as an aside, foreshadows a few of the difficult problems that lie ahead, namely strong vs. weak atomicity. Even though there is only one, if you forget to hold this one global lock while accessing shared data, you’ve still got a data race on your hands. This model won’t save you. We will return to this later on.

Tough Decisions: Life as a Starving Artist

We faced some programming model decisions requiring artistic license early on.

One that we quickly decided was whether to automatically roll back a transaction in response to an unhandled exception thrown from within. Such as with this code:

atomic {

x++;

if (p)

throw new Exception(“Whoops”);

}

If p evaluates to true, and hence an unhandled exception thrown, should that x++ be rolled back?

Most on the team said “Yes” as a gut reaction, whereas some argued we should require the programmer to catch-and-rollback by hand. We settled on the automatic approach because it seemed to do what you would expect in all the cases we looked at. Your transaction failed to complete normally and consistently. We also debated whether to support a unilateral “Transaction.Abort()” capability; while we agreed a “Transaction.Commit()” would be silly – the only way to commit a transaction being to reach its end non-exceptionally – the jury remained split on unilateral abort. We eventually found that, particularly when nesting is involved, the ability to detect a dire problem with the universe and bail unilaterally can be useful.

And we also hit some tough snags early on. Some were trivial, like what happens when an exception is thrown out of an atomic block. Of course that exception was likely constructed within the atomic block (‘throw new SomeException()’ being the most common form of ‘throw’), so we decided we probably need to smuggle at least some of that exception’s state out of the transaction. Like its stack trace. And perhaps its message string. I wrote the initial incarnation of the CLR exception subsystem support, and stopped at shallow serialization across the boundary. But this was a slippery slope, and eventually the general problem was seen, leading to more generalized nesting models (which I shall describe briefly below). Another snag, which was quite non-trivial, was the impact to the debugging experience. Depending on various implementation choices – like in-place versus buffered writes – you may need to teach the debugger about TM intrinsically. And some of the challenges were fundamental to building a first class TM implementation. Clearly the GC needed to know about TM and its logging, because it needs to keep both the “before” and “after” state of the transaction alive, in case it needed to roll back. The JIT compiler was very quickly splayed open and on the surgery table. And so on.

Throughout, it became abundantly clear that TM, much like generics, was a systemic and platform-wide technology shift. It didn’t require type theory, but the road ahead sure wasn’t going to be easy.

So we knocked down many early snags, and kept plowing forward, eagerly and excitedly. None of these challenges were insurmountable. We remained hopeful and happy (perhaps even blissful) to continue exploring the space of possible solutions. More irksome snags lurked right around the corner, however. And little did we know that some decisions we were about to make would subject us to some of the biggest such snags. TM’s greatest feature – slap an atomic around a block of code and it just gets better – would turn out to be its greatest challenge… but alas, I am again jumping ahead; more on that later.

Turtles, but How Far Down? Or, Bounded vs. Unbounded Transactions

Not all transactions are equal. There is a broad spectrum of TMs, ranging from those that are bounded to updating, say, 4 memory locations or cache lines, to those that are entirely unbounded. Indeed TM blurs together with other hardware-accelerated synchronization techniques, like speculative lock elision (SLE). The more constrained TM models are often hardware-hybrids, and the limitations imposed are typically due to physical hardware constraints. Models can be pulled along other axes, however, such as whether memory locations must be tagged in order to be used in a transaction or not, etc. Haskell requires this tagging (via TVars) so that side-effects are evident in the type system as with any other kind of monad.

We quickly settled on unbounded transactions. Everything else looked like multi-word CAS and, although we knew multi-word CAS would be immensely useful for developing new lock-free algorithms, our aim was to build something radically new and with broader appeal. If we ended up with a hardware-hybrid, we would expect the software to pick up the slack; you’d get nice acceleration within the hardware constraints, and then “fall off the silent cliff” to software emulation thereafter. Thus the unbounded approach was chosen.

In hindsight, this was a critical decision that had far-reaching implications. And to be honest, I now frequently doubt that it was the right call. We had our hearts in the right places, and the entire industry was trekking down the same path at the same time (with the notable exception of Haskell). But with the wisdom of age and hindsight, I do believe limited forms of TM could be wildly successful at particular tasks and yet would have avoided many of the biggest challenges with unbounded TM.

And believe me: many such challenges arose in the ensuing months.

An example of one challenge that didn’t threaten the model of TM per se, but sure did make our lives more difficult, is the compilation strategy we were forced to adopt. Transactions cost something. To transact a read or write entails a non-trivial amount of extra work; we spent a lot of time optimizing away redundant work, and developing new optimizations that reduced the overhead of TM. But at the end of the day, the cost is not zero – and in fact, the common case is far from it. Imagine you have an unbounded transaction model and are faced with compiling a particular method from MSIL to native code. A simple separate-module -based compiler (i.e. not whole-program) will not necessarily know whether this method will get called from a transaction, or from non-transactional code, such that in the worst case the method must be prepared for transactional access. There are a variety of techniques to use to produce code that supports both: the two extremes are (1) cloning, or (2) sharing w/ conditional dynamic checking. Neither extreme is particularly attractive, and this choice represents a classic space-time tradeoff that entails finding a reasonable middle ground. A JIT compiler can dynamically produce the version that is needed at the moment, but offline compilers – like the CLR’s NGEN – do not have this luxury. And within Microsoft at least, and among shrink-wrap ISVs, offline compilation is of greater importance than JIT compilation. For better or for worse.

The model of unbounded transactions is the hard part. You surround any arbitrary code with ‘atomic { … }’ and everything just works. It sounds beautiful. But just think about what can happen within a transaction: memory operations, calls to other methods, P/Invokes to Win32, COM Interop, allocation of finalizable objects, I/O calls (disk, network, databases, console, …), GUI operations, lock acquisitions, CAS operations, …, the list goes on and on. Versus bounded transactions, where we could say something like: if you do more than N things, the transaction will fail to run – deterministically.

Unbounded really was the golden nugget. But we should not be shy about what this decision implies.

Implementing the Idea

This leads me to a brief tangent on implementation. Given that we didn’t implement TM with a single global lock, as the naïve mental model above suggests, you may wonder how we actually did do it. Three main approaches were seriously considered:

IL rewriting. Use a tool that passes over the IL post-compilation to inject transaction calls.
Hook the (JIT) compiler. The runtime itself would intrinsically know how to inject such calls.
Library-based. All transactional operations would be explicit calls into the TM infrastructure.

Approaches #1 and #2 would look similar, but the latter would be quite different. Instead of:

atomic {

x++;

}

Or:

Atomic.Run(() => {

x++;

});

You might say something like:

Atomic.Run(() => {

Atomic.Write(Atomic.Read(ref x) + 1);

});

With enough language work, we could have tried to desugar the latter into the former, but when you start crossing method boundaries, everything gets more complicated. (Do you create transactional clones of every method, and rewrite calls from ordinary methods to the transactional clone? This is easy to do with a rewriter or compiler, but quite difficult with a pure library approach.) We also knew we’d need to do some very sophisticated compiler optimizations to get TM’s performance to the point of acceptable. So we chose approach #2 for our “real” prototype, and never looked back.

After this architectural approach was decided, a vast array of interesting implementation choices remained.

We moved on to building the primitive library with all the TM APIs that the JIT would introduce calls into. We quickly settled an approach much like Harris’s (and, at the time, pretty much the industry/research standard): optimistic reads, in-place pessimistic writes, and automatic retry. That means reads do not acquire locks of any sort, and instead, once the end of the transaction has been reached, all reads are validated; if any locations read have been modified concurrently (or an uncommitted value was read), the whole transaction is thrown away and reattempted from the start. Writes work like locks. This approach makes reads cheap: a single read consists of reading the value, and a version number whose address is at a statically known offset. No interlockeds. This is great since reads typically far outnumber writes. Down the line, we explored adding more sophisticated policy than this, which I will detail in brief below.

So the compiler would inject hooks for the above code like so:

while (true) {

TX tx = new TX();

try {

// x++;

tx.OpenReadOptimistic(ref x);

int tmp = x;

tx.OpenWritePessimistic(ref x);

x = (tmp + 1);

if (!tx.Validate())

continue;

tx.Commit();

}

catch {

tx.Rollback();

throw;

}

Notice there are some obvious overheads in here:

The atomic block becomes a loop (to support automated retry).
A new transaction must be allocated and likely placed in TLS (if methods are called).
A try/catch block is used to initiate rollback on unhandled exceptions.
Each unique location read in a block requires at least one call to OpenReadOptimistic.
Each unique location written requires at least one call to OpenWritePessimistic.
Each location read must be validated (at Validate), and finally the transaction is committed (at Commit).

Much of the work in the compiler was meant to reduce these overheads. For example, if the same location is read multiple times, there’s no need to call OpenReadOptimistic more than once. If the compiler can statically detect this, it may elide some of the calls. If the same location is read and then written – as in the above example – only the write lock must be acquired. If no methods are called, the transaction object can be enregistered, and we needn’t add it to TLS so long as the exception trap code knows how to move it from register to TLS on demand. Et cetera.

There are other overheads that are not so obvious. Optimistic reads mandate that there is a version number for each location somewhere, and pessimistic writes mandate that there is a lock for each location somewhere.

A straightforward technique is to use a hashing scheme to associate locations with this auxiliary data: each address is hashed to index into a table of version numbers and locks. This leads to false sharing, of course, but reduces space overhead and makes lookup fast. Unfortunately, in a garbage collected environment, addresses are not stable and therefore hashing becomes complicated. You can use object hash codes for this purpose, but .NET hash codes are overridable; and generating them is not nearly as cheap as using the memory location’s address, which by definition is already in-hand. Other alternatives of course exist. You can associate version numbers and locks with the objects themselves, just like monitors and object headers/sync-blocks in the CLR: this provides object-granularity locking. Ahh, the age old tension of fine vs. coarse grained locking comes up again.

We eventually realized we’d want both optimistic and pessimistic reads, the latter of which worked a lot like reader/writer locks. We crammed all these into a clever little word-sized data structure which worked a lot like Vista’s SRWL data structure. Except that it also contained a version number.

It was always surprising to me what strange things in the runtime we bumped up against. We realized a nice GC optimization: instead of keeping strong references to all intermediary states in a transaction log, we could keep weak references to all but the “before” and “after” state. This is important when transacting synthetic situations like this:

static BigHonkinFoo s_f;

…

atomic {

for (int i = 0; i < 1000000; i++)

s_f = new BigHonkinFoo();

}

Of course you wouldn’t write that code exactly. But there’s no need to keep alive all but the s_f that existed prior to entering the atomic block and the current one at any given time. But this leads to particularly hairy finalization issues. If a finalizable object is allocated within a transaction (say BigHonkinFoo), and is then reclaimed, its Finalize() method will be scheduled to run on a separate thread. Yet the transaction log may contain references to it. Thus there is a race between the transaction’s final outcome and the invocation of the finalizer. We came up with a clever solution for this, but there were countless other clever solutions for various things not worth diving too deep into.

Hacking is fun. However, it was not going to be what made or broke TM as a model.

Disillusionment Part I: the I/O Problem

It wasn’t long before we realized another sizeable, and more fundamental, challenge with unbounded transactions. Finalizers touched on this. What do we do with atomic blocks that do not simply consist of pure memory reads and writes? (In other words, the majority of blocks of code written today.) This was not just a pesky question of how to compile a piece of code, but rather struck right at the heart of the TM model.

You already saw the OpenReadOptimistic, OpenWritePessimistic, Validate, Commit, and Rollback pseudo-TM infrastructure calls, each of which operated on memory locations. But what about a read or write from a single block or entire file on the filesystem? Or output to the console? Or an entry in the Event Log? What about a web service call across the LAN? Allocation of some native memory? And so on. Ordinarily these kinds of operations will be composed with other memory operations, with some interesting invariant relationship holding between the disparate states. A transaction comprised of a mixture still ought to remain atomic and isolated.

The answer seemed clear. At least in theory. The transaction literature, including Reuter and Gray’s classic, had a simple answer for such things: on-commit and on-rollback actions, to perform or compensate the logical action being performed, respectively. (Around this same time, the Haskell folks were creating just this capability in their STM, where ‘onCommit’ and ‘onRollback’ would take arbitrary lambdas to execute the said operation at the proper time.) Because we were working primarily in .NET – with a side project targeting C++ -- we decided to use the new System.Transactions technology in 2.0 to hook into inherently transactional resources, like transacted NTFS, registry, and, of course, databases.

(Digging through my blog, I found this article written back in June 2006 about building a volatile resource manager for memory allocation/free operations, just as an example.)

This worked, though we were quite obviously swimming upstream. Numerous challenges confronted us.

A significant problem was that not all operations are inherently transactional, so in many cases we were faced with the need to add faux transactions on top of existing non-transactional services. (Already-transactional services were easy, like databases. Except that mixing fine-grain TM transactions with distributed DTC transactions makes my skin crawl.) For example, how would you undo a write to the console? Well, you can’t, really. So we decided maybe the right default for Console.WriteLine was to use an on-commit action to perform the actual write only once the transaction had committed.

But in even thinking this thought, we realized we were standing on shaky ground. What if the WriteLine was followed by something like a ReadLine, for example, where the program was meant to wait for the user to enter something into the console (likely in response to the prompt output by WriteLine)? (This example is a toy, of course, but represents a more fundamental pattern common in networked programs.) The basic problem was immediately clear. Adding isolation to an existing non-isolated operation is not always behavior-preserving, particularly when I/O is involved. Sometimes it is necessary to step outside of the isolation that would otherwise get poured on top by a simple transactional model.

This particular problem isn’t specific to traditional I/O per se.

Foreign function interface calls through.NET’s P/Invoke suffer from like problems. A call to CreateEvent may be compensatable (via an on-rollback action) with a call to CloseHandle. But this is flawed. Once that event’s HANDLE is requested, and/or it is passed to other Win32 APIs like MsgWaitForMultipleObjects, then the isolation of the faux transaction is broken, and real state must be provided to the Win32 APIs. And if another thread were to look up that HANDLE – perhaps through a name given to it in the call to CreateEvent – it may be able to see and interact with that event before the enclosing transaction has been committed. The abstraction leaks. And even if the abstraction is perfect, it is obvious there’s quite a bit of work to be had in order to transact all the touch points between .NET and Win32, of which there are many. And I mean many.

Other issues wait just around the corner. For example, how would you treat a lock block that was called from within a transaction? (You might say “that’s just not supported”, but when adding TM to an existing, large ecosystem of software, you’ve got to draw the compatibility line somewhere. If you draw it too carelessly, large swaths of existing software will not work; and in this case, that often meant that we claimed to provide unbounded transactions, and yet we would be placing bounds on them such that a lot of existing software could not be composed with transactions. Not good.) A seemingly straightforward answer is to treat a lock block like an atomic block. So if you encounter:

atomic {

lock (obj) { … }

}

it is logically transformed into:

atomic {

atomic { … }

}

On the face of it, this looks okay. (Forget problems like freeform use of Monitor.Enter/Exit for now.) We’re strengthening the atomicity and isolation, so what could go wrong? Well, it turns out that examples like this can also suffer from the “too much isolation” problem. Adding transactions to a lock-block extends the lifetime of the isolation of that particular block’s effects, possibly leading to lack of forward progress. In fact, you don’t need locks to illustrate the problem. Imagine a simple lock-free algorithm that communicates between threads using shared variables:

volatile int flag = 0;

…

flag = 1; while (flag != 1) ;

while (flag == 1) ; flag = 2;

If you invoke this code from within a transaction (on each thread), you’re apt to lead to deadlock. Both transactions’ effects will be isolated from the others’, whereas we are quite obviously intending to publish the updates to the flag variable immediately.

Anyway, the whole lock thing is a bit of a digression. The simple fact is that very little .NET code would actually run inside an atomic block but for things like collections and pure computations due to the I/O problem. You can develop one-off solutions for each problem that arises – and indeed we did so for many of them – and even hang those solutions underneath one general framework – like System.Transactions – but you cannot help but eventually become overwhelmed by the totality of the situation. The team experimented with static checking to turn these dynamic failures into static ones, but this only marginally improved matters.

I could go on and on about the I/O problem, its various incarnations, and what we did about it. Instead I will sum it up: this problem was, and still is, the “elephant in the room” threatening unbounded TM’s broader adoption.

The question ultimately boils down to this: is the world going to be transactional, or is it not?

Whether unbounded transactions foist unto the world will succeed, I think, depends intrinsically on the answer to this question. It sure looked like the answer was going to be “Yes” back when transactional NTFS and registry was added to Vista. But the momentum appears to have slowed dramatically.

Nesting

Let’s get back to some fun, less depressing material. There are more surprises lurking ahead.

I already mentioned a great virtue of transactions is their ability to nest. But I neglected to say how this works. And in fact when we began, we only recognized one form of nesting. You’re in one atomic block and then enter into another one. What happens if that inner transaction commits or rolls back, before the fate of the outer transaction is known? Intuition guided us to the following answer:

If the inner transaction rolls back, the outer transaction does not necessarily do so. However, no matter what the outer transaction does, the effects of the inner will not be seen.
If the inner transaction commits, the effects remain isolated in the outer transaction. It “commits into” the outer transaction, we took to saying. Only if the outer transaction subsequently commits will the inner’s effects be visible; if it rolls back, they are not.

For example, consider this code:

void f() { void g() {

atomic { // Tx0 atomic { // Tx1

x++; y++;

try { if (p1)

g(); throw new BarException();

} catch { }

if (p0) }

throw; }

}

if (p2)

throw new FooException();

}

Imagine x = y = 0 at the start, and we invoke f. Many outcomes are possible.

If p1 is true, g will throw an exception, aborting Tx1’s write to y. There are then two possibilities. (1)If p0 is true, the exception is repropagated and Tx0 will also abort, rolling back its write to x; this leaves x == y == 0. (2) If p0 is false, the exception is swallowed, and Tx0 proceeds to committing its write to x; this leaves x == 1, whereas y == 1.
If p1 is false, on the other hand, g will not throw anything. Tx1 will commit its write to y “into” the outer transaction Tx0. One of two outcomes will now occur depending on the value of p2. (1) If p2 is true, an exception is thrown out of f, and Tx0 rolls back both the inner transaction Tx1’s effects and its own, leaving x == y == 0. (2) Else, f completes ordinarily, and Tx0 commits both Tx1’s and its own effects, leading to x == y == 1.

We expected most peoples’ intuition to match this behavior.

The canonical working example was a BankAccount class:

class BankAccount
{

decimal m_balance;

public void Deposit(decimal delta) {

atomic { m_balance += delta; }

}

public static void Transfer(

BankAccount a, BankAccount b, decimal delta) {

atomic {

a.Deposit(-delta);

b.Deposit(delta);

}

This was an illustrative and beautiful example. It made beautiful slide-ware. We are composing the Deposit operations of two separate bank accounts into a single Transfer method. Of course doing the a.Deposit(-delta) and b.Deposit(delta) must be made atomic, else a failure could either lead to missing money, and/or someone could witness the world with the money in transit (and nowhere except for one a thread’s stack) rather than having been transferred atomically. And building the same thing with locks is frustratingly difficult: using fine-grained per-account locks can lead to deadlock very quickly.

Intuitively we walked down many variants of this mode of nesting. We reacquainted ourselves with Moss’s great dissertation on the topic, and remembered this intuitive nesting mode as closed nested transactions. And we shortly recognized the need for another mode: open nested transactions.

To motivate the need for open nesting, imagine we’ve got a hashtable whose physical storage is independent from its logical storage. Resizing the table of buckets, for example, has little to do with whether a particular {key, value} pair exists within those buckets. The resizing operation, in fact, is logically idempotent and isolated: the same set of keys will exist within the table both before and after such an operation. So we can actually commit the physical effects of such an operation eagerly. With a naïve TM implementation, two independent keys hashing to the same bucket will conflict, and the reads and writes for such operations will live as long as the enclosing user-level transactions. Instead, we can serialize logical operations with respect to one another at a “higher level” than physically independent operations do, leading to greater concurrency. Two transactions will only conflict in long-running transactions if they truly operate on the same keys, rather than just happening to hash to the same bucket.

Open nesting forced us to contemplate the sharing of state between outer and inner transactions more deliberately, and gave us some troubles syntactically. We had wanted to say:

atomic { // ordinary closed nesting.

Foo f = new Foo();

atomic(open) { /// open nesting.

… f? …

}

But is it really legal for the inner transaction here to access the ‘f’, which has been constructed and is presumably uncommitted in the outer transaction? With closed nested transactions there is lock compatibility between the outer and inner transactions. An inner closed nested transaction can of course read a memory location write-locked by the outer transaction, for example. However, the same must not true of open nesting, because an open nested transaction commits “to the world” rather than into its outer transaction. Allowing it to read and then potentially publish uncommitted state would violate serializability. It’s possible that the inner open nested transaction will commit, whereas the outer will roll back. (The reverse situation is equally problematic.) And yet it’s darn useful to pass state from an outer to an inner transaction – and indeed, often impossible to do anything otherwise – yet what if the key itself were a complicated object graph rather than value, and the key bleeds across transaction boundaries?

Many issues like this arose. Our straightforward answer was that only pass-by-value worked across such a boundary. I don’t think we ever found nirvana here.

We developed other transaction modes also.

As we added data parallel operations within a nested transaction, we realized that we’d need something a lot like closed nesting but with special accommodation for intra-transaction parallelism. This led us to parallel nested transactions, enabling lock sharing from a parent to its many data parallel children. These children could not communicate with one another other than to “commit into” the parent, and subsequently reforking, thereby ensuring non-interference between them. Of course children could share read-locks amongst one another, just not write locks.

And we continued to reject the temptation of adding weakened serializability modes a la relational databases (unrepeatable reads, etc). Although we expected this to arise out of necessity with time, it never did; the various nesting modes we provided seemed to satisfy the typical needs.

A Better Condition Variable

Here’s a brief aside on one of TM’s bonus features.

Some TM variants also provide for “condition variable”-like facilities for coordination among threads. I think Haskell was the first such TM to provide a ‘retry’ and ‘orElse’ capability. When a ‘retry’ is encountered, the current transaction is rolled back, and restarted once the condition being sought becomes true. How does the TM subsystem know when that might be? This is an implementation detail, but one obvious choice is to monitor the reads that occurred leading up to the ‘retry’ – those involved in the evaluation of the predicate – and once any of them changes, to reschedule that transaction to run. Of course, it will reevaluate the predicate and, if it has become false, the transaction will ‘retry’ again.

A simple blocking queue could be written this way. For example:

object TakeOne()

{

atomic {

if (Count == 0)

retry;

return Pop();

}

If, upon entering the atomic block, Count is witnessed as being zero, we issue a retry. The transaction subsystem notices we read Count with a particular version number, and then blocks the current transaction until Count’s associated version number changes. The transaction is then rescheduled, and races to read Count once again. After Count is seen as non-zero, the Pop is attempted. The Pop, of course, may fail because of a race – i.e. we read Count optimistically without blocking out writers – but the usual transaction automatic-reattempt logic will kick in to mask the race in that case.

The ‘orElse’ feature is a bit less obvious, though still rather useful. It enables choice among multiple transactions, each of which may end up issuing a ‘retry’. I don’t think I’ve seen it in any TMs except for ours and Haskell’s.

To illustrate, imagine we’ve got 3 blocking queues like the one above. Now imagine we’d like to take from the first of those three that becomes non-empty. ‘orElse’ makes this simple:

BlockingQueue bq1 = …, bq2 = …, bq3 = …;

atomic {

object obj =

orElse {

bq1.TakeOne(),

bq2.TakeOne(),

bq3.TakeOne()

};

}

While ‘orElse’ is perhaps an optional feature, you simply can’t write certain kinds of multithreaded algorithms without ‘retry’. Anything that requires cross-thread communication would need to use spin variables.

Deliberate Plans of Action: Policy

I waved my hands a bit above perhaps without you even knowing it. When I talk about optimistic, pessimistic, and automatic retry, I am baking in a whole lot of policy. It turns out there is a wide array of techniques. The simplest question we faced early on was, when an optimistic read fails to validate at the end of a transaction, when should we reattempt execution of that transaction?

The naïve answer is “immediately”. But obviously that would lead to livelock under some conditions. A more reasonable answer is “spin for N cycles and then retry”. But this too can lead to livelock. A better answer is to either choose some random strategy, or to make an intelligent adaptive choice. We experimented with many such variants, including random backoff, sophisticated waiting and signaling based on the memory locations in question, among others. We even played games like giving transactions karma points for cooperatively acquiescing to other competing transactions, and allowing those transactions with the most karma points to make more forward progress before interrupting them.

A few good papers supplied useful (and entertaining) reading material on the topic, but to be honest, nobody had a good answer at the time. Thankfully these are all implementation details. So we were free to experiment.

Deadlock breaking also requires policy. Thankfully we can actually roll back the effects of transactions engaged in a deadly embrace with TM, so we merely need to know how often to run the deadlock detection algorithm. There was a similar problem when deciding to back off outer layers of nesting, and in fact this becomes more complicated when deadlocks are involved. Imagine:

atomic { atomic {

x++; y++;

atomic { atomic {

y++; x++;

} }

This deadlock-prone example is tricky because rolling back the inner-most transactions won’t be sufficient to break the deadlock that may occur. Instead the TM policy manager needs to detect that multiple levels of nesting are involved and must be blown away in order to unstick forward progress.

Another variant that went beyond deciding when to favor one transaction over another was to upgrade to pessimistic locking if optimistic let us down. The whole justification behind optimistic is that, …well, we’re optimistic that conflicts won’t happen. So it seems reasonable that, if they do occur, we fall back to something more, …well, pessimistic. There is a dial here too. Perhaps you only want to fall back to pessimistic after failing optimistically N times in a row, where N > 1. As I mentioned above, our single-word lock associated with each object supported both locking and versioning cheaply.

Disillusionment Part II: Weak or Strong Atomicity?

All along, we had this problem nipping at our heels. What happens if code accesses the same memory locations from inside and outside a transaction? We certainly expected this to happen over the life of a program: state surely transitions from public and shared among threads to private to a single thread regularly. But if some location were to be accessed transactionally and non-transactionally concurrently, at once, we’d (presumably) have a real mess on our hands. A supposedly atomic, isolated, etc. transaction would no longer be protected from the evils of racey code.

For example:

atomic { // Tx0 x++; // No-Tx

x++;

}

Can we make any statements about the value of x after Tx0 commits (or rolls back)? Not really. It depends on the way the particular TM being used has been implemented. An in-place model that rolls back could not only roll back Tx0’s but also the unprotected x++’s write. And so on.

On one hand, this code is racey. So you could explain away the undefined behavior as being a race condition. On the other hand, it was also troublesome. All those problems with locks begin cropping up all over the place. It would have been ideal if we could notify developers that they made a mistake. Then we could have made the assertion that data races are simply not possible with TM.

(Except for consistency-related ones, of course.)

At the same time, many hardware models were being explored. And of course in hardware you’ve got the physical addresses that variables resolve to and needn’t worry about aliasing. So it was actually possible to issue a fault if a location was used transactionally and non-transactionally at once. But given that our solution was software-based, we were uncomfortable betting the farm on hardware support.

Another approach was static analysis. We could require transactional locations to be tagged, for example. This had the unfortunate consequence of making reusable data structures less, well, reusable. Collections for example presumably need to be usable from within and outside transactions alike. After-the-fact analysis could be applied without tagging, but false positives were common. We never really took a hard stance on this problem, but always assumed the combination of static analysis, tooling, and, perhaps someday, hardware detection would make this problem more diagnosable. But I think we generally resolved ourselves to the fact that our TM would suffer from weak atomicity problems.

We thought this was explainable. Sadly it led to something that surely was not.

Disillusionment Part III: the Privatization Problem

I still remember the day like it was yesterday. A regular weekly team meeting, to discuss our project’s status, future, hard problems, and the like. A summer intern on board from a university doing pioneering work in TM, sipping his coffee. Me, sipping my tea. Then that same intern’s casual statement pointing out an Earth-shattering flaw that would threaten the kind of TM we (and most of the industry at the time) were building. We had been staring at the problem for over a year without having seen it. It is these kinds of moments that frighten me and make me a believer in formal computer science.

Here it is in a nutshell:

bool itIsOwned = false;

MyObj x = new MyObj();

…

atomic { // Tx0 atomic { // Tx1

// Claim the state for my use: if (!itIsOwned)

itIsOwned = true; x.field += 42;

} }

int z = x.field;

...

The Tx0 transaction changes itIsOwned to true, and then commits. After it has committed, it proceeds to using whatever state was claimed (in this case an object referred to by variable x) outside of the purview of TM. Meanwhile, another transaction Tx1 has optimistically read itIsOwned as false, and has gone ahead to use x. An update in-place system will allow that transaction to freely change the state of x. Of course, it will roll back here, because isItOwned changed to true. But by then it is too late: the other thread using x outside of a transaction will see constantly changing state – torn reads even – and who knows what will happen from there. A known flaw in any weakly atomic, update in-place TM.

If this example appears contrived, it’s not. It shows up in many circumstances. The first one in which we noticed it was when one transaction removes a node from a linked list, while another transaction is traversing that same list. If the former thread believes it “owns” the removed element simply because it took it out of the list, someone’s going to be disappointed when its state continues to change.

This, we realized, is just part and parcel of an optimistic TM system that does in-place writes. I don’t know that we ever fully recovered from this blow. It was a tough pill to swallow. After that meeting, everything changed: a somber mood was present and I think we all needed a drink. Nevertheless we plowed forward.

We explored a number of alternatives. And so did the industry at large, because that intern in question published a paper on the problem. One obvious solution is to have a transaction that commits a change to a particular location wait until all transactions that have possibly read that location have completed – a technique we called quiescence. We experimented with this approach, but it was extraordinarily complicated, for obvious reasons.

We experimented with blends of pessimistic operations instead of optimistic, alternative commit protocols, like using a “commit ticket” approach that serializes transaction commits, each of which tended to sacrifice performance greatly. Eventually the team decided to do buffered writes instead of in-place writes, because any concurrent modifications in a transaction will simply not modify the actual memory being used outside of the transaction unless that transaction successfully commits.

This, however, led to still other problems, like the granular loss of atomicity problem. Depending on the granularity of your buffered writes – we chose object-level – you can end up with false sharing of memory locations between transactional and non-transactional code. Imagine you update two separate fields of an object from within and outside a transaction, respectively, concurrently. Is this legal? Perhaps not. The transaction may bundle state updates to the whole object, rather than just one field.

All these snags led to the realization that we direly needed a memory model for TM.

Disillusionment Part IV: Where is the Killer App?

Throughout all of this, we searched and searched for the killer TM app. It’s unfair to pin this on TM, because the industry as a whole still searches for a killer concurrency app. But as we uncovered more successes in the latter, I became less and less convinced that the killer concurrency apps we will see broadly deployed in the next 5 years needed TM. Most enjoyed natural isolation, like embarrassingly parallel image processing apps. If you had sharing, you were doing something wrong.

In Conclusion

I eventually shifted focus to enforcing coarse-grained isolation through message-passing, and fine-grained isolation through type system support a la Haskell’s state monad. This would help programmers to realize where they accidentally had sharing, I thought, rather than merely masking this sharing and making it all work (albeit inefficiently).

I took this path not because I thought TM had no place in the concurrency ecosystem. But rather because I believed it did have a place, but that several steps would be needed before getting there.

I suspected that, just like with Argus, you’d want transactions around the boundaries. And that you’d probably want something like open nesting for fine-grained scalable data structures, like shared caches. These are often choke points in a coarse-grained locking system, and often cannot be fully isolated, at least in the small. Ironically I am just now arriving there. In the system I work on I see these issues actually staring us in the face.

This is just my own personal view on TM. You may also be interested in reading the current STM.NET team’s views also, available on their MSDN blog.

For me the TM project was particularly enjoyable. And it was a great learning experience. I worked with some amazing people, and it was a privilege. You really had the sense that something big was right around the corner, and every day was a rush of enjoyment. Despite running as fast as we could, it seemed like we could just barely keep pace with the research community. Over time more and more researchers turned to TM, and I distinctly recall reading at least one new TM paper per week.

This was also the first time I realized that Microsoft, at its core, really does operate like a collection of many startups. Our TM work was a grassroots movement, and there was no official sponsorship for our effort at the start. It was just a group of people independently getting together to discuss how TM might fit into the direction the industry was headed. Eventually TM started showing up on slide decks in presentations to management, followed by dedicated TM reviews, and even a BillG review. I will never forget, a couple years after that review – during an overall concurrency review – Bill standing up at the whiteboard, drawing the code “atomic { … }” and asking something to the effect: “Why can’t you just use transactional memory for that?” I guess the idea stuck with him too.

Who knows. Maybe in 10 years, the world will be transactional after all.

Lifting T out of Task with dynamic dispatch

Sun, 01 Nov 2009 21:49:28 GMT

Say you've got a Task<T>. Well, now what?

You know that eventually a T will become available, but until then you're out of luck. You could go ahead and be a naughty little devil by calling Wait on it -- blocking the current thread (eek!) -- or you could call ContinueWith on the task to get back a new Task, representing the work you would do to create some new U object if only you presently had a T in hand. And then perhaps you will find yourself in the same situation for that U.

These are those dataflow graphs I mentioned in the previous blog post. Things of beauty.

To be more concrete about the situation I describe, imagine you've got the following IFoo interface:

interface IFoo

{

int Bar();

string Baz(int x);

}

Now, given a Task<IFoo>, you can't do anything related to an IFoo. And yet presumably that's why you've got the task in the first place: because you care about the IFoo. What if you ultimately want to invoke the Bar method, for example?

Task<IFoo> task = ...;

You can of course block the thread:

// Option A: block the thread.

int resultA = task.Result.Bar();

...

Or you can choose to program in a very clunky way:

// Option B: use dataflow.

Task<int> resultB = task.ContinueWith(t => t.Result.Bar());

But what if, instead, you could do something like this?

// Option C: magic.

Task<int> resultC = task.Bar();

Whoa, wait a minute. We're calling Bar() on a Task<IFoo>? Neat, but how can that be?

This is obviously a trick. All of the members of T are somehow being made available on the Task<T> object, so that they can be called before the task has actually been resolved to a concrete value. Of course, were we to allow this, what you get back to represent the result of such calls would need to be task objects too: hence we get back a Task<int> from the call on Bar(), instead of an int. This is similar to call streams in Barbara Liskov's Argus language (her primary focus immediately after CLU).

This kind of lifting from the inner type outward is much like what you get in languages that allow generic mixins. C# already has one semi-such type, though you may not realize it: Nullable<T> actually allows you to directly access interfaces implemented by T without needing to call Value on it. It's almost like Nullable<T> was defined as deriving from T itself which is clearly not actually possible (for numerous reasons, not the least significant of which is that it's a struct). Try it. This works because the type system treats Nullable<T> and T somewhat uniformly (though you'd be surprised by some dangers lurking within -- effectively Nullable<T> mustn't implement any interfaces *ever* otherwise a type hole would result). But I digress...

Unfortunately without deep language changes we can't get this to work the way we'd like. I have found numerous occasions where a general lifting capability in C# would be useful: Lazy<T> is but one example. That said, each time we run across an instance, it demands slightly different type system treatment, and it seems unlikely such a general facility would be as usable as the one off features.

Type systems aside, I am actually using a very dirty trick to make this work: I'm using the new System.Dynamic features in .NET 4.0 to do it all dynamically. You may love or hate this, depending on your stance on type systems. Being an ML guy, I'll let you figure out what I think. (Hint: gross hack!)

We can go further. (Although sadly I won't demonstrate how to do so in this blog post. I had wanted to go all the way, but need to get some actual language work done today, in addition to a little Riemann study, instead of having endless fun tinkering with Visual Studio 2010. Shucks.) Notice that Baz accepts an int as input. Well, what if all we've got is a Task<int>? We can of course also allow that to get passed in too:

Task<string> resultD = task.Baz(42); // Real input. Fine.

Task<int> arg = ...;

Task<string> resultE = task.Baz(arg); // A task as input! Cool!

But wait, there is more! It slices and dices too. The next trick is difficult -- if not impossible -- to do without far reaching language changes. But we could also even bridge the world of ordinary methods too, not just those that have been accessed by tunneling through a Task<T>. For example:

string f(int x) {...}

...

Task<int> task = ...;

Task<string> result = f(task);

Not to even mention:

Task<int> x = ...;

Task<int> y = ...;

Task<int> z = x + y;

This is deep. What we are saying is that anywhere a T is expected, we can supply a Task<T>. Of course once we've entered the world of tasks, we cannot escape until values actually begin resolving. So when we invoke the method f in this example, we of course get back a Task<string> for its result. Once we've stepped onto a turtle's back, well, it's turtles all the way down.

(Which reminds me of the well known tale:

A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: "What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise." The scientist gave a superior smile before replying, "What is the tortoise standing on?" "You're very clever, young man, very clever", said the old lady. "But it's turtles all the way down!"

Tasks are not greasy hamburgers after all, as I had claimed in the last post, but rather they are turtles.

I've wasted all of my energy speaking of turtle hamburgers drenched in asynchronous aioli, and have left only a little to go over the hacked up implementation of this idea. Sigh. Well, we had better get to it.)

In summary: we'll just rely on dynamic dispatch to do the lifting, thanks to the new .NET 4.0 DynamicObject class. This is wildly less efficient than a proper type system design would yield, not to mention the utter lack of static type checking. Of course a proper implementation that designed for this from Day One would also avoid the tremendous amount of object allocation that relying on the current Task<T> objects and ContinueWith overloads imply. But nevertheless, this approach will allow us to at least have a good ole' time and stimulate the creative side of the noggin.

First, I shall provide an extension method for getting a DynamicTask<T> -- the thing that actually derives from DynamicObject and implements the custom dynamic binding:

public static class DynamicTask

{

public static dynamic AsDynamic<T>(this Task<T> task)

{

return new DynamicTask<T>(task);

}

Notice that this changes our calling conventions ever so slightly. Namely:

// Option C: magic.

Task<int> resultC = task.AsDynamic().Bar();

The AsDynamic places the caller into the lifted context. As invocations are made, the results become real tasks, and not dynamic ones, such that to continue the calling will require many AsDynamic()s. This is a minor inconvenience and we could certainly automatically wrap the return values in DynamicTask<T> objects if we wanted to eliminate this problem, i.e. to make chaining less verbose.

Second, we must implement the DynamicTask<T> class. We will do a very simple translation. Given a member access expression 'x.m', where m is either a field or property of type U, we will morph this into the new expression 'x.Task.ContinueWith(v => v.Result.m)', which is of type Task. Similarly, given a method invocation 'x.M(a1,...,aN)', whose return value is of type U, we will morph it into the new expression 'x.Task.ContinueWith(v => v.Result.M(a1,...,aN))', which is of type Task (or just Task if U is the void type). To support the ability to pass a task argument where an actual one is expected would require packing the argument with the target into an array, and doing a ContinueWhenAll on it.

(Perhaps I will illustrate how to do these other tricks in a later post, but I'm tight for time right now. I'm only sketching the general idea. Even in what I show below, things will be incomplete, because topics such as getting exception propagation right when tasks begin failing are tricky. Ideally the whole dataflow chain will be "broken" by such an exception. Additionally, I've only implemented what was necessary to get a few interesting examples working. The binder, for example, certainly has a few loose ends. Blog reader beware.)

Here is the implementation of DynamicTask<T>:

public class DynamicTask<T> : DynamicObject

{

private Task<T> m_task;

public DynamicTask(Task<T> task)

{

if (task == null) {

throw new ArgumentNullException("task");

}

m_task = task;

}

public Task<T> Task {

get { return m_task; }

}

public override DynamicMetaObject GetMetaObject(Expression parameter) {

if (parameter == null) {

throw new Exception("parameter");

}

return new TaskLiftedObject(this, parameter);

}

class TaskLiftedObject : DynamicMetaObject

{

...

}

Simple. All of the dynamic magic resides in the implementation of TaskLiftedObject, which derives from the DynamicMetaObject class. It is constructed with an instance of the DynamicTask<T> along with the expression tree that can be used to dynamically load up an instance of that task. All of the dynamic features work with expression trees. For example, in response to an attempt to invoke a method M on a DynamicTask<T>, our binder will need to find the right method M on the underlying T, and then return an expression tree that does the ContinueWith and so forth.

Let's start cracking open TaskLiftedObject:

class TaskLiftedObject : DynamicMetaObject

{

private DynamicTask<T> m_task;

public TaskLiftedObject(DynamicTask<T> task, Expression expression) :

base(expression, BindingRestrictions.Empty, task)

{

m_task = task;

}

We will override two of DynamicMetaObject's functions. BindGetMember is called when a member is accessed (like a property or field), whereas BindInvokeMember is called when a method call is made. There are several other methods that a proper binder would need to override in order to make delegate dispatch and such work properly. But this suffices to get started:

public override DynamicMetaObject BindGetMember(GetMemberBinder binder)

{

// We have a member access:

// x.m

// which must become:

// x.Task.ContinueWith(v => { v.Result.m; })

return new DynamicMetaObject(

MakeContinuationTask(Bind(binder.Name, -1), null),

BindingRestrictions.GetInstanceRestriction(Expression, Value),

Value

);

}

public override DynamicMetaObject BindInvokeMember(InvokeMemberBinder binder, DynamicMetaObject[] args)

{

// We have a call:

// x.Foo(a1,...,aN)

// which must become:

// x.Task.ContinueWith(v => { v.Result.Foo(a1,...,aN); })

Expression[] argsEx = new Expression[args.Length];

for (int i = 0; i < args.Length; i++) {

argsEx[i] = args[i].Expression;

}

return new DynamicMetaObject(

MakeContinuationTask(Bind(binder.Name, binder.CallInfo.ArgumentCount), argsEx),

BindingRestrictions.GetInstanceRestriction(Expression, Value),

Value

);

}

Clearly the workhorses here are Bind and MakeContinuationTask. Bind is responsible for performing dynamic lookup for a matching member on T that has the requested Name and, if a method call is being made, the proper number of parameters. For brevity, I've omitted anything to do with argument type checking, an obvious hole that we'd want to fix some day:

private static MemberInfo Bind(string name, int argCount)

{

// Lookup the target member on the T, rather than the (Dynamic)Task<T>.

return

(from m in typeof(T).GetMembers(BindingFlags.Instance | BindingFlags.Public)

where m.Name.Equals(name) &&

(argCount == -1 ?

!(m is MethodInfo) :

((MethodInfo)m).GetParameters().Length == argCount)

select m).

Single();

}

Nothing too interesting here either -- just a bit of hacky reflection code done with a fancy LINQ query. If anything other than exactly one method was found, the call to Single() will throw an exception. If you want to see what a "real" dynamic binder looks like, you won't find it here: check out VB's or IronPython's.

Now for the meat. The MakeContinuationTask method takes the target member that we've found dynamically via Bind, as well as an optional array of expression trees, each representing an argument being passed to the target method (and which will be null for property and field access), and manufactures the expression tree that represents the execution of the dynamic call itself:

private Expression MakeContinuationTask(MemberInfo target, Expression[] targetArgs)

{

var lambdaParam = Expression.Parameter(typeof(Task<T>), "v");

var lambdaParamResult = Expression.Property(lambdaParam, "Result");

Expression lambdaBody;

Type lambdaReturnType;

if (target is MethodInfo) {

lambdaBody = Expression.Call(lambdaParamResult, (MethodInfo)target, targetArgs);

lambdaReturnType = ((MethodInfo)target).ReturnParameter.ParameterType;

}

else if (target is PropertyInfo) {

lambdaBody = Expression.Property(lambdaParamResult, (PropertyInfo)target);

lambdaReturnType = ((PropertyInfo)target).PropertyType;

}

else if (target is FieldInfo) {

lambdaBody = Expression.Field(lambdaParamResult, (FieldInfo)target);

lambdaReturnType = ((FieldInfo)target).FieldType;

}

else {

throw new Exception("Unsupported dynamic invoke: " + target.GetType().Name);

}

return Expression.Call(

Expression.Property(

Expression.Convert(this.Expression, typeof(DynamicTask<T>)),

typeof(DynamicTask<T>).GetProperty("Task")

GetContinueWith(lambdaReturnType), // ContinueWith

new Expression[] {

// v => { v.Result.M(a0,...,aN) }

Expression.Lambda(lambdaBody, lambdaParam)

}

);

}

You should be able to convince yourself that this code generates the desired transformation described earlier. It uses a method to find the overload of Task<T>.ContinueWith that we want to bind against, and invokes that on the Task<T> contained within the DynamicTask<T> against which the dynamic call was made. It is rather unfortunate that the CLR does not allow the void type as a generic type argument, so we have to be a little bit inconsistent with our treatment of void returns, by choosing a different ContinueWith overload.

If the above reflection code was called hacky, the ContinueWith lookup is worse. It's very inefficient, not to mention fragile (because it depends on the current layout of Task<T>'s overloads, what with instantiating generic methods and the like). C'est la vie:

private static MethodInfo GetContinueWith(Type returnType)

{

// @TODO: caching to avoid expensive lookups each time.

if (returnType == typeof(void)) {

return typeof(Task<T>).GetMethod(

"ContinueWith",

new Type[] { typeof(Action<Task<T>>) }

);

}

else {

foreach (MethodInfo mif in typeof(Task<T>).GetMethods()) {

if (mif.Name == "ContinueWith" && mif.IsGenericMethodDefinition) {

MethodInfo mifOfT = mif.MakeGenericMethod(returnType);

ParameterInfo[] mifParams = mifOfT.GetParameters();

if (mifParams.Length == 1 &&

mifParams[0].ParameterType == typeof(Func<,>).MakeGenericType(typeof(Task<T>), returnType)) {

return mifOfT;

}

throw new Exception("Fatal error: ContinueWith overload not found");

}

And that's it. With that, we can get dynamic invocations on unresolved T's via Task<T> objects. Nifty.

I'm not saying any of this is a really good idea. Honestly, I'm not. Of course, there's a kernel of a good idea there and the systems we are working on take this kernel to its extreme. By providing a programming model that encourages deep chains of datafow to be expressed speculatively in a natural and familiar manner, greater degrees of latent parallelism can lie resident in an application waiting to be unlocked as more processors become available. Doing it for real requires impactful changes to the language, supporting infrastructure, and particularly tooling. Just imagine what it means to break into a debugger to inspect deep dataflow graphs that have been constructed by compiler magic underneath you. And the use of ContinueWith is a little lame, because of course the target of our call may be something that can be run speculatively too with first class pipleining, rather than completely delaying the invocation of it.

So we won't be seeing lifted tasks in .NET anytime soon. Writing up this blog post was merely an excuse to toy around with the new C# dynamic features and to have a little recreational time. And to generate excitement about what .NET 4.0 holds in store. I hope you have enjoyed it. Now back to reality.

Tasks and asynchronous control flow

Sun, 01 Nov 2009 04:06:24 GMT

Well, Visual Studio 2010 Beta 2 is out on the street. It contains plenty of neat new things to keep one busy for at least a rainy Saturday. I proved this today.

Of course, Parallel Extensions is in the box. .NET 4.0's Task and Task<T> abstractions are used to implement such things as PLINQ and Parallel.For loops, but of course they are great for representing asynchronous work too. The FromAsync adapters move you from the dark ages of IAsyncResult to the glitzy new space age of tasks.

Not only are tasks tastier than hamburgers, but they enable complex dataflow graphs of asynchronous work to unfold dynamically at runtime, thanks to the ContinueWith method. From a Task<T> you can get a Task that was computed based on the T; ad infinitum. We like dataflow. It is the key to unlocking parallelism, or more accurately, boiling away all else except for dataflow is the key. But what about control flow, you might ask? We like it less. But you can do it, so long as you put in some work. F#'s async workflows make this sort of thing a tad easier, but the raw libraries in .NET 4.0 don't come with any sort of loops or conditional capabilities. Perhaps in the future they will. Nevertheless, in this post I shall demonstrate how to build a couple simple ones.

Not because the lack of them is going to cause unprecidented and unheard of horrors, but rather because in doing so we'll see some neat features of tasks.

The two methods I will illustrate in this post are:

public static class TaskControlFlow

{

public static Task For(int from, int to, Func<int, Task> body, int width)

public static Task While(Func<int, bool> condition, Func<int, Task> body, int width)

}

Notice that each body is given the iteration index and is expected to launch asynchronous work and return a Task. The parameters that these methods take are probably obvious. Well, except for the last one. The "width" indicates how many outstanding asynchronous bodies should be in flight at once. The Task returned by For and While won't be considered done until all iterations are done, and any exceptions will be propagated as you might hope. It would be pretty useless otherwise.

For example, we could write a while loop that does something very silly:

TaskControlFlow.While(

 i => i < 100,

 i => { return CreateTimerTask(250).ContinueWith(_ => Console.WriteLine(i)); },

 4

).Wait();

This just prints returns a "timer task" that completes after 250ms and prints out the iteration to the console. We pass a width of 4, so only four tasks will be outstanding at any given time. Notice we call Wait at the end, since both For and While return tasks representing the in flight work. This could have instead been written using a For loop as follows:

TaskControlFlow.For(0, 100,

i => { return CreateTimerTask(250).ContinueWith(_ => Console.WriteLine(i)); },

4

).Wait();

The CreateTimerTask method, by the way, looks like this:

private static Task CreateTimerTask(int ms)

{

 var tcs = new TaskCompletionSource<bool>();

 new Timer(x => ((TaskCompletionSource<bool>)x).SetResult(true), tcs, ms, -1);

 return tcs.Task;

}

As something more realistic, imagine we wanted to do something with a large number of files, and don't want to block a whole bunch of threads in the process. The following "simple" expression will count up all of the bytes for all of the files in a particular directory, without once blocking the thread -- well, except for the initial call to Directory.GetFiles:

string win = "c:\\...\\";

string[] files = Directory.GetFiles(win);

int total = 0;

TaskControlFlow.For(0, files.Length,

 i => {

 bool eof = false;

 int offset = 0;

 byte[] buff = new byte[4096];

 FileStream fs = File.OpenRead(files[i]);

 return TaskControlFlow.While(

 j => !eof,

 j => Task<int>.Factory.

 FromAsync<byte[],int,int>(

 fs.BeginRead, fs.EndRead, buff, offset, buff.Length,

 null, TaskCreationOptions.None

 ).

 ContinueWith(v => {

 if (eof = v.Result < buff.Length) {

 fs.Close();

 }

 offset += v.Result;

 Interlocked.Add(ref total, v.Result);>
 }),

 1

 );

 },

 8

).Wait();

Console.WriteLine(total);

Pretty neat. We've somewhat arbitrarily chosen a width of 8 for this loop. And notice something very subtle but important here: we've chosen a width of 1 for the inner loop that plows through the bytes of a file. This is because we're sharing state, and it would not be safe to launch numerous iterations at once. The same byte[], eof variable, and so forth, would become corrupt. I will mention in passing that it's unfortunate that we've got that interlocked stuck in there to add to the total. Refactoring this so that we could just do a LINQ reduce over the whole thing would be nice. Indeed, it can be done.

We can do away with the For implementation very quickly. It is just implemented in terms of While:

public static Task For(int from, int to, Func<int, Task> body, int width)

{

return While(i => from + i < to, body, width);

}

And it turns out that the While implementation is not terribly complicated either. Here it is:

public static Task While(Func<int, bool> condition, Func<int, Task> body, int width)

{

 var tcs = new TaskCompletionSource<bool>();

 int currIx = -1; // Current shared index.

 int currCount = width; // The number of outstanding tasks.

 int canceled = 0; // 1 if at least one body was cancelled.

 ConcurrentBag<Exception> exceptions = null; // A collection of exceptions, if any.

 // Generate a continuation action: this fires for each body that completes.

 Action<Task> fcont = null;

 fcont = tsk => {

 if (tsk.IsFaulted) {

 // Accumulate exceptions.

 LazyInitializer.EnsureInitialized(ref exceptions);

 foreach (Exception inner in tsk.Exception.InnerExceptions) {

 exceptions.Add(inner);

 }

 }

 else if (tsk.IsCanceled) {

 // Mark that cancellation has occurred.

 canceled = 1;

 }

 else if (canceled == 0 && exceptions == null) {

 // If no cancellations / exceptions are found, attempt to kick off more work.

 int ix = Interlocked.Increment(ref currIx);

 if (condition(ix)) {

 // Generate a new body task, handling exceptions. Then make sure we

 // tack on the continuation on that new task, so we can keep on going...

 // If the condition yielded 'false', we'll simply fall through and try to finish.

 Task btsk;

 try {

 btsk = body(ix);

 }

 catch (Exception ex) {

 btsk = AlreadyFaulted(ex);

 }

 btsk.ContinueWith(fcont);

 return;

 }

 }

 // If this is the last task, signal completion.

 if (Interlocked.Decrement(ref currCount) == 0) {

 if (exceptions != null) {

 tcs.SetException(exceptions);

 }

 else if (canceled == 1) {

 tcs.SetCanceled();

 }

 else {

 tcs.SetResult(true);

 }

 }

 };

 // Fire off the right number of starting tasks.

 for (int i = 0; i < width; i++) {

 AlreadyDone.ContinueWith(fcont);

 }

 return tcs.Task;

}

I've commented the code inline to illustrate what is going on. The only other part that isn't shown are the AlreadyDone and AlreadyFaulted members, which simply give Tasks that are already in a final state. This isn't strictly necessary, but come in handy in a number of situations:

internal static Task AlreadyDone;

static TaskControlFlow()

{

 var tcs = new TaskCompletionSource<bool>();

 tcs.SetResult(true);

 AlreadyDone = tcs.Task;

}

private static Task AlreadyFaulted(Exception ex)

{

 var tcs = new TaskCompletionSource<bool>();

 tcs.SetException(ex);

 return tcs.Task;

}

And that's it. I'm done for now. Hope you enjoyed it. I've got a few other posts in the works -- primarily the result of a day full of hacking (I got in the office at 7am this morning, and have been here ever since, 14 hours later) -- demonstrating how to do speculative asynchronous work for if/else branches. Finally, I also have a neat example that illustrates how to do deep dataflow-based speculation without having to wait for work to complete. This combines the new .NET 4.0 dynamic capabilities with parallelism, so I'm pretty excited to get it working and write about it.

Jobs

Sat, 31 Oct 2009 21:02:15 GMT

My team is hiring. For example:

https://careers.microsoft.com/JobDetails.aspx?jid=7594&lang=en

As the job description says, we are focused mainly concurrency & parallelism at each layer of the system: kernel-mode, user-mode, frameworks, languages, compilers, ...

The sky is the limit and nothing is off the table. I've never had so much fun in my life as I am having right now, and couldn't imagine a better g r o u p of p e o p l e to do it with. I think this is what NT must have felt like.

If you're interested, email me at joedu AT microsoft DOT com, or apply via that link.

False sharing is no fun

Tue, 20 Oct 2009 00:59:20 GMT

Embarrassingly, I neglected to write about the oldest trick in the book in my last post: designing the producer/consumer data structure to reduce false sharing. As I've written about several times previously (e.g. see here), and more so in the book, false sharing is always deadly and must be avoided.

As a simple example, consider a program that merely increments a shared counter over and over again. If we give P threads their own separate counters, and ask them to increment the respective counter an equal number of times. Each thread can of course do this without synchronization, because the counters are distinct: no locks or even interlocked operations are necessary. Naively, one might expect that running P of them in parallel leads to no interference, and hence perfect parallelization. However, when I run a little benchmark on my 8-way machine, the numbers for increasing values of P tell a very different story:

1 = 22425789

2 = 42023726 (187%)

4 = 175828522 (784%)

8 = 333906288 (1489%)

It is clear that the throughput drops dramatically as P increases. The reason? Each counter, being only 8 bytes wide, shares a cache line with as many as 7 other counters -- or 15 if we're on a machine with 128 byte cache lines. A simple change to the counter's layout, so that individual counters do not share the same cache line, will remedy the situation. The numbers improve dramatically. In fact, they remain constant no matter the value of P:

1 = 21914250

2 = 21900392 (100%)

4 = 21865781 (100%)

8 = 21934008 (100%)

This perfect scaling isn't always possible due to memory bandwidth, but because we're just incrementing a single counter per core this doesn't manifest as a problem.

For what it's worth, the machine I am running these tests on is an 8-way, dual-socket, quad-core. Pairs of cores share an L1 cache, and all cores in a socket share an L2 cache. So the pairs {0,1}, {2,3}, {4,5}, and {6,7} are each expected to have distinct L1 caches and the groups {0,1,2,3} and {4,5,6,7} are expect to have distinct L2 caches. The 2 number above is run with two threads affinitized such that they share the same L1 cache. If we force them apart, however, we get slightly different results:

2 = 42023726 (187%) -- same L1 cache

2 = 54706505 (244%) -- same L2 cache

2 = 75030977 (335%) -- separate sockets

As expected, the more distance in the cache hierarchy, the greater the slowdown due to the increased ping pong paths.

The specific results are of course unique to my machine, but nevertheless the conclusion is clear: reducing sharing leads to substantial performance gains, particularly with large numbers of threads hammering on the shared lines. Often more so than eliminating other sources of wasted cycles, like interlocked operations. Eliminating those sources is clearly important too, but it really is amazing how deadly and yet difficult to discover false sharing can be: few cases are as obvious as this one.

One aside is worth mentioning before winding down. When I first ran this experiment, I had done it two ways: (1) with fields of a shared object, then using StructLayout(LayoutKind=Explicit) to keep fields apart, and (2) with counters crammed into an array, which then contains padding elements to eliminate the false sharing. The former is shown above. If you try the latter, you may be surprised. The layout of arrays on the CLR is such that an array's length resides before the first element. So unless you pad the first element of the array, all accesses will perform bounds checking that touches the first element's line. Because this line is being mutated by the thread incrementing the first counter, terrible false sharing results. To solve this, we must pad the first element too.

For example, here are the array numbers with false sharing:

1 = 27366202

2 = 125264714 (458%)

4 = 1383953372 (7969%)

8 = 3136996731 (11463%)

Notice the P = 8 case is over 100x slower! Yowzas. After fixing things, with the first element padded, we again observe perfect scaling:

1 = 27393869

2 = 27465999 (100%)

4 = 27370901 (100%)

8 = 27408631 (100%)

Clearly false sharing is not merely a theoretical concern. In fact, during our Beta1 performance milestone in Parallel Extensions, most of our performance problems came down to stamping out false sharing in numerous places: the partitioning logic of parallel for loops, polling cancellation token flags, enumerators allocated at the beginning of a PLINQ query and constantly mutated during its execution, and even in our examples (e.g. see Herb's matrix multiplication example), etc. It is terribly simple to make a mistake and, in a complicated system, terribly difficult to pinpoint the origin of what can be a truly crippling scalability bottleneck.

In the next post, we will go back and take a look at our single-producer / single-consumer buffer, and redesign it to have substantially better cache behavior.

For reference, here's the basic program used for a lot of these tests:

//#define CACHE_FRIENDLY

//#define USE_ARRAY

#pragma warning disable 0169

using System;

using System.Diagnostics;

using System.Runtime.InteropServices;

using System.Threading;

class Program

{

 const int P = 1;

#if USE_ARRAY

 class Counters

 {

 long[] m_longs;

 internal Counters(int n) {

#if CACHE_FRIENDLY

 m_longs = new long[(n+1)*16];

#else

 m_longs = new long[n];

#endif

 }

 public void Increment(int i) {

#if CACHE_FRIENDLY

 m_longs[(i+1)*16]++;

#else

 m_longs[i]++;

#endif

 }

 }

#else // USE_ARRAY

#if CACHE_FRIENDLY

 [StructLayout(LayoutKind.Explicit)]

#endif

 struct Counters

 {

#if CACHE_FRIENDLY

 [FieldOffset(0)]

#endif

 public long a;

#if CACHE_FRIENDLY

 [FieldOffset(128)]

#endif

 public long b;

#if CACHE_FRIENDLY

 [FieldOffset(256)]

#endif

 public long c;

#if CACHE_FRIENDLY

 [FieldOffset(384)]

#endif

 public long d;

#if CACHE_FRIENDLY

 [FieldOffset(512)]

#endif

 public long e;

#if CACHE_FRIENDLY

 [FieldOffset(640)]

#endif

 public long f;

#if CACHE_FRIENDLY

 [FieldOffset(768)]

#endif

 public long g;

#if CACHE_FRIENDLY

 [FieldOffset(896)]

#endif

 public long h;

 }

 static Counters s_c = new Counters();

#endif // USE_ARRAY

 public static void Main(string[] args)

 {

 int p = int.Parse(args[0]);

 const int iterations = int.MaxValue / 4;

 ManualResetEvent mre = new ManualResetEvent(false);

#if USE_ARRAY

 Counters c = new Counters(p);

#endif

 Thread[] ts = new Thread[p];

 for (int i = 0; i < ts.Length; i++) {

 int tid = i;

 ts[i] = new Thread(delegate() {

 SetThreadAffinityMask(GetCurrentThread(), new UIntPtr(1u << tid));

 mre.WaitOne();

 for (int j = 0; j < iterations; j++)

#if USE_ARRAY

 c.Increment(tid);

#else

 switch (tid) {

 case 0: s_c.a++; break;

 case 1: s_c.b++; break;

 case 2: s_c.c++; break;

 case 3: s_c.d++; break;

 case 4: s_c.e++; break;

 case 5: s_c.f++; break;

 case 6: s_c.g++; break;

 case 7: s_c.h++; break;

 }

#endif

 });

 ts[i].Start();

 }

 Stopwatch sw = Stopwatch.StartNew();

 mre.Set();

 foreach (Thread t in ts) t.Join();

 Console.WriteLine(sw.ElapsedTicks);

 }

 [System.Runtime.InteropServices.DllImport("kernel32.dll")]

 static extern IntPtr GetCurrentThread();

 [System.Runtime.InteropServices.DllImport("kernel32.dll")]

 static extern UIntPtr SetThreadAffinityMask(IntPtr hThread, UIntPtr dwThreadAffinityMask);

}

Fast synchronization between a single producer and single consumer

Mon, 05 Oct 2009 01:03:17 GMT

Commonly two threads must communicate with one another, typically to exchange some piece of information. This arises in low-level shared memory synchronization as in PLINQ’s asynchronous data merging, in the implementation of higher level patterns like message passing, inter-process communication, and in countless other situations. If only two agents partake in this arrangement, however, it is possible to implement a highly efficient exchange protocol. Although the situation is rather special, exploiting this opportunity can lead to some interesting performance benefits.

The standard technique for shared-memory situations is to use a ring buffer. This buffer is ordinarily an array of fixed length that may become full or empty. The two threads in this arrangement assume the role of producer and consumer: the producer adds data to the buffer and may make it full, whereas the consumer removes data from the buffer and may make it empty. It is possible to generalize this to multi-consumers or multi-producers, with some added cost to synchronization. What is described below is for the two thread case.

We will call this a ProducerConsumerRendezvousBuffer<T>, and its basic structure looks like this:

using System;

using System.Threading;

public class ProducerConsumerRendezvousPoint<T>

{

private T[] m_buffer;

private volatile int m_consumerIndex;

private volatile int m_consumerWaiting;

private AutoResetEvent m_consumerEvent;

private volatile int m_producerIndex;

private volatile int m_producerWaiting;

private AutoResetEvent m_producerEvent;

public ProducerConsumerRendezvousPoint(int capacity)

{

if (capacity < 2) throw new ArgumentOutOfRangeException("capacity");

m_buffer = new T[capacity];

m_consumerEvent = new AutoResetEvent(false);

m_producerEvent = new AutoResetEvent(false);

}

private int Capacity

{

get { return m_buffer.Length; }

}

private bool IsEmpty

{

get { return (m_consumerIndex == m_producerIndex); }

}

private bool IsFull

{

get { return (((m_producerIndex + 1) % Capacity) == m_consumerIndex); }

}

public void Enqueue(T value)

{

...

}

public T Dequeue()

{

...

}

There are some basic invariants to call out:

Our buffer holds our elements, producer index says at what position the next element enqueued will be stored, and the consumer index says from what position the next request to dequeue an element will retrieve its value.
We reserve one element in our buffer to differentiate between fullness and emptiness. This is why we demand that capacity be >= 2. We could alternatively know how to distinguish between a free slot and a used one, such as checking for null, but keep things simple for now.
Thus, IsEmpty is true when the consumer and producer index are the same. Whereas IsFull is true when the consumer is one ahead of the producer, such that producing would make the producer index collide with the consumer index (otherwise leading to IsEmpty).
It should be obvious that our intent is to block consumption when IsEmpty == true and production when IsFull == true. This is the point of the waiting flags and events.

Now let us look at the implementation first of Enqueue and then Dequeue, paying special attention to the necessary synchronization operations. They look nearly identical:

public void Enqueue(T value)

{

if (IsFull) {

WaitUntilNonFull();

}

m_buffer[m_producerIndex] = value;

Interlocked.Exchange(ref m_producerIndex, (m_producerIndex + 1) % Capacity);

if (m_consumerWaiting == 1) {

m_consumerEvent.Set();

}

Enqueue begins, as expected, by checking whether the queue is full. Notice that we have not yet issued any memory fences yet. The only thread that may make the buffer full is the current one, which will obviously not occur before proceeding, and therefore we needn’t perform any expensive synchronization operation for this check. The value seen may of course be stale but we can deal with that possibility inside the slow path, WaitUntilNonFull. We’ll look at that momentarily.

We then proceed to placing the value in the buffer at the current producer’s index. Only the current thread will update the producer index and a consumer will not read from the current value so long as the producer index refers to it. The value may not even be written atomically, e.g. for T’s that are greater than a pointer sized word. This is okay: only the act of incrementing the index allows a consumer to access the element in question. Writes on the CLR 2.0 memory model are retired in order and the reading side will use an acquire load of the index before accessing the element’s words. Indeed we could use complicated multipart value types that are comprised of lengthy buffers, header words, and so on.

We then increment the producer index, handling the possibility of wrap-around by modding with the capacity. This uses an Interlocked.Exchange for one simple reason: we are about to read a consumer waiting flag, and must prevent the load of that flag from moving prior to the producer index write. The consumer sets this flag when it notices the queue is empty and waits. This enables us to use a “Dekker style” check to minimize synchronization. We could have alternatively just unconditionally set the event, doing away with the interlocked operation altogether. But that call, if it involves kernel transitions, which is quite likely, is going to be much more expensive and would occur on every call to Enqueue. And any event of this kind that doesn’t require kernel transitions is going to at least require an interlocked operation for the same reason we require one here. An alternative technique involves setting when we transition the buffer from empty to non-empty or full to non-full, but this wastes a possibly expensive signal if the other party isn't even currently waiting. If full or empty is a rare situation, then full or empty and with a peer actually physically waiting is even rarer.

Let’s now look at the WaitUntilNonFull method. It’s really the reverse of what the consumer does, so based on everything said till this point, I am guessing it’s obvious:

private void WaitUntilNonFull()

{

Interlocked.Exchange(ref m_producerWaiting, 1);

try {

while (IsFull) {

m_producerEvent.WaitOne();

}

finally {

m_producerWaiting = 0;

}

We begin by issuing a memory fence and setting the producer waiting flag. This memory fence is necessary to advertise that we are about to wait, and also to ensure the subsequent check of IsFull is synchronized. The consumer does something very much like the producer does (above) after taking an element: if the producer is waiting, the consumer has made space for it and therefore must signal. But it could be the case that the consumer has already made the queue non-full before it could notice the producer’s waiting flag. We catch this by ensuring the producer’s check of IsFull cannot go before setting the producer waiting; similarly, the consumer cannot make IsFull false without subsequently noticing that the producer is waiting; this avoids deadlock.

Everything else is self explanatory. Well, almost. We need a loop here to catch one subtle situation. Imagine a producer enters into this method thinking the buffer is full. It sets its flag, and then immediately notices that the buffer is not full anymore. A consumer has generated a new item of interest. But imagine that consumer noticed that the producer was waiting and hence set its event. This is an auto-reset event, so the next time the producer must wait, the event will have already been set and it’ll likely wake up before IsFull has become true. An alternative way of dealing with this is to call Reset on the event if we didn’t actually wait on the event, but again we keep things simple.

At this point, the consumer side is going to look very familiar:

public T Dequeue()

{

if (IsEmpty) {

WaitUntilNonEmpty();

}

T value = m_buffer[m_consumerIndex];

m_buffer[m_consumerIndex] = default(T);

Interlocked.Exchange(ref m_consumerIndex, (m_consumerIndex + 1) % Capacity);

if (m_producerWaiting == 1) {

m_producerEvent.Set();

}

return value;

}

private void WaitUntilNonEmpty() {

Interlocked.Exchange(ref m_consumerWaiting, 1);

try {

while (IsEmpty) {

m_consumerEvent.WaitOne();

}

finally {

m_consumerWaiting = 0;

}

This is near-identical to Enqueue and WaitUntilNonFull, and so needs little explanation. The acquire load inside IsEmpty of the producer index ensures that we observe the producer index for this particular value being beyond the current consumer’s index before loading the value itself, thereby ensuring we see the whole set of written words. The one other thing to point out is that we “null out” the element consumed which, for large buffers, helps to avoid space leaks that would have otherwise been possible.

There are certainly some opportunities for improving this.

For example, we might add a little bit of spinning in the wait cases. This would be worthwhile for cases that exchange data at very fast rates and have small buffers, meaning that the chance of hitting empty and full conditions is quite high. Avoiding the context switch thrashing is likely to lead to hotter caches, because threads will remain runnable for longer, and the raw costs of switching themselves.

Additionally, we technically could use a single event if we wanted to spend the effort. We’d have to handle a few tricky cases, however: namely, the case where a producer or consumer ends up waiting on an event because it “just missed” the event of interest, thus satisfying the event. Indeed both threads could actually end up waiting on the event simultaneously and we need to somehow ensure the right one eventually gets awakened. This leads to some chatter and probably isn’t worth the added complexity.

Here is a peek at some rough numbers from a little benchmark that has two threads enqueuing and dequeuing elements as fast as humanly (or computerly) possible. This is a particularly unique and unlikely situation, but stresses the implementation in a few interesting ways. In particular, it will stress the contentious slow paths; although these are expected to be rarer, the fast paths are just so easy to get right in this data structure that they are mostly uninteresting to stress performance-wise. There are then a few variants, each based on the original version shown above:

2 element capacity, which means we’ll be transitioning from empty to full and back a lot.
1024 element capacity, which means we won’t.
With spinning, using .NET 4.0’s new System.Threading.SpinWait type.
An implementation that overuses interlocked operations as many naïve programmers would do.

The 2 element capacity situation is common in some message passing systems, e.g. Ada rendezvous, Comega joins, and the like. Whereas the 1,024 element capacity situation is common for more general purpose channels, where some amount of pipelining is anticipated.

I whipped together a benchmark -- so quickly that we can barely trust it, I might add -- to measure these things. Here’s a small table, showing the observed relative costs:

2 capacity 1024 capacity

As-is No-spin 100.00% 1.93%

Spin 56.41% 1.66%

Naïve No-spin 101.20% 2.09%

Spin 67.73% 1.87%

As with most microbenchmarks, take the results with a grain of salt. And there are certainly more interesting variants to compare this against, including a monitor-based implementation that locks around access to the buffer itself. Nevertheless, we can draw a few conclusions from this: as expected, the version that uses a single interlocked on enqueue and single interlocked on dequeue is faster than the naïve version that uses multiple (surprise!); spinning makes a much more interesting difference on the 2 element capacity situation, as expected, because it reduces the number of context switches dramatically; and, finally, the larger capacity enables a producer to race ahead of the consumer, hence avoiding far fewer transitions from full to empty to full and so forth.

This post was more of a case study than anything else. There is nothing conclusive or groundbreaking here, and I suppose I should have said that would be the case up front. That said, I’ve seen this technique used in over a dozen situations in actual product code now, so I figured I’d write a little about it, with a focus on how to minimize the synchronization operations. We even contemplated shipping such a type in Parallel Extensions to .NET, but it’s just too darn specialized to warrant it. So the closest thing we provided is BlockingCollection<T>. Enjoy.

2nd edition of Concurrent Programming on Windows

Tue, 29 Sep 2009 00:47:03 GMT

I've officially started down the long road of writing a 2nd edition of Concurrent Programming on Windows, and would like your help.

There are many great new features in Windows 7 and the next versions of .NET, Visual C++ / CRT, and Visual Studio. The book will of course cover them all.

But I am also looking to reshape the 1st edition in many dimensions. I'd like to focus on readability, conciseness, and clearly separating the "must know" topics from the more geeky and advanced ones. This is a common conundrum when writing a technical book. The advanced topics are more likely to appeal to readers of my blog, for instance, but may be daunting for newcomers to concurrency. Tradeoffs abound. Nevertheless the 2nd edition is likely to be slimmed down compared to the 1st.

Any and all feedback, suggestions, and ideas are welcome. What did you like about the 1st edition, and what did you not like? If you could change a handful of things, what would make the top of your list? And was it missing something crucial that you would like to see covered? Please send your feedback to joe AT@ acm DOT org, or simply leave comments here on the blog. Regardless of whether you've read the 1st edition or not.

I sincerely look forward to hearing from you. Cheers.

On effects and ubiquitous parallelism

Mon, 27 Jul 2009 22:57:18 GMT

I had originally entitled this post "Having your concurrency cake and eating it too", but it sounded a little too silly.

I have grown convinced over the past few years that taming side effects in our programming languages is a necessary prerequisite to attaining ubiquitous parallelism nirvana. Although I am continuously exploring the ways in which we can accomplish this -- ranging from shared nothing isolation, to purely functional programming, and anything and everything in between -- what I wonder the most about is whether the development ecosystem at large is ready and willing for such a change.

It is this that I find the most frightening. I know we can give the world Haskell, or Erlang, or simple incremental steps within familiar environments, like Parallel Extensions. (Indeed, the world already has these things.) But elevating effects to a first class concern in day-to-day programming turns out to be a tough pill to swallow. Particularly since the incremental degrees of parallelism that this switch will unlock are questionable (see this and this); and even if they were pervasive and impressive, it's unclear what percentage of developers will pay what specific price for a 2x, 4x, or even 16x increase in compute performance. It sounds great on paper, but the cost / benefit equation is a complicated one.

"Pay for play" is the standard terminology we use for such things around here, and the solution needs to have the right amount of it.

Many folks with embarrassingly parallel algorithms will succeed just fine in a shared memory + locks + condition variables world, and indeed have already begun to do so. And specialized tools -- like GPGPU programming -- have popped up that, when small kernels of computations are written in a highly constrained way, will parallelize, sometimes impressively. Is this enough? Perhaps for the next 5 years, but surely not much longer after that. It is in my opinion qualitatively very important for the future of computer science that we provide programming environments that are more conducive to safe and automatic parallelism. And yet I cannot stand up with a straight face and proclaim that each and every developer on the face of the planet should practice side effect abstinence. A healthy balance between cognitive familiarity and pragmatic [r]evolution must be found. Many promising approaches are in the works (see UIUC's DPJ), but we are years away.

Until then, parallelism on broadly deployed commercial platforms will likely remain in the realm of specialists.

Of course, Haskell and Erlang both accomplish the no effects feat in a sneaky way. For those interested in foisting parallelism unto the masses, lessons can be learned from these communities. If you buy into purely functional programming, you necessarily buy into programming without effects, and the (sparing) use of monads to represent them. (Or, as my colleague Erik calls it, fundamentalist functional programming).) And if you buy into large scale message passing, you (typically) necessarily also buy into programming without shared memory, leaving behind only strongly isolated effects. The key here is that developers gain many other benefits by switching to these platforms -- and the lack of effects is admittedly a consequential byproduct of this switch. The lack of effects are not center stage. The two approaches have recently begun to converge in what I believe to be the appropriate long-term approach: strong isolation with effects within, and safe, deterministic data parallelism through careful control over sharing, aliasing, and heap separation.

That said, though not center stage, the switch to effectless programming is certainly not painless.

Enabling side effects among otherwise functional code, I think, is a good thing, because it allows familiar algorithms to be encoded in an ordinary imperative way. Familiarity is key: it may sound two faced, but I don't think parallelism is sufficiently top of mind that developers will want to completely rearrange the way that they write software. Perhaps we will evolve in this direction, but a significant leap will fall flat. Moreover, many algorithms actually depend on stateful updates to achieve adequate performance, like write in place graphics buffers. The Haskell state monad strikes a nice balance between embedding imperative-looking effects, when coupled with the do notation, within a strictly functional language.

Furthermore, I really respect that Haskell discourages cheating. (Any unsafePerformIO is viewed with great suspicion.) I quite like mostly-functional programming languages like ML and Scheme, because they tend to be easier on programmers with C backgrounds, but strongly dislike that a mutation can lurk within what appears to be an otherwise pure function. Documenting side effects in the type system is healthy and allows better symbolic reasoning about the dependencies and implicit parallelism contained within, transitively, while still providing a way to get at effectful programming. Haskell does a great job at this. The elimination of dependence ought to be the focus of programmers, and not the elimination of ad-hoc and unstructured access to shared, mutable state. These are algorithmic and important concerns.

What remains unclear is where the boundaries lie. Part and parcel of documenting effects is thinking about them when designing your software. You need to consider whether IList<T>'s Contains method may mutate the list or not, for example, and hold the line on implementations of the interface. Either it returns an 'a' or an 'IO a' -- and this decision is one that has far reaching implications. This is a wholly separate kind of interface contract than what most programmers are accustomed to having to think about during the code-debug-edit cycle. And surely Python and JavaScript developers will not care one way or the other, particularly if it forces more design decisions up front than what is customary today. This bifurcation seems inevitable, and yet there is substantial crossover: C# developers will write Python scripts, and Python developers will consume components written in C#.

And yet, I think we need to venture down this path in order achieve automatically scalable software. Parallel computers have become incredibly cheap, and so the historical barriers into high performance technical computing have been whittled away to the software skills necessary to write scalable programs; we will likely succeed at expanding this market without radical changes, but if we stopped there, vast reams of client-side software will be left in the dust. I've been making inroads into solving the problem on my end, with a new language that sits between C# and Haskell. I'm biased, have been hard at work on this problem for many years, and yet still struggle to answer these fundamental questions. I am a big believer that there's got to be a happy medium out there. But I'm still very perplexed, and face some very high walls to hurdle. Who will discover the right balance, and when will they do so?