|
Personal Info:
Joe  leads the architecture of an experimental OS's developer platform, where
he is also chief architect of its programming language. His current mission is to enable
writing large-scale software that is reliable, secure, and scalable by-construction. Before this, Joe
founded the Parallel Extensions to .NET project.
He has been granted 19 patents, with 49 pending. When not working, Joe enjoys travelling with his wife,
writing books, writing music,
studying music theory & mathematics, and doing anything involving food & wine.
My books
My music
Disclaimer:
The content of this site are my own personal opinions and do
not represent my employer's view in anyway.
© 2012, Joe Duffy
|
|
 Wednesday, June 09, 2004
I've been toying around with C# iterators a bit more lately, particularly regarding non-mainstream applications. I discussed partial algorithm computation during a couple earlier posts (here and here), but now wish to turn to producer/consumer models. A producer/consumer model is one in which a single producer is responsible for generating items of interest, which are later processed by one or more consumers.
I really should do a bit more explanation, but for now I'll simply present some code examples with brief comments.
To wire up threaded producers and consumers, quite a bit of plumbing is required; thus, I have encapsulated it all into a couple common abstract classes:
public abstract class Producer<T>
{
public Producer()
{
worker = new Thread(new ThreadStart(this.ProductionCycle));
}
private Queue<T> buffer = new Queue<T>();
public Thread worker;
private bool done;
public bool Done
{
get
{
return done;
}
}
public IEnumerable<T> ConsumerChannel
{
get
{
if (done)
throw new InvalidOperationException("Production is not currently active");
while (!done)
{
Nullable<T> consumed = new Nullable<T>();
//BUG: compiler crashes when using lock(...) construct within iterator
Monitor.Enter(buffer);
if (buffer.Count == 0)
Monitor.Wait(buffer);
if (buffer.Count > 0)
consumed = new Nullable<T>(buffer.Dequeue());
Monitor.Exit(buffer);
if (consumed.HasValue)
yield return consumed.Value;
}
yield break;
}
}
public void BeginProduction()
{
done = false;
worker.Start();
}
public void EndProduction()
{
done = true;
lock (buffer)
{
Monitor.PulseAll(buffer);
}
}
private void ProductionCycle()
{
while (!done)
{
T t = ProduceNext();
lock (buffer)
{
buffer.Enqueue(t);
Monitor.Pulse(buffer);
}
}
}
protected abstract T ProduceNext();
}
public abstract class Consumer<T>
{
public Consumer(Producer<T> producer)
{
this.producer = producer;
worker = new Thread(new ThreadStart(this.ConsumerCycle));
}
private Producer<T> producer;
public Thread worker;
private bool done = false;
public bool Done
{
get
{
return done;
}
}
public void BeginConsumption()
{
done = false;
worker.Start();
}
public void EndConsumption()
{
done = true;
}
private void ConsumerCycle()
{
foreach (T t in producer.ConsumerChannel)
{
Consume(t);
if (done)
break;
}
}
protected abstract void Consume(T t);
}
(Note: I haven't spent enough time on the threading issues, and as such may have overlooked a thing or two. It's pretty messy as it stands, but it'll do for now...)
To create specific concrete classes from these, it's a matter of simply overriding a couple methods. For instance, the following producer generates a never-ending sequence of random numbers, while the consumer simply prints them out:
class RandomNumberProducer : Producer<int>
{
public RandomNumberProducer() : base()
{
rand = new Random();
}
private Random rand;
protected override int ProduceNext()
{
return rand.Next();
}
}
class RandomNumberConsumer : Consumer<int>
{
public RandomNumberConsumer(RandomNumberProducer p) : base(p)
{
}
private static int counter = 0;
private int id = ++counter;
protected override void Consume(int t)
{
Console.Out.WriteLine("#{0}: consumed {1}", id, t);
}
}
To wire them up, and kick off the cycles, the following test code does the trick:
RandomNumberProducer p = new RandomNumberProducer();
RandomNumberConsumer c1 = new RandomNumberConsumer(p);
RandomNumberConsumer c2 = new RandomNumberConsumer(p);
RandomNumberConsumer c3 = new RandomNumberConsumer(p);
p.BeginProduction();
c1.BeginConsumption();
c2.BeginConsumption();
c3.BeginConsumption();
Thread.Sleep(2500);
c3.EndConsumption();
c2.EndConsumption();
c1.EndConsumption();
p.EndProduction();
These examples probably don't quite illustrate the advantages of this approach very well. Obviously, using well factored base classes provides a lot of benefit, namely that they encapsulate and implement the threading and synchronization boilerplate. Interestingly, the iterators remove the need for the consumer to worry about any synchronization.
This means that the following simple consumer code is actually thread safe - even if there are other consumers wired up to the producer:
foreach (int i in p.ConsumerChannel)
{
Console.Out.WriteLine("consumed: {0}", i);
}
In theory, the threading and blocking is all handled by the producer's iterator (although I question if my implementation is entirely correct).
Pretty nifty, if you ask me...
[I apologize for the hideous HTML for the code samples - I really wanted to retain the Visual Studio color scheme, and the only easy way I could find to accomplish this was to cut and paste into Word. Probably not an issue on most browsers, but just in case...]
Tuesday, June 08, 2004
Two great books are at the top of my reading stack at the moment,
 |
Distributed Systems: Principles and Paradigms
by Andrew S. Tanenbaum, Maarten van Steen
Decent book thus far (I'm only a couple chapters in), although it feels dated. If I make it through, I'll post a detailed review about it. |
and
 |
ANSI Common Lisp
by Paul Graham
This is a re-read. Unfortunately I'm not able to work with Lisp as often as I'd like, and thus every once in a while I dust off the cobwebs. Each time I go back, I learn something new and am treated to a new perspective on everyday challenges. I want to get ahold and paw through a copy of Guy Steele's Common LISP: The Language, but as of yet have not had a chance. |
Upgraded to dasBlog 1.6... reworked the layout and color scheme of the site...
Just making sure it functions properly!
Friday, June 04, 2004
...comes in many forms.
(defun fib (n)
(if (< n 2)
1
(+ (fib (- n 1)) (fib (- n 2))))) |
|
 |
 |
Doug Purdy posted an amusing response (and here) to the whole "DataSets are evil" argument.
To be clear, my aversion to DataSets has nothing to do with their use in web services, but rather as an overall architectural principle. As he points out (and Dare reiterates), once it escapes your application we're simply talking about schema and XML data. (Note: it's interesting to question what "it" represents in the previous sentence; it sure as hell isn't an instance of a DataSet once it reaches the wire!) The programming model is an implementation detail, and is certainly hidden behind an interface.
So if application edges are rigid and abstracted by an interface which represents data as XML, who cares in what fashion such XML is constructed? Once could - if so inclined - represent objects internally in a picture perfect domain model, yet serialize them over the wire in a horrid alphabet soup of nonsensical tags; and, conversely a DataSet could manifest on the wire as a beautiful schema-driven thing of wonder.
So, again... who really cares?
Well, I do. My comments were made focused on proper separation of concerns within application boundaries (e.g. between tiers) and through the prism of proper domain modeling. So while advertising this as a web services-specific problem is a miscategorization, I still firmly believe it's a problem.
But then again, it's all about value delivered to the end consumer. My beliefs and experience are simply that careful domain modeling more often than not results in better value and agility in the end product. And thus, I stick to my story.
Thursday, June 03, 2004
"Human beings have been reduced to consumers."
"Americans are just now waking up to the real world [of Globalization]."
"Globalization has to be a two way traffic."
The Discovery channel is currently airing a special entitled "Thomas L. Fieldman Reporting: The Other Side of Outsourcing." Its content is mainly concerned with the impacts of outsourcing on India's traditionally family- and ritual-centric culture. I still need to digest some of the information, but it's definitely an eye-opening viewpoint.
I usually don't post simply for “Mindless Link Propagation” - as Dare calls it - but this post is of particular importance.
Josh Ledgard, Chief Community Evangelist at Microsoft, talks about the legal implications and challenges (via Scoble) that arise when early community involvement is solicited in the product lifecycle. It sucks big time when such legal barriers have the potential to force one to sidestep in the wrong direction... but consider the consequences. Yikes.
(BTW, I find Scoble's response pretty ignorant and childish, and akin to a teenage fit of rebellion. Dude, there's a reason legal departments restrict certain things... because there are implications should these safeguards not be put into place. Microsoft owns its IP, and as such can tell anybody - yes, even Scoble - what they are or are not able to do with it. Regardless of on whose time it occurs.)
Wednesday, June 02, 2004
There has been a good deal of buzz over the last couple days about DataSets and their use for representing data across boundaries.
See here, here, and here.
I particularly like Scott's analogy that domain object is to DataSet what Apple is to a Bowl of Apple Descriptions.
I have what may be considered an old school, minimalist view on this matter. In fact, it is very similar to the movement in the Java community towards POJOs and away from heavyweight containers and intrusive abstractions, both of which comprimise one's domain modeling ability and freedom. DataSets, DataReaders, DataWhatever instances should seldom cross tier boundaries, especially never across application boundaries, and certainly should never allowed to leak into the business tier. (There are some exceptions to this guideline, for example within data-oriented applications where a domain model either does not exist or whose sole purpose is to facilitate interactions with the data tier (e.g. ad-hoc reporting applications).) These are simply mechanisms to represent the results of data store operation, belong in the data tier and only in the data tier, and should never see the light of day outside their little tikki hut.
I am a big fan of managing interactions between the domain and data tier through Set-like patterns. A Set is modifiable, maintains identity, allows finding items which match a particular criteria, and is an easy abstraction to work with. The Set translates business-sensical operations into concrete commands to the data tier, and is then responsible for constructing domain objects and mapping data resulting from database operations. Life is simple, and the business tier and its consumers are entirely ignorant of data concepts which are of no interest. Unfortunately, the Set I describe differs fundamentally from the DataSet, namely in that the DataSet attempts to represent the domain model in addition to the data interface. While the DataSet's query and database interface mechanisms may be useful for integrating with a data store, this is an implementation detail and should be hidden as such. If the DataSet query syntax is appropriate for the business tier, for example, simply supply a query to our Set which will relay it to an internal DataSet in the data tier to perform the operation. The mapping still takes place, and the result is either a single object graph or a collection of object graphs, each of which is a strongly typed domain entity.
I understand that today in ASP.NET and WinForms DataSets make binding situations much easier, but the decision to go in one direction versus the other should be a conscious tradeoff. With the new ObjectDataSource functionality in 2.0, proper domain modeling and easy data binding are no longer exclusive choices.
ABC DEF
Anything significant about this sequence?
Yup. The seating pattern on the 737 I am currently flying on. In seat F.
Just a moment ago, man in seat D says to woman in seat E, "Huh, huh... if it smells in a minute, it's because I farted."
Loudly. He had headphones on, and some people's ears don't take to altitudes of 35,000 feet very well, so he may not have realized it. But that's no excuse. I heard it loud and clear. But even worse...
Moments later, man in seat D's ass particles entered my nostrils. Splendid.
Krzysztof reminded me yesterday that
An enumerator remains valid as long as the collection remains unchanged. If changes are made to the collection, such as adding, modifying or deleting elements, the enumerator is irrecoverably invalidated and the next call to MoveNext or Reset throws an InvalidOperationException.
(For some reason, I can't locate the ECMA BCL standard to find what it says about this situation, and unfortunately don't have my SLAR book with me. The previous excerpt was taken from the MSDN documentation of the IEnumerator interface.)
Given my earlier use of iterators to incrementally execute and return results from an algorithm, I wonder what impact this requirement has? Does it have an impact? I guess my confusion comes from not understanding if this is simply a documented optional behavior, or if it is a semantic constraint by the ECMA specification. The C# compiler is responsible for baking up such IEnumerator implementations from a given iterator, which has the indirect effect of requiring that people implementing iterators toss an InvalidOperationException from their iterator methods under certain circumstances. I'm not sure if this is very evident.
If the sequence to be calculated changes sometime after the IEnumerator creation, does this trigger an exception? How would we possibly recognize this situation? If the algorithm relies on input variables, all of which are locked down at the construction of our IEnumerable, this may be good enough; but, what if the nature of the algorithm is such that it relies on realtime input or needs to access input which cannot be copied or locked at creation? For example, one could imagine an iterator which uses sensor data that is read as quickly as the program can consume; a foreach loop is used to consume the data, while an iterator provides a reading when it is asked for the next value.
Unfortunately, I guess I have more answers than questions right now. Comments, answers, etc. would be great!
|
|
Recent Entries:
Search:
Browse by Date:
Browse by Category:
Notables:
|