RSS 2.0

Personal Info:

Joe Send mail to the author(s) leads the architecture of an experimental OS's developer platform, where he is also chief architect of its programming language. His current mission is to enable writing large-scale software that is reliable, secure, and scalable by-construction. Before this, Joe founded the Parallel Extensions to .NET project. He has been granted 19 patents, with 49 pending. When not working, Joe enjoys travelling with his wife, writing books, writing music, studying music theory & mathematics, and doing anything involving food & wine.

My books

My music

Disclaimer:
The content of this site are my own personal opinions and do not represent my employer's view in anyway.

© 2012, Joe Duffy

 
 Saturday, March 25, 2006

The profiler that ships with Visual Studio is great for "real" CPU profiling. But let's face it: there are still some situations where a good ole' stopwatch works just fine (of the System.Diagnostics.Stopwatch variety). For example, when you're trying to do some quick and dirty measurement on a very specific region of code, and don't want to deal with the rest of the noise.

The BCL stopwatch isn't inherently thread-safe. Even if you protect access to it (and somehow account for the overhead of doing so), it maintains a single counter. In the past, I've wanted to measure the total amount of time spent inside a select few regions of code, across all threads. The profiler works for this, but you need to get the sampling granularity right, and deal with all of the extra data collected. Then you have to mine it.

So I whipped up a stopwatch that maintains a counter that is the cumulative total of all threads that have started/stopped it, across an entire AppDomain. It's nothing incredibly clever, but I've found it to be quite useful. In many code projects I have, I've simply set up a file that declares a bunch of static ThreadSafeStopwatches, which I can then just call from anywhere.

Here's the code (also available for download here: ThreadSafeStopwatch.cs):

using System;

using System.Collections.Generic;

using System.Diagnostics;

using System.Threading;

 

/// <summary>

/// This class enables AppDomain-wide profiling of multi-threaded

/// code, by tracking a cumulative number of ticks spent across all

/// threads. If multiple threads start the same watch in parallel,

/// this class ensures that we count time for both threads, and that

/// they do not interfere with each other. It does this by storing an

/// internal stop-watch in TLS, and an instance-wide tick counter.

///

/// Note that the cumulative count is only ever incremented if a

/// thread actually calls Stop in its own stop-watch. If a thread

/// routine terminates w/out stopping the watch, it's as if it never

/// began.

/// </summary>

public sealed class ThreadSafeStopwatch : IDisposable

{

 

    /** Fields **/

    private long ticks;

 

    // These thread-specific fields are used to maintain a cache

    // of thread-safe stopwatches to actual stopwatches. For long

    // running threads, we can build up some amount of trash over

    // time, so a cache management scheme is implemented via a

    // combination of mechanisms: Dispose, ~ThreadSafeStopwatch,

    // GetWatch, and PruneCache.

    [ThreadStatic]

    private static Dictionary<WeakReference, Stopwatch> threadWatches;

    [ThreadStatic]

    private static int cacheCounter;

 

    /** Properties **/

    public long ElapsedTicks

    {

        get { return ticks; }

    }

    public float ElapsedMilliseconds

    {

        get { return (float)ticks / TimeSpan.TicksPerMillisecond; }

    }

 

    /** Methods **/

    ~ThreadSafeStopwatch()

    {

        Dispose(false);

    }

 

    public void Dispose()

    {

        Dispose(true);

    }

 

    private void Dispose(bool disposing)

    {

        // Clean up the associated cache entry with this object.

 

        // NOTE: I am explicitly not guarding this code-path by if

        // (disposing) { ... } because the Dictionary is not finaliz-

        // able. Thus, I know it is safe to access it. In the case of

        // non-process-exit finalizers, we do want to clean up the

        // associated cache entry, thus we access another object on

        // the finalizer code-path.

 

        if (!Environment.HasShutdownStarted)

        {

            Dictionary<WeakReference, Stopwatch> watches = threadWatches;

            WeakReference thisRef = new WeakReference(this);

            if (watches != null && watches.ContainsKey(thisRef))

            {

                // Deallocate the cache entry:

                watches.Remove(thisRef);

            }

        }

    }

 

    const int threadCachePruneCount = 100;

    private Stopwatch GetWatch()

    {

        if (threadWatches == null)

        {

            // First time called on this thread, allocate a new dictionary:

            threadWatches = new Dictionary<WeakReference, Stopwatch>(

                WeakRefEqualityComparer.Comparer);

        }

        else

        {

            // This has been called before. Increment the thread counter;

            // every so often, we prune out trash being held alive by our

            // cache.

            if (++cacheCounter % threadCachePruneCount == 0)

            {

                PruneCache();

                cacheCounter = 0;

            }

        }

 

        // Now look for the associated stopwatch:

        Stopwatch sw;

        if (threadWatches.TryGetValue(new WeakReference(this), out sw))

            return sw;

 

        // If we didn't find the stopwatch, simply return null to

        // indicate that the caller needs to allocate a new one:

        return null;

    }

 

    private void PruneCache()

    {

        // BUGBUG: This is probably a poor cache management policy. But

        // I'm only enabling this in DEBUG builds for now, and until I

        // run into a real problem, I'm not spending time on it.

 

        List<WeakReference> toRemoveWrefs = null;

 

        // Look for dead references, and add them to the list (which is

        // lazily created, by the way).

        foreach (WeakReference wr in threadWatches.Keys)

        {

            // If the weak-reference is no longer alive, we add it to the

            // list of 'to-remove' stop-watches.

            if (!wr.IsAlive)

            {

                if (toRemoveWrefs == null)

                    toRemoveWrefs = new List<WeakReference>();

                toRemoveWrefs.Add(wr);

            }

        }

 

        // If we found any dead entries, remove them now.

        if (toRemoveWrefs != null)

        {

            foreach (WeakReference wr in toRemoveWrefs)

            {

                threadWatches.Remove(wr);

            }

        }

    }

 

    [Conditional("DEBUG")]

    public void Start()

    {

        // We look in TLS to see if this thread has already allocated a

        // stopwatch for the current thread-safe stopwatch. This is

        // thread-safe, of course, since each thread gets their own list

        // of Stopwatches (reentrancy aside--there aren't any blocking

        // points below):

 

        // Since we are about to retrieve something from TLS, and use it

        // across a set of paired operations (Start/Stop), we mark the

        // beginning of a thread-affinity region.

        Thread.BeginThreadAffinity();

 

        // Access TLS:

        Stopwatch sw = GetWatch();

 

        if (sw == null)

        {

            // No watch was found, allocate a new one and publish it.

            sw = new Stopwatch();

            threadWatches.Add(new WeakReference(this), sw);

        }

 

        // First, ensure we haven't begun it yet. If the stopwatch is

        // already running, we ignore this call. This is consistent with

        the System.Diagnostics.Stopwatch

        // class's behavior.

        if (!sw.IsRunning)

        {

            // And if that check succeeds, start the stop-watch ticking.

            sw.Start();

        }

    }

 

    [Conditional("DEBUG")]

    public void Reset()

    {

        // Get the current stopwatch in TLS -- see above comments (in

        // Start) for details on thread-safety.

        Stopwatch sw = GetWatch();

 

        // If we found one, reset it.

        if (sw != null)

            sw.Reset();

 

        // And also set our cumulative ticks to 0.

        ticks = 0;

    }

 

    [Conditional("DEBUG")]

    public void Stop()

    {

        // Get the current stopwatch in TLS -- see above comments (in

        // Start) for details on thread-safety.

        Stopwatch sw = GetWatch();

 

        // First, ensure we are running. If the stopwatch isn't running

        // yet, we ignore this call. This is consistent with the System.

        // Diagnostics.Stopwatch class's behavior.

        if (sw != null && sw.IsRunning)

        {

            // Add the stopwatch's total time to our instance counter.

            // This has to be an interlocked operation, because the whole

            // point of this class is to be shared across threads. 'ticks'

            // is the only instance state.

            Interlocked.Add(ref ticks, sw.ElapsedTicks);

 

            // We reset the stopwatch because we want to start at 0 upon

            // the next invocation to 'Start' -- the cumulative time is

            // kept in the 'ticks' variable.

            sw.Reset();

 

            // We can now end the thread-affinity that was started in the

            // Start operation above.

            Thread.EndThreadAffinity();

        }

    }

 

    class WeakRefEqualityComparer : IEqualityComparer<WeakReference>

    {

        internal static WeakRefEqualityComparer Comparer =

            new WeakRefEqualityComparer();

 

        public bool Equals(WeakReference wr1, WeakReference wr2)

        {

            // For purposes of our hash-table, if two weak-references

            // refer to the same object, we consider them equal.

            object o1 = wr1.Target;

            object o2 = wr2.Target;

 

            if (!wr1.IsAlive || !wr2.IsAlive)

                return false;

 

            // We shouldn't ever have null weak-references that aren't

            // dead.

            Debug.Assert(o1 != null && o2 != null);

 

            // If the two underlying objects are equal, we pretend the

            // weak-refs are too.

            return o1.Equals(o2);

        }

 

        public int GetHashCode(WeakReference wr)

        {

            object o = wr.Target;

 

            // Just return 0 for dead objects. We actually shouldn't

            // ever use a dead object for hashing, although there could

            // be some benign races above that result in this case. I

            // haven't convinced myself otherwise, and they will get clean-

            // ed up with the normal finalization code-path. It's A-OK.

            if (!wr.IsAlive)

                return 0;

 

            // Again, shouldn't get a live weak-ref that has a null

            // object ref.

            Debug.Assert(o != null);

 

            // Now, simply return the underlying object's hash-code.

            return o.GetHashCode();

        }

    }

}

3/25/2006 3:19:52 PM (Pacific Daylight Time, UTC-07:00)  #   

The Whidbey version of Rotor just went up for download on MSDN on Thursday. I downloaded it and built it this morning.

While some of the big rock features are getting press (e.g. generics, LCG, anonymous methods/closures, etc.), some of the smaller features are plenty cool, and can be grokked in their entirety in much less time. For example, do a 'grep -i nullable clr/src/*/*.*' if you want to see what went into implementing the Nullable DCR that Soma mentioned over here. And check out stuff like the WrapNonCompliantException function in vm/excep.cpp (and its callsites) to see how non-CLS exceptions get wrapped. And of course there's all that reliability work that went into Whidbey, leading to things like SafeHandles (vm/safehandle.cpp), and OOM and SO hardening.

Most of it's there for you to tinker with. Or to simply print out and enjoy, reading it as you sit beside the fireplace with a nice Bourdeaux. To each his (or her) own.

3/25/2006 3:00:51 PM (Pacific Daylight Time, UTC-07:00)  #   

 Tuesday, March 21, 2006

Ahh, I fondly remember the days of all-nighters.

I found this great interview, done in September '05, where Max (co-founder of PayPal) discusses the virtues of entrepreneurship and late-night hacking.

I miss those days. Oh, how I miss them.

3/21/2006 11:14:07 PM (Pacific Daylight Time, UTC-07:00)  #   

 Wednesday, March 15, 2006

Jim recently asked me to put together a few stand-alone articles, loosely excerpted from my upcoming book.

The first one just went online: One Platform to Rule Them All. This is based on a section from pretty early in the book. In fact, it's near the front of the first chapter.

I chose the title based on T-shirts the CLR team had printed up a few years back, which prominently displays this slogan on it along with some nifty graphics (e.g. a ring inscribed with code). As you can imagine, it's based on a famous book series, and more recently, movie series...

3/15/2006 11:59:07 PM (Pacific Daylight Time, UTC-07:00)  #   

 Wednesday, March 08, 2006

Here are a few recent happenings.

And... my new book is just about to come out (late March/early April). I'm happy to report that I recently accepted an offer to write another, Concurrent Programming on Windows, the goal for which is to be the definitive guide to concurrent programming on Windows, WinFX, and the .NET Framework. It'll show you how to write kick-butt software on these new shnazzy multi-core machines.

3/8/2006 4:16:15 PM (Pacific Standard Time, UTC-08:00)  #   

 Thursday, February 23, 2006

When you perform a wait on the CLR, we make sure it happens in an STA-friendly manner. This entails using msg-waits, such as MsgWaitForMultipleObjectsEx and/or CoWaitForMultipleHandles. Doing so ensures we pick up and dispatch incoming RPC work mid-stack, while the STA isn't necessarily sitting in a top-level message loop. In fact, an STA that doesn't pump temporarily can easily lead to temporary and permanent hangs (i.e. deadlocks), especially in common COM scenarios where reentrant calls across apartments are made (e.g. MTA->STA->MTA->STA). Even where deadlock isn't possible, failing to pump can have a ripple effect across your process, as components wait for other components to complete intensive work.

I was recently writing about this fact for an article, and realized I had leaped to an incorrect assumption. OLE creates special RPC windows for processing these apartment transitions. I knew that. But I had assumed that we blatently pump for messages for any windows. But in reality, we don't. We only pump the special "WIN95 RPC Wmsg", "OleMainThreadWndClass", and "OleObjectRpcWindow" RPC windows. Consider why.

If you were on a UI and you pumped your window's message queue, you could end up dispatching new events before old events had completed. If you dispatched a WM_CLOSE message on the same stack you were doing some other UI processing, you'd destroy the window before that other processing was done. Without the GUI message loop taking this into account somehow, you'd crash. There are other factors. Imagine you were processing a click event that required movement of several UI elements. If some bit of infrastructure--or perhaps your code--ended up pumping, UI invariants could be broken. (Lots of FX code pumps for RPC, by the way.) At best, this could lead to strange visual artifacts, and at worst a crash.

You'd also have to deal with the subtleties of reentrancy. I've written about this before. Imagine a UI event had waited on an auto-reset event, did some work, and then set the event. If another event--perhaps the same type--were dispatched while it was doing "some work," it might try to wait on this same event. This would be a deadlock. A pretty difficult one to track down too, especially if it only occurred if the user clicked on certain elements at precise timings to get the reentrancy to occur. This is already a problem with COM interop. But thankfully we don't burden Windows Forms and WPF programming with it too.

2/23/2006 12:37:08 PM (Pacific Standard Time, UTC-08:00)  #   

 Tuesday, February 21, 2006

I've been pretty busy lately orchestrating all the concurrency work the CLR is undertaking for the next few versions.

We have several long lead incubation projects underway to explore new programming models which--some of us at least--believe will make concurrent programming at least an order of magnitude easier. While I hope we ship them, the fate of those projects is currently unknown. We'll understand better over time what is feasible and in what timeframe.

But there's also a set of features and work items we are actively planning for in the mainline product. This includes looking at new abstractions (e.g. futures, barriers, better lock factoring, a richer variety of locks, lock leveling), fixing existing ones (e.g. nagging issues and obvious missing features (e.g. prioritization, isolation, diagnostics) with our thread-pool, improving RWL's performance), debugging and tool support, and general cleanup activities (e.g. getting our own locking practices in order).

If there's anything you'd like to see us consider in this category, please let me know. Either leave a comment here or send me an email at joedu at microsoft. I'll share it with the rest of the concurrency team, and we'll seriously consider it.

I also have a couple related articles to watch out for: one on deadlock avoidance and detection in next month's MSDN magainzine, and another on application responsiveness in WPF in DDJ. I'm also going to be on .NET Rocks next week discussing some of this.

Unfortunately, all of this has left very little time to blog. If you have ideas for topics, drop me a line.

2/21/2006 7:55:42 PM (Pacific Standard Time, UTC-08:00)  #   

 Tuesday, February 07, 2006

I was on a mail thread today, the topic for which was the meaning—and perhaps lack of comprehensiveness—of the statement: “This type is thread safe.” Similar statements are scattered throughout our product documentation, without any good central explanation of its meaning and any caveats.

It’s relatively difficult to make such a statement. The .NET Framework is generally written so that all static members are thread-safe, while instance members are not. There are some notable exceptions, mostly to do with immutable types (e.g. the primitives, System.DateTime, System.Type, etc.), but they are infrequent.

This brings me to the types of thread safety issues we’re generally concerned with.

The first problem is torn reads and writes. Operations that deal with data whose size is greater than the native machine-sized pointer (i.e. sizeof(void*)) are not atomic in the ISA. This applies to 64-bit operations on 32-bit platforms, 128-bit on 64-bit, etc. We happen to provide you two intrinsic 64-bit types—Int64 (C# long) and Double (C# double)—which makes this issue a tad tricky. So this code

static long x;
// …
x = 1111222233334444;

consists of two DWORD MOVs at the machine level, one to the most significant half and then the least significant half (in that order, at least in the Whidbey x86 JIT). Reads likewise involve two MOV instructions.

If you’re doing naked reads and writes with such values, instructions can be interleaved such that only one DWORD has been written. That means a thread racing with the above assignment can see x with a value of (x & 0xFFFFFFFF00000000), or 2524709548 in decimal. This is obviously surprising, as the two values (at first glance) don’t seem to be related. And this same principle applies to reads and writes of any value type instances whose size is greater than sizeof(void*).

This can be solved by protecting all access to the data under a lock or via an interlocked operation. Interlocked.Exchange will do the trick for writes, and Interlocked.Read for reads. Note that most platforms offer 128-bit interlocked instructions. Unfortunately, because of the platform-specificity, the Win32 APIs and our System.Threading APIs don’t broadly support them. Hopefully this changes over time. For the same reason you often need two void*-sized writes on 32-bit, you often need the same on 64-bit.

In summary, any type that exposes a writable 64-bit field, or which returns a 64-bit value which has been copied by a field that might be in motion, is not thread-safe. And any internal reads and writes need to be done under the protection of a lock or interlocked operation. A method that updates an internal field, for example, can race with a property that returns the current value.

The second problem is read/modify/write sets of instructions. If reads and writes can be multiple instructions long, it should be clear that by default a read/modify/write is at least three. For a 64-bit value on a 32-bit platform, it’s six. At any point in that invocation, interleaved execution can cause an update to go missing. The solution here, again, is to do this inside a lock. The Interlocked.CompareExchange function is great for this purpose, as it takes advantage of hardware-level read/modify/write instructions. Thankfully they are supported by all modern hardware ISAs.

The last problem is that of ensuring coarser-grained data structure invariants can never be seen in a broken state by concurrent execution. This is especially difficult since arbitrary managed programs don’t capture such invariants in the program itself. Aside from static state, most Framework types don’t even come close to attempting to provide this level of guarantee. Static caches and lazily initialized state, for example, are places where the Framework needs to account for concurrent access. Old-style collections with SyncRoots tried to provide similar protection, but the new generic collections don’t any longer, mostly because of the performance hit you take on sequential code-paths. But those cases are the exception, not the norm.

The immutable types mentioned above are nice in that, aside from initialization-time, they never break their internal invariants. Thus, aside from assignments of instances to shared variables, you needn’t worry about any special synchronization.

In summary, any type that breaks invariants must do so in such a way that these invariants can never be observed due to concurrent execution. This means all access to data needs to be serialized with respect to coarser grained operations updating state. Our Framework isn’t written in this way, so if you share it, you usually take responsibility for locking it.

My preference is for developers to assume that all types are unsafe, and to explicitly lock when accessing them concurrently. Regardless of the documentation’s claims. We simply do not check for these things across releases, and some code that works today might break tomorrow because we accidentally forgot to account for a torn read.

2/7/2006 10:33:13 PM (Pacific Standard Time, UTC-08:00)  #   

 Thursday, January 26, 2006

Vance Morrison's excellent MSDN article from a few months back talks about why double checked locking is guaranteed to work on the CLR v2.0, and why it is one of the few safe lock-free mechanisms on the runtime. He also sent an email to the develop.com mailing list a while back explaining why this pattern wasn’t guaranteed to work on the ECMA memory model. We did quite a bit of implementation work and testing to tame the crazy memory model of IA-64 on 2.0. (Note that none of this is in the ECMA specification, so if you’re worried about CLI compatibility, beware.)

These modifications not only enable the double checked locking pattern, but also prevent constructors from publishing the newly allocated object before their state has been initialized, as I mentioned in my PDC presentation on concurrency last year. We accomplish this by ensuring writes have 'release' semantics on IA-64, via the st.rel instruction. A single st.rel x guarantees that any other loads and stores leading up to its execution (in the physical instruction stream) must have appeared to have occurred to each logical processor at least by the time x's new value becomes visible to another logical processor. Loads can be given 'acquire' semantics (via the ld.acq instruction), meaning that any other loads and stores that occur after a ld.acq x cannot appear to have occurred prior to the load. The 2.0 memory model does not use ld.acq’s unless you are accessing volatile data (marked w/ the volatile modifier keyword or accessed via the Thread.VolatileRead API). This can lead to some subtle problems.

For example, a slight variant of the double checked lock will not work under our model:

class Singleton {
    private static object slock = new object();
    private static Singleton instance;
    private static bool initialized;
    private Singleton() {}
    public Instance {
        get {
            if (!initialized) {
                lock (slock) {
                    if (!initialized) {
                        instance = new Singleton();
                        initialized = true;
                    }
                }
            }
            return instance;
        }
    }
}

You might have decided to use this pattern to determine whether to initialize a value-type, since checking the variable for null isn’t possible. If you had some more complex set of state, perhaps you want to use a single Boolean rather than checking, say, 10 separate variables to see if they have each been initialized. Whatever your reasoning, as written the above code is prone to a subtle race condition.

The problem here is that both reads of initialized and instance do not have 'acquire' semantics. Thus, instance could appear to have been read before initialized, e.g. as follows:

Time Thread A Thread B
0   Reads instance as null
1 Reads initialized as false
2 Sets instance to ref to new obj
3 Sets initialized to true
4 Uses instance (initialized)
5   Reads initialized as true
6   Uses instance (null!)

Thread B ends up returning a null reference. If a caller tried to use it, they might encounter a spurious NullReferenceException, the cause of which is incredibly hard to debug. For example:

void f() {
    Singleton s = Singleton.Instance;
    s.DoSomething(); // Boom!
}

For this to have happened, Thread B would have had to read instance entirely out of order. It might have done so for any number of reasons. If it recently executed some code that pulled it into cache—either directly or due to locality—it isn’t required to invalidate the cache with non-acquire reads, even though it observed a new write with release semantics, because it's as if the load was moved before the load of initialized. Or superscalar execution might perform branch prediction and retrieve the value of instance, assuming that initialized will be false, pulling it into cache ahead of the read of initialized. Again, because it is a non-acquire read, this is a valid thing to do. If it reads initialized as true, its prediction was actually correct, and it just returns the null value that was pre-fetched. It might even be the case that a compiler along the way moved the read, which is also entirely legal with our memory model.

One possible solution for this is to employ a volatile-read on the first read of the initialized variable, prohibiting the read of instance from moving prior to the read of initialized. Control dependency prevents us from having to use a volatile-read for the reads of both variables.

class Singleton {
    private static object slock = new object();
    private static Singleton instance;
    private static int initialized;
    private Singleton() {}
    public Instance {
        get {
            if (Thread.VolatileRead(ref initialized) == 0) {
                lock (slock) {
                    if (initialized == 0) {
                        instance = new Singleton();
                        initialized = 1;
                    }
                }
            }
            return instance;
        }
    }
}

You could have instead inserted a call to Thread.MemoryBarrier instead, which is a two way memory-fence, in between if-block and the read of instance, but the cost of a barrier is generally higher than both a st.rel and ld.acq because it affects surrounding instructions and movement in both directions.

The take-away here is not that you must understand the specifics of how cache coherency, speculative execution, and our memory model interact. Rather, it should be that once you venture even slightly outside of the bounds of the few "blessed" lock-free practices mentioned in the article mentioned above, you are opening yourself up to the worst kind of race conditions. Using locks is a simple way to avoid this pain. And hopefully someday in the future, transactional memory will enable performant execution of code with lock elision techniques that lead to the performance of lock-free code, but without any of the mental illness that such techniques have been proven to cause.

1/26/2006 11:37:58 AM (Pacific Standard Time, UTC-08:00)  #   

 Saturday, January 21, 2006

I drink about 4x more tea than I do coffee. Actually, I wouldn't drink coffee at all if there were decent tea stores in the area with readily available to-go offerings.

I just ordered a bunch of great teas from Upton Tea Imports, including a new 2006 Darjeeling First Flush. I can't wait until they arrive:

  • TD56: Tindharia Estate FTGFOP1 First Flush (EX-1)
  • TS70: Temi Estate FTGFOP1 CL
  • TA93: Nahorhabi Estate FTGFOP1 SPL CL
  • ZO87: Ginseng Tie-Guan-Yin Oolong
  • ZM44: Osmanthus Oolong Se Chung
  • ZW84: Organic Fuding White Treasure
  • ZW90: Organic White Point Reserve
  • ZW99: China White Paklum Tips Reserve
  • TA98: Mothola Estate White Tea
  • TJ77: Spring Harvest Kabusencha

If you're not a tea drinker, or the closest thing to real tea you've had is a soppy Stash tea bag, I encourage you to try one of Upton's sampler sets.

1/21/2006 11:48:26 AM (Pacific Standard Time, UTC-08:00)  #   

 

Recent Entries:

Search:

Browse by Date:
<March 2006>
SunMonTueWedThuFriSat
2627281234
567891011
12131415161718
19202122232425
2627282930311
2345678

Browse by Category:

Notables: