RSS 2.0

Personal Info:

Joe Send mail to the author(s) leads the architecture of an experimental OS's developer platform, where he is also chief architect of its programming language. His current mission is to enable writing large-scale software that is reliable, secure, and scalable by-construction. Before this, Joe founded the Parallel Extensions to .NET project. He has been granted 19 patents, with 49 pending. When not working, Joe enjoys travelling with his wife, writing books, writing music, studying music theory & mathematics, and doing anything involving food & wine.

My books

My music

Disclaimer:
The content of this site are my own personal opinions and do not represent my employer's view in anyway.

© 2012, Joe Duffy

 
 Saturday, April 29, 2006

I don't know what's publicly available about our future ship schedules. But regardless, we begin M1 -- our first real coding milestone for the next version of the CLR -- on Monday. There's been some work going on in the meantime, of course, limited mostly to prototyping, design, and prioritization, but it's finally time to get serious, write real product code, and start hitting dates.

One fairly large item on our schedule is revamping our thread-pool. Our primary aim there is to enable fine-grained parallelism, and to supply new scheduling features that many people have asked for in the past. Today, coarse-grained parallelism is more attractive due to the costs associated with scheduling and dispatching work items, but we are going to change that.

This includes these tentative high level items:

  • Low performance overhead of queueing and dispatching work
  • Deadlock avoidance (surging) due to 100% blocking
  • Queue partitioning and isolation
  • Prioritization of work items
  • Cancellation of work items, possibly with support for Vista IO Cancellation
  • NUMA awareness such as CPU affinitization and/or user-hinted node affinitization
  • And, of course, enhanced debugging and diagnostics

We'd love any feedback on any of these, including which sound more or less important to you. And if you have an interesting problem or scenario we might not have considered, please, please, please let me know.

A colleague of mine recently referred me to the Cilk work at MIT. This paper supplies a good overview. We've been slowly arriving at a similar design, so it's great to have prior art from which to draw. The idea most important with respect to the thread-pool is how multiple queues can be backed by a single physical thread store, and further the way in which queues are dynamically load balanced via thread leases and work stealing.

4/29/2006 6:57:05 PM (Pacific Daylight Time, UTC-07:00)  #   

 Saturday, April 22, 2006

By now you’ve probably read things like Herb Sutter’s free lunch paper. And if you follow my blog at all, you’ll know that I do a bit of writing and thinking about how Microsoft can make our platform better suited for the multi-core era that stands in front of us.

Most people, when considering the topic of parallelism vis-à-vis multi-core, start by jumping straight to the bottom of the stack. I’ll admit that I sure did. They think about threads, locks, and the associated headaches. Some even think about the chip architecture and memory hierarchy. They take it for granted that the work exists. But these same people seldom stop to think—or when they do think often hit the same wall—about what workloads will actually substantially benefit from massive amounts of parallelism. This is a difficult topic.

Scientific computing of course has this nailed pretty good already. But how much of the code do you write that actually resembles scientific problems, like n-bodies, heat transfer, fluid dynamics, and the like? My guess is that, for most of Microsoft’s customers, the answer is: Not much. That’s especially true on the client, where data-intensive operations are often shipped to a high-end server for processing, leaving what amounts to quasi-workflow orchestration initiated by UI events, for example. I’m not going to refute the massive gains in CPU scalability we’ve seen over the past 10 years due to superscalar execution, via techniques like pipelining and branch prediction, and the effect that has had on client and server programs alike. But for most application code today, the network and disk are the limiting factors, not the CPU.

Of course, to the extent that there is work the CPU must perform for any problem—even for IO-bound ones—code needs to be architected to separate logical tasks, ensuring that a bunch of otherwise ready-to-run work doesn’t get backed up behind a blocking call unnecessarily. And of course, separating logical work is important for other reasons, like avoiding a hung UI thread. Unfortunately, we don’t make this overly easy today. Win32 and WinFX APIs (nor the associated documentation or tool support) are not overly helpful when it comes to figuring out the performance characteristics of the code they invoke, including latency and blocking. This makes it tricky to architect things as I suggest. New programming models like the CCR provide the infrastructure that could facilitate such a shift, but it will take hard work to get to a reasonable place.

Back to workloads. Consider server applications for a moment. The model of concurrency there is actually quite simple. And in fact I believe the majority of server programs will be equipped to exploit multi-core right away. Each incoming request is considered a logical task and is assigned to an available thread of work, often using the CLR’s thread-pool. Sharing between concurrent requests is (hopefully) minimal, meaning that the one-thread-per-request model leads to naturally good scaling. This works up to a point. Once the average number of available CPUs surpasses the average number of incoming workers, the need to assign multiple CPUs to a single request becomes more important. This is obviously very workload dependent. Databases already do this with individual queries. Their use a single-thread-per-request model, but often use individual query parallelization to get better utilization. SQL Server added support for this in 7.0. I’ve been working quite a bit over the past year on similar techniques for LINQ. I’m almost to the point where I can disclose more information publicly, in the form of a paper.

Search is clearly a workload of recent importance that, whether on the client or server, benefits tremendously from parallel execution. This applies not only to the act of searching, but also to the act of indexing the data in preparation for search. MSN and Google’s current desktop search products are cognizant not to interrupt your primary work by doing indexing while your computer is idle. But given a bunch more cores, they needn’t wait. Further, parallelizing search is a well researched topic. You still need to solve some tough problems like ensuring parallel tasks aren’t contending heavily for the disk (becoming IO bound), but it’s very possible.

There are of course other workloads. Graphics processing on modern computers is extremely parallel, currently handled by the GPU. But I am going to wrap up, and summarize all of this by saying: It remains to be seen whether most mainstream Windows programs can become highly parallel, and if they can, how profitable it will be. We'll also find out over time whether reaching that stage will require radically new programming models and a gradual shift over time. I am optimistic, and confident that parallel execution is the direction we ultimately need to go down. Surely the workloads are there, seemingly obscured by the traditional sequential approach to software.

4/22/2006 8:41:54 PM (Pacific Daylight Time, UTC-07:00)  #   

 Sunday, April 16, 2006

I'm writing an article for an upcoming MSDN Magazine CLR Inside Out column. And I am looking for topic suggestions.

Of course, my expertise is around concurrency, but I'm also a CLR internals-kinda geek. So, what do you want to read about?

I have some ideas. But I'll post them after I hear yours.

4/16/2006 6:46:56 PM (Pacific Daylight Time, UTC-07:00)  #   

I wrote about torn reads previously, in which, because loads from and stores to > 32-bit data types are not actually "atomic" on a 32-bit CPU, obscure magic values are seen in the program from time to time. This isn't as scary as "out of thin air" values, but can be troublesome nonetheless. I noted that, by using a lock, you can serialize access to the location to ensure safety.

You can of course write such thread-safe code that avoids taking a lock, usually motivated by performance. Vance has a pretty detailed write-up of this on MSDN. Most of the time, you shouldn't try to be so clever, as it will get you in trouble sooner or later, and is even worse to debug than a typical race. But for really hot code-paths, it can make a measurable difference. (Note the key word: measurable. If you've measured a problem, you might consider such techniques... but otherwise, stay far, far away. (Have I made enough qualifications and disclaimers yet?))

If you access individual pointer-sized byte segments of the data structure, such as 32-bit aligned segments (e.g. volatile or __declspec(align(x)) in VC++, all values on the CLR), you can load and store in a known order. Furthermore, you need to use the appropriate types of loads and stores with fences in the appropriate places; load/acquire and store/release are usually adequate. You can then use the intrinsic properties of this order to make statements about the correctness of your algorithm.

For example, imagine you have some code that increments a 64-bit counter on a 32-bit system. Aside from overflow, the value always increases. If you always increment the low 32-bits, followed by the high, and if you always read the high, followed by the low, you'll be guaranteed that, should you read a torn value, it will be too low rather than too high (not counting for overflow, of course). Sometimes it can be really low, such as when the low 32-bits wrap back to 0, in which case the higher 32-bit increment needs to carry one. Depending on your situation, this might be precisely what you are looking for. (I wrote some code last week that needed exactly this.)

For example, your typical code might read and write under a lock, in VC++/Win32:

ULONGLONG ReadCounter_Lock(
   
volatile ULONGLONG * pTarget, CRITICAL_SECTION * pCs)
{
    ULONGLONG val;

    EnterCriticalSection(pCs);
    val = *pTarget;
    LeaveCriticalSection(pCs);

    return val;
}

ULONGLONG IncrCounter_Lock(
    volatile ULONGLONG * pTarget, CRITICAL_SECTION * pCs)
{
    ULONGLONG val;

    EnterCriticalSection(pCs);
    val = *pTarget;
    *pTarget = val + 1;
    LeaveCriticalSection(pCs);

    return val;
}

But, using the load/store order described above, it can become lock free:

#define LO_LONG(x) (reinterpret_cast<volatile LONG *>((x)))
#define HI_LONG(x) (reinterpret_cast<volatile LONG *>((x)) + 1)

ULONGLONG ReadCounter_NoLock(volatile ULONGLONG * pTarget)
{
    ULONGLONG val;

#ifdef _Win64

    val = *pTarget;

#else

    // Read high 32-bits first, then low:
    *HI_LONG(&val) = *HI_LONG(pTarget);
    *LO_LONG(&val) = *LO_LONG(pTarget);

#endif

    return val;
}

ULONGLONG IncrCounter_NoLock(
    volatile ULONGLONG * pTarget)
{
    ULONGLONG oldVal;

#ifdef _Win64

    oldVal = static_cast<LONGLONG>(
        InterlockedIncrement64(static_cast<LONGLONG *>(pTarget)));

#else

    // Increment the low 32-bits first, then high:
    if ((*LO_LONG(&oldVal) =
        InterlockedIncrement(LO_LONG(pTarget))) == 0)
    {
        *HI_LONG(&oldVal) = InterlockedIncrement(HI_LONG(pTarget));
    }
    else
    {
        *HI_LONG(&oldVal) = *HI_LONG(pTarget);
    }

#endif

    return oldVal;
}

It's obvious which is simpler to code, understand, and maintain. But the latter technique can come in handy when you're in a pinch.

For information on other similar techniques, including multi-word CAS and object-based STM, Tim Harris's recent "Concurrent programming with locks" paper is an excellent read. Most of it isn't built and ready for you to use today, but the details of the algorithms are in there if you'd like to play around a little. And there's a lot of literature out there about creating lock-free data structures. Interestingly, you can end up worse off than if you'd used a lock in the first place. Many such lock free algorithms are optimistic meaning that they do a bunch of work hoping not to run into contention; when they do, they have to throw away work, rinse, and repeat. Your mileage can vary dramatically based on workload.

4/16/2006 6:32:52 PM (Pacific Daylight Time, UTC-07:00)  #   

 Thursday, April 06, 2006

My new book is finally on book shelves and in my hands. What a relief.

Now it's time to rinse and repeat. I've been quite lazy with regards to the new book project, but it's time to get serious. I'm laying down some pretty intense milestones over the next few months. We'll see if I can hit the dates.

While the .NET Framework book used primarily a breadth-oriented approach, the Concurrency book is quite different. It covers a smaller set of topics, albeit very depth-oriented.

4/6/2006 8:16:42 PM (Pacific Daylight Time, UTC-07:00)  #   

 Saturday, April 01, 2006

I was in Las Vegas for the better part of last week. Aside from winning money (for once), I also ate at some great places: the 5-course prix fixe menu at Michael Mina (northwest seafood, at the Bellagio), Bartolotta (modern Italian, at the Wynn), brunch at the Mesa Grill (southwestern, at Caesar's Palace), Olives (new Mediterranean, at the Bellagio), and the best of all, the 16-course prix fixe at Joël Robuchon at The Mansion (modern French, at the MGM Grand).

In fact, Joël Robuchon's beat my favorite two dining spots to date: Aujourd'hui, and the Herbfarm. Not to mention that they were nice enough to accomodate and craft up an entirely custom 16-course vegetarian tasting menu. Dinner came complete with fancy scroll-like copies of the menus to take home, wrapped in purple silk. But I figured I'd also digitize it to help remember. Here's the non-vegetarian menu:

Joël Robuchon at The Mansion
March 25th, 2006

Menu Dégustation
Tasting Menu

La Pomme
cuillère de perles, de son jus rafraîchi d’un granite de vodka
Apple pearl, vodka granite

Le Caviar Osciètre
dans une délicate gelée recouverte d’une onctueuse crème de chou-fleur
Oscetra caviar topped with a delicate gelée and a smooth cauliflower cream

Le Foie Gras
en mille-feuille caramélise d’anguille fumée aux saveurs orientales
Foie gras, mille-feuille of smoked eel with oriental flavors

Le Thon
en tartare, poivron rouge confit a la bergamote et au jambon sèche
Tuna tartar, cold red bell pepper confit with bergamot and dry cured ham

La Langoustine
truffée et cuite en ravioli a l’étuvée de chou vert
Truffled langoustine ravioli with steamed green cabbage

La Laitue
en fin veloute sur un flan tremblotant a l’oignon doux
Light lettuce cream on top of a delicate sweet onion custard

La Noix de Saint-Jacques
en cannelloni aux courgettes sous un voile de lard d’Arnad et une émulsion de parmigiano
Cannelloni of scallops and zucchini, parmesan emulsion

Le Homard
au coulis de pissenlit avec quelques feuilles crues de barbes-de-capucin relevées d’une vinaigrette coralline
Lobster, pissenlit coulis, capucin leaves and sea urchin vinaigrette

L’Os a Mœlle
de bœuf de Kobe aux légumes printaniers
Kobe beef bone marrow, spring vegetables

L’Ormeau
et l’artichaut poivrade dans un court bouillon au gingembre
Abalone, baby artichokes in a ginger bouillon

Le Bar
pole a la citronnelle avec une étuvée de jeunes poireaux
Pan-fried sea bass with a lemon grass foam and stewed baby leeks

L’Amadai
cuit en écailles et servi sur une nage au yuriné
Amadai in a lily bulb broth

Le Veau
en cote au plat avec un jus gras et escorte de taglierinis de légumes au pistou
Sautéed veal chop with natural jus and vegetable taglierinis flavored with pesto

L’Epeautre
du pays de Sault mitonne et dore a la feuille d’or
Sault wild oatmeal, gold leaf

Le Bahia
en fin crémeaux de papaye, jus de cassis
Guava and papaya granite, cream of cassis and orange macaroon

La Fraise
glacée aux coquelicots, en popcorns caramélises, sirop de cachaça
Poppy sorbet, caramelized popcorns, cachaça syrup

Le Café Express
Espresso

Petits Fours

Yes, this was a tasting menu. It was not a la carte. I ate each of those dishes in the course of about 4 1/2 hours. And had a 2002 Puligny-Montrachet 1er Cru, Les Pucelles to go along with all of it. Yes, just one bottle. Excess was not on the agenda for that night...

4/1/2006 10:26:56 PM (Pacific Daylight Time, UTC-07:00)  #   

 Thursday, March 30, 2006

Wow, not only does Vance Morrison have a blog--he's a performance architect on the CLR team--but he recently wrote two articles on reader-writer locks:

In them, he walks through a custom implementation of a lock, and then does some insightful performance analysis on it. As usual with Vance's writing, it's very detailed and precise.

3/30/2006 1:07:33 PM (Pacific Daylight Time, UTC-07:00)  #   

 Saturday, March 25, 2006

The profiler that ships with Visual Studio is great for "real" CPU profiling. But let's face it: there are still some situations where a good ole' stopwatch works just fine (of the System.Diagnostics.Stopwatch variety). For example, when you're trying to do some quick and dirty measurement on a very specific region of code, and don't want to deal with the rest of the noise.

The BCL stopwatch isn't inherently thread-safe. Even if you protect access to it (and somehow account for the overhead of doing so), it maintains a single counter. In the past, I've wanted to measure the total amount of time spent inside a select few regions of code, across all threads. The profiler works for this, but you need to get the sampling granularity right, and deal with all of the extra data collected. Then you have to mine it.

So I whipped up a stopwatch that maintains a counter that is the cumulative total of all threads that have started/stopped it, across an entire AppDomain. It's nothing incredibly clever, but I've found it to be quite useful. In many code projects I have, I've simply set up a file that declares a bunch of static ThreadSafeStopwatches, which I can then just call from anywhere.

Here's the code (also available for download here: ThreadSafeStopwatch.cs):

using System;

using System.Collections.Generic;

using System.Diagnostics;

using System.Threading;

 

/// <summary>

/// This class enables AppDomain-wide profiling of multi-threaded

/// code, by tracking a cumulative number of ticks spent across all

/// threads. If multiple threads start the same watch in parallel,

/// this class ensures that we count time for both threads, and that

/// they do not interfere with each other. It does this by storing an

/// internal stop-watch in TLS, and an instance-wide tick counter.

///

/// Note that the cumulative count is only ever incremented if a

/// thread actually calls Stop in its own stop-watch. If a thread

/// routine terminates w/out stopping the watch, it's as if it never

/// began.

/// </summary>

public sealed class ThreadSafeStopwatch : IDisposable

{

 

    /** Fields **/

    private long ticks;

 

    // These thread-specific fields are used to maintain a cache

    // of thread-safe stopwatches to actual stopwatches. For long

    // running threads, we can build up some amount of trash over

    // time, so a cache management scheme is implemented via a

    // combination of mechanisms: Dispose, ~ThreadSafeStopwatch,

    // GetWatch, and PruneCache.

    [ThreadStatic]

    private static Dictionary<WeakReference, Stopwatch> threadWatches;

    [ThreadStatic]

    private static int cacheCounter;

 

    /** Properties **/

    public long ElapsedTicks

    {

        get { return ticks; }

    }

    public float ElapsedMilliseconds

    {

        get { return (float)ticks / TimeSpan.TicksPerMillisecond; }

    }

 

    /** Methods **/

    ~ThreadSafeStopwatch()

    {

        Dispose(false);

    }

 

    public void Dispose()

    {

        Dispose(true);

    }

 

    private void Dispose(bool disposing)

    {

        // Clean up the associated cache entry with this object.

 

        // NOTE: I am explicitly not guarding this code-path by if

        // (disposing) { ... } because the Dictionary is not finaliz-

        // able. Thus, I know it is safe to access it. In the case of

        // non-process-exit finalizers, we do want to clean up the

        // associated cache entry, thus we access another object on

        // the finalizer code-path.

 

        if (!Environment.HasShutdownStarted)

        {

            Dictionary<WeakReference, Stopwatch> watches = threadWatches;

            WeakReference thisRef = new WeakReference(this);

            if (watches != null && watches.ContainsKey(thisRef))

            {

                // Deallocate the cache entry:

                watches.Remove(thisRef);

            }

        }

    }

 

    const int threadCachePruneCount = 100;

    private Stopwatch GetWatch()

    {

        if (threadWatches == null)

        {

            // First time called on this thread, allocate a new dictionary:

            threadWatches = new Dictionary<WeakReference, Stopwatch>(

                WeakRefEqualityComparer.Comparer);

        }

        else

        {

            // This has been called before. Increment the thread counter;

            // every so often, we prune out trash being held alive by our

            // cache.

            if (++cacheCounter % threadCachePruneCount == 0)

            {

                PruneCache();

                cacheCounter = 0;

            }

        }

 

        // Now look for the associated stopwatch:

        Stopwatch sw;

        if (threadWatches.TryGetValue(new WeakReference(this), out sw))

            return sw;

 

        // If we didn't find the stopwatch, simply return null to

        // indicate that the caller needs to allocate a new one:

        return null;

    }

 

    private void PruneCache()

    {

        // BUGBUG: This is probably a poor cache management policy. But

        // I'm only enabling this in DEBUG builds for now, and until I

        // run into a real problem, I'm not spending time on it.

 

        List<WeakReference> toRemoveWrefs = null;

 

        // Look for dead references, and add them to the list (which is

        // lazily created, by the way).

        foreach (WeakReference wr in threadWatches.Keys)

        {

            // If the weak-reference is no longer alive, we add it to the

            // list of 'to-remove' stop-watches.

            if (!wr.IsAlive)

            {

                if (toRemoveWrefs == null)

                    toRemoveWrefs = new List<WeakReference>();

                toRemoveWrefs.Add(wr);

            }

        }

 

        // If we found any dead entries, remove them now.

        if (toRemoveWrefs != null)

        {

            foreach (WeakReference wr in toRemoveWrefs)

            {

                threadWatches.Remove(wr);

            }

        }

    }

 

    [Conditional("DEBUG")]

    public void Start()

    {

        // We look in TLS to see if this thread has already allocated a

        // stopwatch for the current thread-safe stopwatch. This is

        // thread-safe, of course, since each thread gets their own list

        // of Stopwatches (reentrancy aside--there aren't any blocking

        // points below):

 

        // Since we are about to retrieve something from TLS, and use it

        // across a set of paired operations (Start/Stop), we mark the

        // beginning of a thread-affinity region.

        Thread.BeginThreadAffinity();

 

        // Access TLS:

        Stopwatch sw = GetWatch();

 

        if (sw == null)

        {

            // No watch was found, allocate a new one and publish it.

            sw = new Stopwatch();

            threadWatches.Add(new WeakReference(this), sw);

        }

 

        // First, ensure we haven't begun it yet. If the stopwatch is

        // already running, we ignore this call. This is consistent with

        the System.Diagnostics.Stopwatch

        // class's behavior.

        if (!sw.IsRunning)

        {

            // And if that check succeeds, start the stop-watch ticking.

            sw.Start();

        }

    }

 

    [Conditional("DEBUG")]

    public void Reset()

    {

        // Get the current stopwatch in TLS -- see above comments (in

        // Start) for details on thread-safety.

        Stopwatch sw = GetWatch();

 

        // If we found one, reset it.

        if (sw != null)

            sw.Reset();

 

        // And also set our cumulative ticks to 0.

        ticks = 0;

    }

 

    [Conditional("DEBUG")]

    public void Stop()

    {

        // Get the current stopwatch in TLS -- see above comments (in

        // Start) for details on thread-safety.

        Stopwatch sw = GetWatch();

 

        // First, ensure we are running. If the stopwatch isn't running

        // yet, we ignore this call. This is consistent with the System.

        // Diagnostics.Stopwatch class's behavior.

        if (sw != null && sw.IsRunning)

        {

            // Add the stopwatch's total time to our instance counter.

            // This has to be an interlocked operation, because the whole

            // point of this class is to be shared across threads. 'ticks'

            // is the only instance state.

            Interlocked.Add(ref ticks, sw.ElapsedTicks);

 

            // We reset the stopwatch because we want to start at 0 upon

            // the next invocation to 'Start' -- the cumulative time is

            // kept in the 'ticks' variable.

            sw.Reset();

 

            // We can now end the thread-affinity that was started in the

            // Start operation above.

            Thread.EndThreadAffinity();

        }

    }

 

    class WeakRefEqualityComparer : IEqualityComparer<WeakReference>

    {

        internal static WeakRefEqualityComparer Comparer =

            new WeakRefEqualityComparer();

 

        public bool Equals(WeakReference wr1, WeakReference wr2)

        {

            // For purposes of our hash-table, if two weak-references

            // refer to the same object, we consider them equal.

            object o1 = wr1.Target;

            object o2 = wr2.Target;

 

            if (!wr1.IsAlive || !wr2.IsAlive)

                return false;

 

            // We shouldn't ever have null weak-references that aren't

            // dead.

            Debug.Assert(o1 != null && o2 != null);

 

            // If the two underlying objects are equal, we pretend the

            // weak-refs are too.

            return o1.Equals(o2);

        }

 

        public int GetHashCode(WeakReference wr)

        {

            object o = wr.Target;

 

            // Just return 0 for dead objects. We actually shouldn't

            // ever use a dead object for hashing, although there could

            // be some benign races above that result in this case. I

            // haven't convinced myself otherwise, and they will get clean-

            // ed up with the normal finalization code-path. It's A-OK.

            if (!wr.IsAlive)

                return 0;

 

            // Again, shouldn't get a live weak-ref that has a null

            // object ref.

            Debug.Assert(o != null);

 

            // Now, simply return the underlying object's hash-code.

            return o.GetHashCode();

        }

    }

}

3/25/2006 3:19:52 PM (Pacific Daylight Time, UTC-07:00)  #   

The Whidbey version of Rotor just went up for download on MSDN on Thursday. I downloaded it and built it this morning.

While some of the big rock features are getting press (e.g. generics, LCG, anonymous methods/closures, etc.), some of the smaller features are plenty cool, and can be grokked in their entirety in much less time. For example, do a 'grep -i nullable clr/src/*/*.*' if you want to see what went into implementing the Nullable DCR that Soma mentioned over here. And check out stuff like the WrapNonCompliantException function in vm/excep.cpp (and its callsites) to see how non-CLS exceptions get wrapped. And of course there's all that reliability work that went into Whidbey, leading to things like SafeHandles (vm/safehandle.cpp), and OOM and SO hardening.

Most of it's there for you to tinker with. Or to simply print out and enjoy, reading it as you sit beside the fireplace with a nice Bourdeaux. To each his (or her) own.

3/25/2006 3:00:51 PM (Pacific Daylight Time, UTC-07:00)  #   

 Tuesday, March 21, 2006

Ahh, I fondly remember the days of all-nighters.

I found this great interview, done in September '05, where Max (co-founder of PayPal) discusses the virtues of entrepreneurship and late-night hacking.

I miss those days. Oh, how I miss them.

3/21/2006 11:14:07 PM (Pacific Daylight Time, UTC-07:00)  #   

 

Recent Entries:

Search:

Browse by Date:
<April 2006>
SunMonTueWedThuFriSat
2627282930311
2345678
9101112131415
16171819202122
23242526272829
30123456

Browse by Category:

Notables: