The Darkside

Speculative execution is a term applied to the execution of code that you may not need the results of. With the abundance of spare processing power on servers these days (and workstations for that matter), you can easily make an application be far more responsive to the user or make processing tasks on a server application quicker.

A contrived example that I've come up: Consider a process that runs periodically on a server that:

1. Finds a list of all your customers from the SQL Server-based accounting system that have purchased an item from your online store in the past 24 hours (quick DB call) (1 sec in total)
2. For each customer, sum up the values of all their transactions for the past 6 months - it's a longer process because you have to fetch the data from data warehouse tables on a different SQL Server (2 sec per call)
3. For customers who have purchased more than ZAR100*, check if they have received a cheesy gift from the online loyalty app that you outsource to. (2 sec per call).
4. If they haven't received the gift yet, make another call to the online loyalty app to request that a gift be delivered (2 sec)

Most likely you have these running sequentially; looking at the tasks that need to be performed, they all seem to be dependant on the previous step. A quick tally based on a made of set of results returned in each step:

1. 5 customers are returned. Total running time elapsed: 1 sec
2. Sum up the values of each customer @ 2 seconds/call. Total running time elapsed: 11 sec
3. Let's say only 2 make it to this step that need to be checked. Total running time elapsed: 15 sec
4. One has already received a gift, so it's only 1 call left. Total running time elapsed: 16 sec

Here's an example of what the code may look like: 

IList<Customer> customers = GetListOfCustomers();

foreach (var customer in customers)

{

    if (GetPurchaseTotalForSixMonths(customer) > 100)

    {

        if (!HasCustomerReceivedGift(customer))

            RequestGiftForCustomer(customer);

    }

}

 Considering that in every step from the process your server application is waiting for a response from another server, it's most likely idling away. So let's run some of the tasks in parallel. We can run each call made in (2) at the same time as the call that may or may not have been made initially in (3). We can make the call to the online loyalty app, and if the result from the call in two requires the data from the call in (3), you use it. If not, too bad - you just discard the results. Your code may look something like this: 

        private static void Speculative()

        {

            IList<Customer> customers = GetListOfCustomers();

            foreach (var customer in customers)

            {

                var c = customer;   //**              

                bool meetsPurchaseLimit=false;

                bool hasReceivedGift=false;

                Fork.Begin()

                    .Call(() => meetsPurchaseLimit = (GetPurchaseTotalForSixMonths(c) > 100))

                    .Call(() => hasReceivedGift = HasCustomerReceivedGift(c))

                    .End();

                if (meetsPurchaseLimit && !hasReceivedGift)

                    RequestGiftForCustomer(customer); 

            } 

        }

 The Fork code that you see here is the same as what I've described in the article "An easy to use asynchronous fork".

Re-doing the calculation on the elapsed time, but now using speculative execution on the call made in (3), we get:

1. 5 customers are returned. Total running time elapsed: 1 sec
2. Sum up the values of each customer @ 2 seconds/call, but at the same time check if the customer received a gift. Total running time elapsed: 11 sec
3. Marrying up the results from the parallel calls, make the single call for the gift. Total running time elapsed: 12 sec

Back to the example: As you can see, you have a 25% saving in time. Yes, this is a contrived example with naive timing, but the general idea is what matters most. Don't feel bad about using your CPU's idle time to get work done, even if you're going to throw it away later.

* ZAR100 is equivalent to about USD500 or GBP300. Ok, it isn't so. Check here if you really want to.

** You may or may not be wondering why I have the line:  

                var c = customer;  

 It's to remove the warning I get from ReSharper - it complains about a possible "access to a modified closure". Have a look at this article from Eric Lippert explaining it.