So the method we'll be using here is just to mimic what Sharepoint does when you check the box to "Also refresh as soon as possible" in the schedule configuration page. To accomplish this we open the profiler, connect it to whatever instance our Sharepoint PowerPivot database is hosted on and filter you TextData to "%Schedule%" or filter to just the Sharepoint PowerPivot database (SP2010_PowerPivot_Service_Application in our case) - then open up your test PowerPivot schedule configuration, check the box, click OK, wait for the schedule history to come back up and then stop the trace. Now you know you've got what you need, you just have to find it:

So here we have Sharepoint asking for information about the schedule (identified by the ItemID from DataRefresh.Items) and then passing the data it received to PersistSchedule. If you take a close look at the PersistSchedule parameters, the last one is @RunNow = 1. So clearly, this is what we're after. We went the most tedious route and manually looked up the ItemID in DataRefresh.Items:

SET NOCOUNT ON
 
DECLARE @ItemID UNIQUEIDENTIFIER = '30D39B0E-DA8B-489B-A713-3B29D208B51F'
 
DECLARE
	@ItemName [NVARCHAR](200),
	@SPSiteID [UNIQUEIDENTIFIER],
	@SPWebID [UNIQUEIDENTIFIER],	
	@LastModifiedBy [NVARCHAR](450),
	@ScheduleLastUpdatedBy [NVARCHAR](450),
	@UserIdentity [NVARCHAR](450),
	@EmailNotification [BIT], 
	@EmailList [NVARCHAR](450),
	@SecurityConfiguration [INT],
	@SSApplicationID [NVARCHAR](256),
	@ProcessAllDataSources [BIT],
	@RV [BIGINT],
	-- Schedule info
	@Enabled [BIT], 
	-- Schedule details
	@NextProcessDate [DATETIME],
	@ScheduleStartDate [DATETIME],
	@FrequencyKey [NCHAR](1),
	@RepeatFrequency [INT],
	@Sunday [BIT],
	@Monday [BIT],
	@Tuesday [BIT],
	@Wednesday [BIT],
	@Thursday [BIT],
	@Friday [BIT],
	@Saturday [BIT],
	@Weekday [BIT], 
	@WeekendDay [BIT], 
	@DAY [BIT], 
	@MonthlyPeriod [TINYINT], 
	@MonthlySpecificDay [INT], 
	@ProcessAfterBusinessHours [BIT], 
	@SpecificTime [DATETIME],
	@RunNow [BIT]
 
DECLARE @Schedule TABLE (
	[ItemID] [UNIQUEIDENTIFIER] NOT NULL,
	[Enabled] [BIT] NOT NULL,
	[RV] [BIGINT] NULL,
	[ItemName] [NVARCHAR](200) NOT NULL,
	[SPSiteID] [UNIQUEIDENTIFIER] NOT NULL,
	[SPWebID] [UNIQUEIDENTIFIER] NOT NULL,
	[UserIdentity] [NVARCHAR](450) NULL,
	[LastModifiedBy] [NVARCHAR](450) NOT NULL,
	[EmailNotification] [BIT] NOT NULL,
	[EmailList] [NVARCHAR](450) NULL,
	[SecurityConfiguration] [INT] NOT NULL,
	[SSApplicationID] [NVARCHAR](256) NULL,
	[ProcessAllDataSources] [BIT] NOT NULL,
	[ScheduleDetailID] [UNIQUEIDENTIFIER] NOT NULL,
	[ScheduleStartDate] [DATETIME] NOT NULL,
	[FrequencyKey] [NCHAR](1) NOT NULL,
	[RepeatFrequency] [INT] NOT NULL,
	[Sunday] [BIT] NOT NULL,
	[Monday] [BIT] NOT NULL,
	[Tuesday] [BIT] NOT NULL,
	[Wednesday] [BIT] NOT NULL,
	[Thursday] [BIT] NOT NULL,
	[Friday] [BIT] NOT NULL,
	[Saturday] [BIT] NOT NULL,
	[WeekDay] [BIT] NOT NULL,
	[WeekendDay] [BIT] NOT NULL,
	[DAY] [BIT] NOT NULL,
	[MonthlyPeriod] [TINYINT] NOT NULL,
	[MonthlySpecificDay] [INT] NOT NULL,
	[ProcessAfterBusinessHours] [BIT] NOT NULL,
	[SpecificTime] [DATETIME] NULL
)
 
INSERT INTO @Schedule
EXEC DataRefresh.GetSchedule @ItemID
 
SELECT 
	 @ItemName = [ItemName]
	,@SPSiteID = [SPSiteID]
	,@SPWebID = [SPWebID]
	,@LastModifiedBy = [LastModifiedBy]
	,@ScheduleLastUpdatedBy = N'DOMAIN\tlaqua'
	,@UserIdentity = [UserIdentity]
	,@EmailNotification = [EmailNotification]
	,@EmailList = [EmailList]
	,@SecurityConfiguration = [SecurityConfiguration]
	,@SSApplicationID = [SSApplicationID]
	,@ProcessAllDataSources = [ProcessAllDataSources]
	,@RV = [RV]
		-- Schedule info
	,@Enabled = [Enabled]
		-- Schedule details
	,@NextProcessDate = CAST(GETDATE() AS DATE)
	,@ScheduleStartDate = [ScheduleStartDate]
	,@FrequencyKey = [FrequencyKey]
	,@RepeatFrequency = [RepeatFrequency]
	,@Sunday = [Sunday]
	,@Monday = [Monday]
	,@Tuesday = [Tuesday]
	,@Wednesday = [Wednesday]
	,@Thursday = [Thursday]
	,@Friday = [Friday]
	,@Saturday = [Saturday]
	,@Weekday = [Weekday]
	,@WeekendDay = [WeekendDay]
	,@DAY = [DAY]
	,@MonthlyPeriod = [MonthlyPeriod]
	,@MonthlySpecificDay = [MonthlySpecificDay]
	,@ProcessAfterBusinessHours = [ProcessAfterBusinessHours]
	,@SpecificTime = [SpecificTime]
	,@RunNow = 1
FROM @Schedule
 
EXEC DataRefresh.PersistSchedule
	 @ItemID = @ItemID
	,@ItemName = @ItemName
	,@SPSiteID = @SPSiteID
	,@SPWebID = @SPWebID
	,@LastModifiedBy = @LastModifiedBy
	,@ScheduleLastUpdatedBy = @ScheduleLastUpdatedBy
	,@UserIdentity = @UserIdentity
	,@EmailNotification = @EmailNotification
	,@EmailList = @EmailList
	,@SecurityConfiguration = @SecurityConfiguration
	,@SSApplicationID = @SSApplicationID
	,@ProcessAllDataSources = @ProcessAllDataSources
	,@RV = @RV
		-- Schedule info
	,@Enabled = @Enabled
		-- Schedule details
	,@NextProcessDate = @NextProcessDate
	,@ScheduleStartDate = @ScheduleStartDate
	,@FrequencyKey = @FrequencyKey
	,@RepeatFrequency = @RepeatFrequency
	,@Sunday = @Sunday
	,@Monday = @Monday
	,@Tuesday = @Tuesday
	,@Wednesday = @Wednesday
	,@Thursday = @Thursday
	,@Friday = @Friday
	,@Saturday = @Saturday
	,@Weekday = @Weekday
	,@WeekendDay = @WeekendDay
	,@DAY = @DAY
	,@MonthlyPeriod = @MonthlyPeriod
	,@MonthlySpecificDay = @MonthlySpecificDay
	,@ProcessAfterBusinessHours = @ProcessAfterBusinessHours
	,@SpecificTime = @SpecificTime
	,@RunNow = @RunNow

The only parameters that don't come from GetSchedule are NextRunDate which we set to midnight of today and RunNow which is, of course, 1 and ScheduleLastModifiedBy (I just tossed my username in there, I assume you could use the system identity as well).

And then we set it up to refresh every 10 minutes via Agent job to test:

Happy refreshing! Let me know if you have any questions.

First, we assume that our dimension already exists (and we were using some other checksum or MERGE to maintain it). We have to add a column to store the MD5 hash:

ALTER TABLE dbo.DimCustomer ADD
	MD5 VARCHAR(34) NOT NULL DEFAULT ''

Second, we need a staging table to store updated/changed rows. Script out the current dimension as a CREATE, remove all unneeded constraints and indexes, and create a staging table as a heap:

CREATE TABLE [dbo].[Staging_DimCustomer_UpdatedRows](
	[CustomerKey] [INT] NOT NULL,
	[GeographyKey] [INT] NULL,
	[CustomerAlternateKey] [NVARCHAR](15) NOT NULL,
	[Title] [NVARCHAR](8) NULL,
	[FirstName] [NVARCHAR](50) NULL,
	[MiddleName] [NVARCHAR](50) NULL,
	[LastName] [NVARCHAR](50) NULL,
	[NameStyle] [BIT] NULL,
	[BirthDate] [DATETIME] NULL,
	[MaritalStatus] [NCHAR](1) NULL,
	[Suffix] [NVARCHAR](10) NULL,
	[Gender] [NVARCHAR](1) NULL,
	[EmailAddress] [NVARCHAR](50) NULL,
	[YearlyIncome] [MONEY] NULL,
	[TotalChildren] [TINYINT] NULL,
	[NumberChildrenAtHome] [TINYINT] NULL,
	[EnglishEducation] [NVARCHAR](40) NULL,
	[SpanishEducation] [NVARCHAR](40) NULL,
	[FrenchEducation] [NVARCHAR](40) NULL,
	[EnglishOccupation] [NVARCHAR](100) NULL,
	[SpanishOccupation] [NVARCHAR](100) NULL,
	[FrenchOccupation] [NVARCHAR](100) NULL,
	[HouseOwnerFlag] [NCHAR](1) NULL,
	[NumberCarsOwned] [TINYINT] NULL,
	[AddressLine1] [NVARCHAR](120) NULL,
	[AddressLine2] [NVARCHAR](120) NULL,
	[Phone] [NVARCHAR](20) NULL,
	[DateFirstPurchase] [DATETIME] NULL,
	[CommuteDistance] [NVARCHAR](15) NULL,
	[MD5] [VARCHAR](34) NOT NULL)

Now in to SSIS - We will be building:

1. Execute SQL Task to Truncate our Staging table(s)
2. Data Flow Task to Insert new rows and Stage updated rows
1. OLE DB Source to retrieve our source data
2. Script Component to Generate Row Numbers
3. Conditional Split to Evenly Distribute Rows
4. Script Component to Generate MD5 Hashes
5. Union All to Squish it all back together
6. Lookup to get the existing MD5 Hash (if it exists)
7. Conditional Split to separate Unchanged and Changed rows
8. RowCount Transformation
9. OLE DB Destination for Changed rows
10. OLE DB Destination for New rows
3. Execute SQL Task to Update changed rows

Completed Control Flow

Completed Data Flow

1: Execute SQL Task to Truncate our Staging table(s)

TRUNCATE TABLE Staging_DimCustomer_UpdatedRows;

2.1: OLE DB Source to retrieve our source data

Configure the Connection Manager - for the source we'll just select from the dimension minus the surrogate key and MD5 column:

You most likely won't need to exclude any columns - here we unselect the surrogate key and the MD5 column:

2.2: Script Component to Generate Row Numbers

Add one output column to Output 0 in your Row Number Script Transformation

The following code will assign the current row number and increment it for the next row through the buffer:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    private int _rowCount = 1;
 
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        Row.RowNumber = _rowCount++;
    }
}

2.3: Conditional Split to Evenly Distribute Rows

Now in the conditional split, we distribute rows across multiple outputs by using the modulo operator. Modify to suit the number of threads you need to remove any bottlenecks in the MD5 calculation (you may only need one thread):

2.4: Script Component to Generate MD5 Hashes

Add one output column to Output 0 in your Generate MD5 Script Transformation

Select all input columns or just select ones that are going to be in the hash. Since we explicitly specify columns in the script, the hash will only be generated with exactly what columns you tell it to use.

The following script will generate a string representation of a MD5 hash. We do this so that we can do direct comparisons with HASHBYTES converted output (and virtually everything likes dealing with strings better than binary). We do end up doubling the storage cost of the MD5 hash by storing it as a string, but we were fine with that:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    private System.Security.Cryptography.MD5 _md5 =
                new System.Security.Cryptography.MD5CryptoServiceProvider();
 
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        try
        {
            string hashSource =
                (Row.GeographyKey_IsNull ? "" : Row.GeographyKey.ToString())
                + (Row.Title_IsNull ? "" : Row.Title)
                + (Row.FirstName_IsNull ? "" : Row.FirstName)
                + (Row.MiddleName_IsNull ? "" : Row.MiddleName)
                + (Row.LastName_IsNull ? "" : Row.LastName)
                + (Row.NameStyle_IsNull ? "" : Row.NameStyle.ToString())
                + (Row.BirthDate_IsNull ? "" : Row.BirthDate.ToString())
                + (Row.MaritalStatus_IsNull ? "" : Row.MaritalStatus)
                + (Row.Suffix_IsNull ? "" : Row.Suffix)
                + (Row.Gender_IsNull ? "" : Row.Gender)
                + (Row.EmailAddress_IsNull ? "" : Row.EmailAddress)
                + (Row.YearlyIncome_IsNull ? "" : Row.YearlyIncome.ToString())
                + (Row.TotalChildren_IsNull ? "" : Row.TotalChildren.ToString())
                + (Row.NumberChildrenAtHome_IsNull ? "" : Row.NumberChildrenAtHome.ToString())
                + (Row.EnglishEducation_IsNull ? "" : Row.EnglishEducation)
                + (Row.SpanishEducation_IsNull ? "" : Row.SpanishEducation)
                + (Row.FrenchEducation_IsNull ? "" : Row.FrenchEducation)
                + (Row.EnglishOccupation_IsNull ? "" : Row.EnglishOccupation)
                + (Row.SpanishOccupation_IsNull ? "" : Row.SpanishOccupation)
                + (Row.FrenchOccupation_IsNull ? "" : Row.FrenchOccupation)
                + (Row.HouseOwnerFlag_IsNull ? "" : Row.HouseOwnerFlag)
                + (Row.NumberCarsOwned_IsNull ? "" : Row.NumberCarsOwned.ToString())
                + (Row.AddressLine1_IsNull ? "" : Row.AddressLine1)
                + (Row.AddressLine2_IsNull ? "" : Row.AddressLine2)
                + (Row.Phone_IsNull ? "" : Row.Phone)
                + (Row.DateFirstPurchase_IsNull ? "" : Row.DateFirstPurchase.ToString())
                + (Row.CommuteDistance_IsNull ? "" : Row.CommuteDistance)
                ;
 
            byte[] hashBytes = _md5.ComputeHash(
                System.Text.UnicodeEncoding.Unicode.GetBytes(hashSource));
 
            System.Text.StringBuilder sb = new System.Text.StringBuilder();
 
            for (int i = 0; i < hashBytes.Length; i++)
            {
                sb.Append(hashBytes[i].ToString("X2"));
            }
 
            Row.MD5 = "0x" + sb.ToString();
        }
        catch (Exception e)
        {
            Row.MD5 = "Error";
        }
    }
}

To quickly generate the string concatenation statement:

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: + (Row.\1_IsNull ? "" : Row.\1)
6. Replace All, remove the leading +, and remove the garbage at the bottom, remove surrogate and business keys
7. Paste results in to the script
8. Add .ToString() to non-string columns and fix any column names (issues will be underlined in red)

Copy your completed script component once for each thread you'll be running. When you copy a script component, you need to actually open the script after copying it and click Edit Script before it will validate.

2.5: Union All to Squish it all back together

2.6: Lookup to get the existing MD5 Hash (if it exists)

Configure the Lookup component to redirect rows with no matches to the No Match output:

For your connection, select the surrogate key, business key(s), and MD5 hash from your existing dimension table:

Lookup based on the business key(s) and bring back the existing MD5 and surrogate key

2.7: Conditional Split to separate Unchanged and Changed rows

Name the output for rows where the ExistingMD5 is the same as the new MD5 "Ignore" and name the Default output "Update"

2.8: RowCount Transformation for Ignored Rows

For your Ignored Rows Row Count transformation, you'll have to create a variable to store the rowcount in (we don't use it, but that's just how the transformation works):

2.9: OLE DB Destination for Changed rows

We'll insert rows that need to be updated in to the previously created Staging_DimCustomer_UpdatedRows table:

2.10: OLE DB Destination for New rows

Finally, insert new rows directly in to the dimension:

3: Execute SQL Task to Update changed rows

The final step is to take those rows we staged for updating and actually perform the update:

UPDATE a SET
 [GeographyKey] = b.[GeographyKey]
,[CustomerAlternateKey] = b.[CustomerAlternateKey]
,[Title] = b.[Title]
,[FirstName] = b.[FirstName]
,[MiddleName] = b.[MiddleName]
,[LastName] = b.[LastName]
,[NameStyle] = b.[NameStyle]
,[BirthDate] = b.[BirthDate]
,[MaritalStatus] = b.[MaritalStatus]
,[Suffix] = b.[Suffix]
,[Gender] = b.[Gender]
,[EmailAddress] = b.[EmailAddress]
,[YearlyIncome] = b.[YearlyIncome]
,[TotalChildren] = b.[TotalChildren]
,[NumberChildrenAtHome] = b.[NumberChildrenAtHome]
,[EnglishEducation] = b.[EnglishEducation]
,[SpanishEducation] = b.[SpanishEducation]
,[FrenchEducation] = b.[FrenchEducation]
,[EnglishOccupation] = b.[EnglishOccupation]
,[SpanishOccupation] = b.[SpanishOccupation]
,[FrenchOccupation] = b.[FrenchOccupation]
,[HouseOwnerFlag] = b.[HouseOwnerFlag]
,[NumberCarsOwned] = b.[NumberCarsOwned]
,[AddressLine1] = b.[AddressLine1]
,[AddressLine2] = b.[AddressLine2]
,[Phone] = b.[Phone]
,[DateFirstPurchase] = b.[DateFirstPurchase]
,[CommuteDistance] = b.[CommuteDistance]
,[MD5] = b.[MD5] 
FROM DimCustomer a
	 INNER JOIN Staging_DimCustomer_UpdatedRows b
		ON a.CustomerKey = b.CustomerKey

To quickly generate the update statement:

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: ,[\1] = b.[\1]
6. Replace All, remove surrogate key and garbage at the bottom, remove leading comma
7. Add UPDATE a SET at the top
8. Add appropriate FROM clause at the bottom

And bob's your uncle.

In closing...

Why not write a reusable code block for the MD5 hash that you could cut and paste? Why hardcode columns?
Well, there's really only two ways to do it dynamically. One is to loop over each column in the InputBuffer and concatenate them. Man, that's a bunch of extra logic when we know very well what columns are, null handling is a little sketch, and proper datatype handling requires some special logic. Looping over the InputBuffer also just plain feels dirty. The second option is Reflection. Now we could approach reflection two ways - first we could iterate over the Row definition and concatenate the columns for each InputRow, but this is INCREDIBLY expensive (reflection is crazy slow). The second Reflection approach would be to construct a DynamicMethod at runtime and call that dynamic method for each row. While this could certainly be as fast as hardcoding, it's very obscure and difficult for most people to understand. Do you want code in your packages that's hard to understand?

We chose hardcode the columns because this is the absolute minimum amount of work that needs to be done to generate a reliable MD5 hash and is therefore the fastest possible solution.

Why not do that Reflection and DynamicMethod thing in a custom transformation?
Because one of the main goals here was to rid ourselves of 3rd party dependencies (all custom assembly dependencies really) and we wanted to do the minimum amount of work required to generated a reliable MD5 hash.

Why bulk update at the end rather than update each row via OLE DB Command?
Because the OLE DB Command isn't set based and it can block/be blocked by inserts. Also, parameter mapping to all those Param_XX parameters and counting question marks is just terrible. So, so terrible.

How do I know how many threads I need to distribute the MD5 calculation over?
First we'll usually watch the source query in the database engine to see what it's waiting on. PREEMPTIVE_OS_WAITFORSINGLEOBJECT tends to point at SQL Server waiting to send data to SSIS because it's not accepting rows fast enough. For a starting point, take your source query and see how long it takes to just insert the entire thing in to a heap - somewhere around there is what we're shooting for. Faster if you ignore many rows, slower if you update/insert many rows.

How does it perform?
It's ridiculously fast. One of our dimensions (55 columns, varying datatypes, ~5 million deep, ~4.5 million evaluated for changes 3 times a day) went from 50+ minutes per run to 5-9 minutes per run. More importantly, we were previously using the CRC32 algorithm in the Checksum transform and that wasn't producing reliable checksums so we would either update unchanged rows or even miss changes to rows - now the hashes are dead on and reproducible via HASHBYTES.

As a side note, we were able to get Checksum to keep up with sql server by spreading it across 6 threads but the lack of reliable output forced us to abandon it.

Is there a quick way to mimic the SSIS MD5 using HASHBYTES?

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: +ISNULL(CAST([\1] AS NVARCHAR(255)),N'')
6. Replace All, remove surrogate key, business key(s), leading +, and garbage at the bottom
7. Add CONVERT(VARCHAR(34), HASHBYTES('md5', at the top
8. Add ),1) AS [Hashbytes] and an appropriate FROM clause at the bottom

Now if all your columns are chars, nchars, varchars, nvarchars, ints, and decimals - you're good. If some of your columns are the more unique datatypes (datetime, money, bit, etc) - then you've got more work to do. The issue lies in the difference between how sql server converts these datatypes to NVARCHAR and how C# converts them to strings. Here are a few examples:

The dates are the biggest debacle as they're sensitive to localization settings. If you actually had to generate the hashes using HASHBYTES on a regular basis, we would recommend modifying the date formatting when concatenating columns for the hashSource variable in the script so they match sql server. For us, we only generate MD5s using HASHBYTES for troubleshooting and updating the hash using purely dim columns rather than relying on the source as some business keys no longer exist in the source.

Here's the finished tsql for the above hash:

SELECT 
 CustomerKey
,CustomerAlternateKey
,MD5
,CONVERT(VARCHAR(34), HASHBYTES('md5',
ISNULL(CAST([GeographyKey] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Title] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FirstName] AS NVARCHAR(255)),N'')
+ISNULL(CAST([MiddleName] AS NVARCHAR(255)),N'')
+ISNULL(CAST([LastName] AS NVARCHAR(255)),N'')
+ISNULL(CASE WHEN [NameStyle] = 1 THEN N'True' ELSE N'False' END,N'')
+ISNULL(
CAST(DATEPART(MONTH,[BirthDate]) AS NVARCHAR(2))
+'/'+CAST(DATEPART(DAY,[BirthDate]) AS NVARCHAR(2))
+'/'+CAST(DATEPART(YEAR,[BirthDate]) AS NVARCHAR(4))
+' '+LEFT(RIGHT(CONVERT(NVARCHAR, [BirthDate], 109), 14),8)
+' '+RIGHT(CONVERT(NVARCHAR, [BirthDate], 109), 2)
,N'')
+ISNULL(CAST([MaritalStatus] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Suffix] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Gender] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EmailAddress] AS NVARCHAR(255)),N'')
+ISNULL(CAST(CAST(YearlyIncome AS FLOAT) AS NVARCHAR(255)),N'')
+ISNULL(CAST([TotalChildren] AS NVARCHAR(255)),N'')
+ISNULL(CAST([NumberChildrenAtHome] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EnglishEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([SpanishEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FrenchEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EnglishOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([SpanishOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FrenchOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([HouseOwnerFlag] AS NVARCHAR(255)),N'')
+ISNULL(CAST([NumberCarsOwned] AS NVARCHAR(255)),N'')
+ISNULL(CAST([AddressLine1] AS NVARCHAR(255)),N'')
+ISNULL(CAST([AddressLine2] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Phone] AS NVARCHAR(255)),N'')
+ISNULL(
CAST(DATEPART(MONTH,DateFirstPurchase) AS NVARCHAR(2))
+'/'+CAST(DATEPART(DAY,DateFirstPurchase) AS NVARCHAR(2))
+'/'+CAST(DATEPART(YEAR,DateFirstPurchase) AS NVARCHAR(4))
+' '+LEFT(RIGHT(CONVERT(NVARCHAR, DateFirstPurchase, 109), 14),8)
+' '+RIGHT(CONVERT(NVARCHAR, DateFirstPurchase, 109), 2)
,N'')
+ISNULL(CAST([CommuteDistance] AS NVARCHAR(255)),N'')
),1) AS [Hashbytes]
FROM DimCustomer

And the results:

These are the queries used for this test:

Query A

SELECT 
	{Actual} ON 0
FROM [GL] 
CELL PROPERTIES VALUE

Query B

SELECT 
	{Actual} ON 0,
	NON EMPTY Hierarchize({DrilldownLevel({[Account].[Accounts].[All]},,,INCLUDE_CALC_MEMBERS)}) ON 1
FROM [GL] 
CELL PROPERTIES VALUE

Query C

SELECT 
	{Actual} ON 0,
	NON EMPTY Hierarchize({DrilldownLevel({[Date].[Fiscal].[All]},,,INCLUDE_CALC_MEMBERS)}) ON 1
FROM [GL] 
CELL PROPERTIES VALUE
GO

Results (Cache was cleared with ClearCache on the entire Database before each Series)

Series 1	Series 2	Series 3

So basically the Grand Total of this GL cube is a random number based on whoever sneaks the first query in after the cache is cleared (processing, sync, etc).

And for all of you that think the MDX script breaks everything everywhere - I did comment out my entire MDX script before running these tests.

A quick note on what that "Update FTP Connection" task is doing - it's modifying the properties of the "FTP" connection manager to the appropriate Server/Username/Password for this particular client:

1
2
3
4
5
6
7
8
9
10

        public void Main()
        {
            ConnectionManager ftp = Dts.Connections["FTP"];
            ftp.Properties["ServerName"].SetValue(ftp, (string)Dts.Variables["FTPServer"].Value);
            ftp.Properties["ServerUserName"].SetValue(ftp, (string)Dts.Variables["FTPUser"].Value);
            ftp.Properties["ServerPassword"].SetValue(ftp, (string)Dts.Variables["User::FTPPassword"].Value);
 
            Dts.TaskResult = (int)ScriptResults.Success;
 
        }

The problem: Anyone who has tried to upload multiple files to multiple FTP sites in an SSIS package very quickly ran in to FTP errors (that's what happens when you let users tell you what their FTP url and authentication is). You can handle these errors and log them via the standard OnError handler dumping the ErrorCode, ErrorDescription, and usually SourceName out to a flat file or table.

Now, when you look at the ErrorDescription for an FTP Task - you'll be disappointed. It usually states "Unable to connect to FTP server..." or something to that effect. Looking at the execution results in the progress tab, you'll see that the FTP Connection Manager threw a more detailed description of the error:

Good luck getting at that detailed error description - My first attempt here was to acquire a connection and connect via the connection manager and trap the error that came back. While that worked, the error that was trapped was HRESULT 0xC001602A, which basically means that something bad happened.

Now we move on to phase two: Retry the FTP operation via .NET FtpWebRequest and trap that error. This can either flat out replace the FTP Task as a script task or, not sure why I did it this way - I suppose I like to try and use SSIS built in tasks as much as possible - you can retry the operation in the OnError handler assuming you have the requisite information (url, username, and password) available.

Create the OnError handler (either at the package level for everything or just on the executable you care about - I chose the latter):

Then add a data flow task - inside that add a Script Component (as a source) and an appropriate destination (Here it's going to an OleDb destination):

Configure the Script Component for ReadOnly access to the requisite variables:

And configure the Output buffer:

Now go edit the actual script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Net;
using System.IO;
using System.Text;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    public override void CreateNewOutputRows()
    {
        string errorDescription = Variables.ErrorDescription;
 
        if (Variables.ErrorDescription.Contains("Unable to connect to FTP server"))
        {
            try
            {
                FtpWebRequest ftp = (FtpWebRequest)WebRequest.Create("ftp://" + Variables.FTPUrl);
                ftp.Method = WebRequestMethods.Ftp.UploadFile;
                ftp.Credentials = new System.Net.NetworkCredential(Variables.FTPUser, Variables.FTPPassword);
 
                StreamReader sourceStream = new StreamReader(Variables.FTPLocalPath);
                byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
                sourceStream.Close();
                ftp.ContentLength = fileContents.Length;
 
                Stream requestStream = ftp.GetRequestStream();
                requestStream.Write(fileContents, 0, fileContents.Length);
                requestStream.Close();
 
                FtpWebResponse response = (FtpWebResponse)ftp.GetResponse();
 
                response.Close();
 
                errorDescription = "Upload retry attempt via .NET was successful.";
            }
            catch (Exception e)
            {
                errorDescription = e.Message;
            }
        }
 
 
        ErrorSourceBuffer.AddRow();
        ErrorSourceBuffer.ErrorCode = Variables.ErrorCode.ToString();
        ErrorSourceBuffer.ErrorDescription = errorDescription;
        ErrorSourceBuffer.ErrorSource = Variables.SourceName;
        ErrorSourceBuffer.FTPUrl = Variables.FTPUrl;
        ErrorSourceBuffer.FTPUser = Variables.FTPUser;
        ErrorSourceBuffer.FTPPassword = Variables.FTPPassword;
    }
}

And bob's your uncle. Now you have meaningful text in the ErrorDescription column of the output buffer instead of some vague "something's wrong" message.

Here's a flat file destination walkthrough for the OnError handler: http://vsteamsystemcentral.com/cs21/blogs/applied_business_intelligence/archive/2007/04/08/ssis-design-pattern-custom-error-handling.aspx

"Excel freezes (or hangs) when I try to change this filter"

Most of us have seen this at one point or another and shrugged it off as a busy time or processing is going on or there are cats clogging up the tubes, etc. Tonight, I finally decided to figure out what's causing it.

When you have an OLAP cube filter in your pivot table and expand the items list to select your filter items, Excel fires a bunch of Discover commands at the cube (these are pretty light in most cases). If you don't have anything selected yet (so you just have the default member selected) Excel isn't doing much work here and the list pops right up. Here's two basic flow examples for what discover commands are fired off when you DO have an item selected:

If you have one level 1 member selected

1. MDSCHEMA_CUBES (whatever cube you're connected to)
2. MDSCHEMA_CUBES (again, same thing)
3. MDSCHEMA_MEMBERS (LEVEL_UNIQUE_NAME of the filter), no TREE_OP
4. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member) TREE_OP 4, only immediate parent
5. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent) TREE_OP 8, returns itself
6. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent) TREE_OP 1, returns all children

If you have one level 2 member selected in a hierarchy

1. MDSCHEMA_CUBES (whatever cube you're connected to)
2. MDSCHEMA_CUBES (again, same thing)
3. MDSCHEMA_MEMBERS (LEVEL_UNIQUE_NAME of the filter), no TREE_OP
4. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member) TREE_OP 4, only immediate parent
5. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent's parent) TREE_OP 4, immediate parent
6. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent's parent) TREE_OP 8, returns itself
7. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent's parent) TREE_OP 1, all children
8. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent) TREE_OP 8, returns itself
9. MDSCHEMA_MEMBERS (MEMBER_UNIQUE_NAME of the selected member's parent) TREE_OP 1, returns all children

So, that's six requests for a level 1 member and 9 requests for a level 2 member - both of these take milliseconds to complete the whole flow, so no big deal. When does this become a big deal? When you do something like click the "Select multiple items" checkbox, expand a hierarchy and UNCHECK something... like if you opened the Adventure Works 2008 cube, added Delivery Date.Date as a filter, expanded the "All Periods" level and deselected the first day:

Now click OK and expand the items box - we get a familiar popup:

It comes back after a few seconds with the expected item list, but why did it take so long? Well, based on our flows above, we have a whole bunch of level 1 members selected - 1187 of them to be exact. That means there will be 1187 requests for their parent member (TREE_OP 4) which will all return the [All Periods] member. To illustrate this, I traced that exact operation:

The profiler captured 1192 discover requests - 1187 of them were getting the parents of selected members, two were MDSCHEMA_CUBES requests, one was getting [All Periods], one was getting all the children of [All Periods], and one was getting the Unique Name of the root level. Not cool. Imagine what happens when your talking about tens of thousands of members at a particular level! Granted, that seems unlikely but understand that the logic behind this is "I don't want to see this", "hide just this member", and so forth - in some cases people have no idea what actually happened was they unchecked one of 6,000 members.

So what to do about this? Educate. To fix the immediate problem of a seemingly broken Excel pivot table - just wait for the filter to load, it'll come back eventually. In the future when working with Excel filters, always favor selecting vs. deselecting and filtering at higher levels in the hierarchy whenever possible.

To simplify the maintenance and get rid of the "black box" factor that utility apps like this tend to have, I put together a PowerShell script to do the same thing and a stored procedure to call the script. Really, it doesn't matter what you use as you're most likely using an almost identical chunk of code to get your new partition created - my argument is that using PowerShell instead of C# or SSIS reduces the cost of maintenance, improves readability, and facilitates better understanding throughout your team.

The PowerShell Script: cloneSSASPartition.ps1

param(
   [string] $ServerName, 		<# The name of the Analysis Services Instance #>
   [string] $DatabaseName, 		<# The name of the Database our Cube is in #>
   [string] $CubeName,     		<# The name of the Cube that our Measure Group is in #>
   [string] $MeasureGroupName, 	<# The name of the Measure Group that our Partition is in #>
   [string] $SourcePartitionName, 	<# The name of the Partition we want to merge in to another partition #>
   [string] $NewPartitionName, 	<# The desired name of the new partition #>
   [string] $NewPartitionQuery 	<# The Query source for the new partition (will use the same datasource as the SourcePartition #>
)
 
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.AnalysisServices") >$NULL
 
$server = New-Object Microsoft.AnalysisServices.Server
$server.Connect($ServerName)
 
$database = $server.Databases.GetByName($DatabaseName)
$cube = $database.Cubes.GetByName($CubeName)
$measureGroup = $cube.MeasureGroups.FindByName($MeasureGroupName)
$sourcePartition = $measureGroup.Partitions.GetByName($SourcePartitionName)
$sourceQueryBinding = $sourcePartition.Source
 
$newPartition = $sourcePartition.Clone()
$newPartition.ID = $NewPartitionName
$newPartition.Name = $NewPartitionName
$measureGroup.Partitions.Add($newPartition) > $NULL
$newPartition.Source = New-Object Microsoft.AnalysisServices.QueryBinding($sourceQueryBinding.DataSourceID, $NewPartitionQuery)
$newPartition.Update()
 
$server.Disconnect()

The stored procedure to call the script: [Utility_CloneSSASPartition_S01] - assumes you placed cloneSSASPartition.ps1 at the root of the C: drive. If you put it elsewhere, update the proc to reflect the actual location.

SET ANSI_NULLS ON
GO
 
SET QUOTED_IDENTIFIER ON
GO
 
/*************************************************************************************
***
*** Procedure:  		[Utility_CloneSSASPartition]
*** Purpose:		Executes PowerShell script to clone a SSAS partition
***			
***			
*** Author:		tl
*** Date Created:		2010-01-15
*** 
*** Revision History
*** Date		Author			Description
*** 2010-01-15	tl			Created
*************************************************************************************/
 
CREATE PROCEDURE [dbo].[Utility_CloneSSASPartition_S01]
(
	@ServerName AS VARCHAR(255),
	@DatabaseName AS VARCHAR(255),
	@CubeName AS VARCHAR(255),
	@MeasureGroupName AS VARCHAR(255),
	@SourcePartitionName AS VARCHAR(255),
	@NewPartitionName AS VARCHAR(255),
	@NewPartitionQuery AS VARCHAR(255)
)
AS 
 
SET NOCOUNT ON
DECLARE @cmd VARCHAR(8000)
SET @cmd = 
	'powershell.exe -Command "& {'
	+ ' C:\cloneSSASPartition.ps1'
	+ ' -ServerName:\"' + @ServerName + '\"'
	+ ' -DatabaseName:\"' + @DatabaseName + '\"'
	+ ' -CubeName:\"' + @CubeName + '\"'
	+ ' -MeasureGroupName:\"' + @MeasureGroupName + '\"'
	+ ' -SourcePartitionName:\"' + @SourcePartitionName + '\"'
	+ ' -NewPartitionName:\"' + @NewPartitionName + '\"'
	+ ' -NewPartitionQuery:\"' + @NewPartitionQuery + '\"}"'
 
PRINT @cmd
EXEC xp_cmdshell @cmd

And, finally, we execute it

EXEC [dbo].[Utility_CloneSSASPartition_S01]
	 @ServerName = 'TESTSERVER'
	,@DatabaseName = 'TestDatabase'
	,@CubeName = 'TestCube'
	,@MeasureGroupName = 'Test Measure Group'
	,@SourcePartitionName = 'Test Partition - 200912'
	,@NewPartitionName = 'Test Partition - 201001'
	,@NewPartitionQuery = 'SELECT * FROM dbo.FactTableMonth(''1/1/2010'', ''2/1/2010'')'

Again, I'm not saying there's anything "wrong" with using C# or SSIS Packages to do this sort of thing, I'm just pointing out that C# isn't always the answer - especially when someone other than you needs to maintain this thing in the future. Keep it simple.

But I digress - asking cubes questions is still pretty fun and some of the random things it returns when you point it at your own cubes can be flat out hilarious.

Here's a few questions thrown at the Adventure Works cube in the Adventure Works DW 2008 Analysis Services database

what was sales for nichole nara
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Customer].[Customer].[Nichole Nara]} ON 0 FROM [Adventure Works])
$13,295.38
what were sales for nichole nara in cy 2004
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Date].[Calendar Year].[CY 2004]} ON 0 FROM (SELECT {[Customer].[Customer].[Nichole Nara]} ON 0 FROM [Adventure Works]))
$2,419.06
what were sales for nichole nara in cy 2004 in north america
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Sales Territory].[Sales Territory Group].[North America]} ON 0 FROM (SELECT {[Date].[Calendar Year].[CY 2004]} ON 0 FROM (SELECT {[Customer].[Customer].[Nichole Nara]} ON 0 FROM [Adventure Works])))
No Results Found
what were sales for nichole nara in cy 2004 for bikes
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Product].[Category].[Bikes]} ON 0 FROM (SELECT {[Date].[Calendar Year].[CY 2004]} ON 0 FROM (SELECT {[Customer].[Customer].[Nichole Nara]} ON 0 FROM [Adventure Works])))
$2,384.07
what were sales in north america in fy 2003 for bikes
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Product].[Category].[Bikes]} ON 0 FROM (SELECT {[Date].[Fiscal Year].[FY 2003]} ON 0 FROM (SELECT {[Sales Territory].[Sales Territory Group].[North America]} ON 0 FROM [Adventure Works])))
$1,739,306.95
sales for cy 2004 in cy q1, cy q2, and cy q4 in north america and canada
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Sales Territory].[Sales Territory Group].[North America],[Customer].[Country].[Canada]} ON 0 FROM (SELECT {[Date].[Calendar Quarter of Year].[CY Q1],[Date].[Calendar Quarter of Year].[CY Q2],[Date].[Calendar Quarter of Year].[CY Q4]} ON 0 FROM (SELECT {[Date].[Calendar Year].[CY 2004]} ON 0 FROM [Adventure Works])))
I don't know - Query Failure: Members belong to different hierarchies in the  function.
sales for cy 2004 in cy q1, cy q2, and cy q4 in north america
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Sales Territory].[Sales Territory Group].[North America]} ON 0 FROM (SELECT {[Date].[Calendar Quarter of Year].[CY Q1],[Date].[Calendar Quarter of Year].[CY Q2],[Date].[Calendar Quarter of Year].[CY Q4]} ON 0 FROM (SELECT {[Date].[Calendar Year].[CY 2004]} ON 0 FROM [Adventure Works])))
$3,967,371.16
sales between cy 2003 and cy 2004 for large resellers
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT [Large Resellers] ON 0 FROM (SELECT {[Date].[Calendar Year].[CY 2003]:[Date].[Calendar Year].[CY2004]} ON 0 FROM [Adventure Works]))
$19,561,960.04
sales for top 50 customers in fy 2004 in fy q1 and fy q3
SELECT {[Measures].[Internet Sales Amount]} ON 0  FROM (SELECT {[Date].[Fiscal Quarter of Year].[FY Q1],[Date].[Fiscal Quarter of Year].[FY Q3]} ON 0 FROM (SELECT {[Date].[Fiscal Year].[FY 2004]} ON 0 FROM (SELECT [Top 50 Customers] ON 0 FROM [Adventure Works])))
$177,263.41

The above questions returned pretty much exactly what I was looking for except for the one that threw us an error about members belonging to different hierarchies. That one was because "Canada" was in [Customer].[Country] and "north america" was in [Sales Territory].[Sales Territory Group]. Yeah, it was a dumb question because canada is in north america, but it illustrates that when you don't qualify things, you can get some pretty confusing results.

How's it work?

1. Enumerate all the Attribute Values for each Attribute Hierarchy and store them as "keywords"
2. Get some input
3. Look for the aforementioned "keywords" in the input string
4. Construct an MDX query based on what "keywords" were found in the string
5. Make sure there aren't any extra (non-noise) words in the string
6. Run Query
7. Return Results

How the queries are constructed

• Attribute Values just create a subcube - it creates a new nested subcube for each list, range, or named set it finds
• It only supports one measure as written and it just selects that from whatever quagmire of subcubes it's created

There are tons of issues with this methodology when we have role playing dimensions because the attribute values will be the same for these dimensions - so only the first one enumerated will get in (there's a way to ignore certain dimensions in the app.config - ignore all but one role playing dimension). It throws away context and calls it "noise" - now that's dangerous; if you throw away the context and just look for keywords, it's pretty easy to answer a question that wasn't really asked (oops).

As far as configuring this thing goes - take a look at the app.config and throw in appropriate values for your setup. You'll have to tweak the *Pattern values to fit things you want to match or exclude in your particular cube. It's configured for the Adventure Works cube right now, but I didn't work with it much - I just flat out excluded a few dims and hierarchies that would obviously confuse it.

Also note that, as written, it needs an Analysis Services 2008 cube.

Let me know if you want to give it a try and I'll be more than happy to work with you on configuring/modifying it for your setup. Also post comments on interesting observations you have when asking cubes unqualified questions

Download the VS 2008 Project: SmartCube.zip

Also note, the project uses an interface called IODoodad for getting questions and responding - this has two public methods, GetInput() and Respond(). The original theory here is that we could make an old-school IRC bot type thing. We did manage to hook it to Yammer for its IO, respond to private messages and public messages prefixed with @cube, etc. I didn't really explain the config for YammerIO (there's a custom config section for that that needs a bunch of OAuth stuff). Again, if you're interested in getting that working, let me know and I'll post instructions. The project above is set to use ConsoleIO.

I don't plan on doing any more development here, it was just a quick PoC to see what it looked like and get some initial feedback. If anyone's interested in moving forward with it and making it smarter (the project arguably should have been titled DumbCube at this point), let me know so I can post a link to your work and fork away

• Red: Cube Processing on a 2 Minute schedule
• Green: Cube Processing on a 1 hour schedule
• Blue: Cube Processing on a 1 hour schedule
• Black: Query Activity

Most of the activity here is from SSRS subscriptions firing around 8AM


This is the report filtered to only show events that had a duration greater than 10,000 MS (10 seconds) - Red cube is a trouble maker

And this is general SSAS activity over a few hours with the nearly instant events filtered out

The following query assumes that the trace data ended up in a table:

• @StartMillisecond: INT; Minimum relative start time (millisecond value, kinda have to guess and check)
• @EndMillisecond: INT; Maximum relative start time (millisecond value, kinda have to guess and check)
• @MinDuration: INT; Minimum duration in milliseconds

;WITH [ProcessEvent] AS
(
SELECT
	'Process' AS [Event]
	,cast([TextData] AS XML).value
		('declare namespace B="http://schemas.microsoft.com/analysisservices/2003/engine";           
		(//B:DatabaseID)[1]'
		,'VARCHAR(1024)') AS [DatabaseID]
	,cast([TextData] AS XML).value
		('declare namespace B="http://schemas.microsoft.com/analysisservices/2003/engine";           
		(//B:DimensionID)[1]'
		,'VARCHAR(1024)') AS [DimensionID]
	,cast([TextData] AS XML).value
		('declare namespace B="http://schemas.microsoft.com/analysisservices/2003/engine";           
		(//B:MeasureGroupID)[1]'
		,'VARCHAR(1024)') AS [MeasureGroupId]
	,[StartTime]
	,[Duration]
FROM dbo.zTrace_MSAS_20091023_2 T
WHERE 
	T.[EventClass] = 16 
	AND T.[EventSubclass] IN (3, 12)
	AND [TextData] LIKE '%Process%'
),
[QueryEvent] AS
(
SELECT
	'Query' AS [Event]
	,[DatabaseName] AS [DatabaseID]
	,NULL AS [DimensionID]
	,NULL AS [MeasureGroupId]
	,[StartTime]
	,[Duration]
FROM dbo.zTrace_MSAS_20091023_2 T
WHERE T.[EventClass] = 10
),
[Event] AS
(
	SELECT * FROM [ProcessEvent]
	UNION
	SELECT * FROM [QueryEvent]
)
SELECT
	 ROW_NUMBER() OVER (ORDER BY [StartTime] ASC) AS [EventId]
	,DATEADD(HOUR, -5, [StartTime]) AS [StartTime]
	,DATEDIFF(MILLISECOND, (SELECT MIN([StartTime]) FROM Event), [StartTime]) [RelativeStartTime]
	,[Duration]
	,[Event]
	,[DatabaseID]
	,CASE
		WHEN [DimensionID] IS NOT NULL THEN 'Dimension'
		WHEN [MeasureGroupID] IS NOT NULL THEN 'Measure Group'
		ELSE 'None'
	 END AS [ObjectType]	,CASE
		WHEN [DimensionID] IS NOT NULL THEN [DimensionID]
		WHEN [MeasureGroupID] IS NOT NULL THEN [MeasureGroupID]
		ELSE 'None'
	 END AS [ObjectId]	
FROM [Event]
WHERE
	DATEDIFF(MILLISECOND, (SELECT MIN([StartTime]) FROM Event), [StartTime]) >= @StartMillisecond
	AND DATEDIFF(MILLISECOND, (SELECT MIN([StartTime]) FROM Event), [StartTime]) < @EndMillisecond
	AND Duration >= @MinDuration

A few notes on the report:

• [EventId] is the Series Group
• This is a Range chart type, the Bottom value was [RelativeStartTime] and the Top value was [RelativeStartTime] + [Duration]
• Coloring the series and markers uses the following expression

  =Iif(Fields!Event.Value = "Process"
  	,Iif(Fields!DatabaseID.Value = "SSASDB"
  		,"Red"
  		,Iif(Fields!DatabaseID.Value = "OtherSSASDB"
  			,"Green"
  			,Iif(Fields!DatabaseID.Value = "YetAnotherSSASDB"
  				,"Blue"
  				,"Pink"
  			)
  		)
  	)
  	,"Black"
  )

And that's about it - if you have any more questions about it just let me know. Just seemed like an interesting way to look at trace data It also wouldn't be that difficult to generate events for other things, like your ETL processing and such so you could see that on top of everything else.

In the ReportServerService log, you'll see something like this:
Microsoft.AnalysisServices.AdomdClient.AdomdErrorResponseException: Query (..., ...) Set used with the complement operator must have all members from the same level.

Original filter
Dimension: Time
Hierarchy: Calendar Date
Operator: Not In
Filter Expression: [Today]

New filter
Dimension: Time
Hierarchy: Calendar Date
Operator: MDX
Filter Expression: Except([Time].[Calendar Date].[Calendar Date].MEMBERS, [Today])

A few days ago, one of our SSIS packages that maintained a Type 1 Slowly Changing Dimension (SCD) of about 1 million rows crept up to 15 minutes of runtime. Now this doesn't sound too bad, but this is part of our hourly batches, so 15 minutes is 25% of our entire processing window. The package was using the Slowly Changing Dimension Wizard transformation - we were doing the standard OLEDB Source (which basically represented how the SCD "should" look) and then sending it to the SCD transform and letting it figure out what needed to be inserted and updated. One option was to switch to lookups instead of the SCD wizard to speed things up, maybe even some fancy checksum voodoo for the updates (see http://blog.stevienova.com/2008/11/22/ssis-slowly-changing-dimensions-with-checksum/ for an example). Then after thinking about it a little more - why are we sending a million rows down the pipeline every hour? We know only a small percentage of these are new - and another small percentage needs to be updated. Well, we can just write a quick SQL query to get us just those sets and the package would be much more efficient!

Wait a tick - why would we give the rows to SSIS if all it is going to do insert one set and update the other? Let's just do it all in T-SQL:

The following tables and dim are fictional - I just need to make up a star schema DIM to illustrate the approach

CREATE TABLE #Temp_SCDInserts ([PartId] INT)
 
-- INSERTS for new Parts
INSERT INTO #Temp_SCDInserts
SELECT   
	[PartId]
FROM 
	Part a WITH(NOLOCK)
	INNER JOIN PartType b WITH(NOLOCK) 
		ON a.PartTypeId = b.PartTypeId
	INNER JOIN PartSupplier c WITH(NOLOCK) 
		ON a.PartSupplierId = c.PartSupplierId
	INNER JOIN PartSupplierCategory d WITH(NOLOCK) 
		ON c.PartSupplierCategoryId = d.PartSupplierCategoryId
EXCEPT
SELECT
	[PartId]	
FROM
	DW.dbo.Dim_Part WITH(NOLOCK)
 
-- UPDATES
SELECT 
	 dim.[PartKeyId]
	,a.[PartName]
	,b.[PartTypeId]
	,b.[PartTypeName]
	,c.[PartSupplierId]
	,c.[PartSupplierName]
	,d.[PartSupplierCategoryId]
	,d.[PartSupplierCategoryName]
INTO #Temp_SCDUpdates
FROM DW.dbo.Dim_Part dim WITH(NOLOCK)
	INNER JOIN Part a WITH(NOLOCK)
		ON dim.PartId = a.PartId -- Business Key
	INNER JOIN PartType b WITH(NOLOCK) 
		ON a.PartTypeId = b.PartTypeId
	INNER JOIN PartSupplier c WITH(NOLOCK) 
		ON a.PartSupplierId = c.PartSupplierId
	INNER JOIN PartSupplierCategory d WITH(NOLOCK) 
		ON c.PartSupplierCategoryId = d.PartSupplierCategoryId
WHERE
	dim.PartName <> a.PartName
	OR dim.PartTypeId <> b.PartTypeId
	OR dim.PartTypeName <> b.PartTypeName
	OR dim.PartSupplierId <> c.PartSupplierId
	OR dim.PartSupplierName <> c.PartSupplierName
	OR dim.PartSupplierCategoryId <> d.PartSupplierCategoryId
	OR dim.PartSupplierCategoryName	<> d.PartSupplierCategoryName
 
-- INSERT new records
INSERT INTO [DW].[dbo].[Dim_Part]
	([PartId]
	,[PartName]
	,[PartTypeId]
	,[PartTypeName]
	,[PartSupplierId]
	,[PartSupplierName]
	,[PartSupplierCategoryId]
	,[PartSupplierCategoryName])
SELECT 
	 a.[PartId]
	,a.[PartName]
	,b.[PartTypeId]
	,b.[PartTypeName]
	,c.[PartSupplierId]
	,c.[PartSupplierName]
	,d.[PartSupplierCategoryId]
	,d.[PartSupplierCategoryName]
FROM #Temp_SCDInserts i
	INNER JOIN Part a WITH(NOLOCK)
		ON i.PartId = a.PartId -- Business Key
	INNER JOIN PartType b WITH(NOLOCK) 
		ON a.PartTypeId = b.PartTypeId
	INNER JOIN PartSupplier c WITH(NOLOCK) 
		ON a.PartSupplierId = c.PartSupplierId
	INNER JOIN PartSupplierCategory d WITH(NOLOCK) 
		ON c.PartSupplierCategoryId = d.PartSupplierCategoryId
 
-- UPDATE existing records
UPDATE [DW].[dbo].[Dim_Part]
SET
	 [PartName] = b.[PartName]
	,[PartTypeId] = b.[PartTypeId]
	,[PartTypeName] = b.[PartTypeName]
	,[PartSupplierId] = b.[PartSupplierId]
	,[PartSupplierName] = b.[PartSupplierName]
	,[PartSupplierCategoryId] = b.[PartSupplierCategoryId]
	,[PartSupplierCategoryName] = b.[PartSupplierCategoryName]
FROM 
	[DW].[dbo].[Dim_Part] AS a
	INNER JOIN #Temp_SCDUpdates AS b 
		ON a.PartKeyId = b.PartKeyId  -- Join on Business Keys or Surrogate Key
 
 
DROP TABLE #Temp_SCDInserts
DROP TABLE #Temp_SCDUpdates

The results? Our 15 minute SSIS package has been replaced with a few lines of T-SQL and it now runs in less than 90 seconds. Nice.