So the method we'll be using here is just to mimic what Sharepoint does when you check the box to "Also refresh as soon as possible" in the schedule configuration page. To accomplish this we open the profiler, connect it to whatever instance our Sharepoint PowerPivot database is hosted on and filter you TextData to "%Schedule%" or filter to just the Sharepoint PowerPivot database (SP2010_PowerPivot_Service_Application in our case) - then open up your test PowerPivot schedule configuration, check the box, click OK, wait for the schedule history to come back up and then stop the trace. Now you know you've got what you need, you just have to find it:

So here we have Sharepoint asking for information about the schedule (identified by the ItemID from DataRefresh.Items) and then passing the data it received to PersistSchedule. If you take a close look at the PersistSchedule parameters, the last one is @RunNow = 1. So clearly, this is what we're after. We went the most tedious route and manually looked up the ItemID in DataRefresh.Items:

SET NOCOUNT ON
 
DECLARE @ItemID UNIQUEIDENTIFIER = '30D39B0E-DA8B-489B-A713-3B29D208B51F'
 
DECLARE
	@ItemName [NVARCHAR](200),
	@SPSiteID [UNIQUEIDENTIFIER],
	@SPWebID [UNIQUEIDENTIFIER],	
	@LastModifiedBy [NVARCHAR](450),
	@ScheduleLastUpdatedBy [NVARCHAR](450),
	@UserIdentity [NVARCHAR](450),
	@EmailNotification [BIT], 
	@EmailList [NVARCHAR](450),
	@SecurityConfiguration [INT],
	@SSApplicationID [NVARCHAR](256),
	@ProcessAllDataSources [BIT],
	@RV [BIGINT],
	-- Schedule info
	@Enabled [BIT], 
	-- Schedule details
	@NextProcessDate [DATETIME],
	@ScheduleStartDate [DATETIME],
	@FrequencyKey [NCHAR](1),
	@RepeatFrequency [INT],
	@Sunday [BIT],
	@Monday [BIT],
	@Tuesday [BIT],
	@Wednesday [BIT],
	@Thursday [BIT],
	@Friday [BIT],
	@Saturday [BIT],
	@Weekday [BIT], 
	@WeekendDay [BIT], 
	@DAY [BIT], 
	@MonthlyPeriod [TINYINT], 
	@MonthlySpecificDay [INT], 
	@ProcessAfterBusinessHours [BIT], 
	@SpecificTime [DATETIME],
	@RunNow [BIT]
 
DECLARE @Schedule TABLE (
	[ItemID] [UNIQUEIDENTIFIER] NOT NULL,
	[Enabled] [BIT] NOT NULL,
	[RV] [BIGINT] NULL,
	[ItemName] [NVARCHAR](200) NOT NULL,
	[SPSiteID] [UNIQUEIDENTIFIER] NOT NULL,
	[SPWebID] [UNIQUEIDENTIFIER] NOT NULL,
	[UserIdentity] [NVARCHAR](450) NULL,
	[LastModifiedBy] [NVARCHAR](450) NOT NULL,
	[EmailNotification] [BIT] NOT NULL,
	[EmailList] [NVARCHAR](450) NULL,
	[SecurityConfiguration] [INT] NOT NULL,
	[SSApplicationID] [NVARCHAR](256) NULL,
	[ProcessAllDataSources] [BIT] NOT NULL,
	[ScheduleDetailID] [UNIQUEIDENTIFIER] NOT NULL,
	[ScheduleStartDate] [DATETIME] NOT NULL,
	[FrequencyKey] [NCHAR](1) NOT NULL,
	[RepeatFrequency] [INT] NOT NULL,
	[Sunday] [BIT] NOT NULL,
	[Monday] [BIT] NOT NULL,
	[Tuesday] [BIT] NOT NULL,
	[Wednesday] [BIT] NOT NULL,
	[Thursday] [BIT] NOT NULL,
	[Friday] [BIT] NOT NULL,
	[Saturday] [BIT] NOT NULL,
	[WeekDay] [BIT] NOT NULL,
	[WeekendDay] [BIT] NOT NULL,
	[DAY] [BIT] NOT NULL,
	[MonthlyPeriod] [TINYINT] NOT NULL,
	[MonthlySpecificDay] [INT] NOT NULL,
	[ProcessAfterBusinessHours] [BIT] NOT NULL,
	[SpecificTime] [DATETIME] NULL
)
 
INSERT INTO @Schedule
EXEC DataRefresh.GetSchedule @ItemID
 
SELECT 
	 @ItemName = [ItemName]
	,@SPSiteID = [SPSiteID]
	,@SPWebID = [SPWebID]
	,@LastModifiedBy = [LastModifiedBy]
	,@ScheduleLastUpdatedBy = N'DOMAIN\tlaqua'
	,@UserIdentity = [UserIdentity]
	,@EmailNotification = [EmailNotification]
	,@EmailList = [EmailList]
	,@SecurityConfiguration = [SecurityConfiguration]
	,@SSApplicationID = [SSApplicationID]
	,@ProcessAllDataSources = [ProcessAllDataSources]
	,@RV = [RV]
		-- Schedule info
	,@Enabled = [Enabled]
		-- Schedule details
	,@NextProcessDate = CAST(GETDATE() AS DATE)
	,@ScheduleStartDate = [ScheduleStartDate]
	,@FrequencyKey = [FrequencyKey]
	,@RepeatFrequency = [RepeatFrequency]
	,@Sunday = [Sunday]
	,@Monday = [Monday]
	,@Tuesday = [Tuesday]
	,@Wednesday = [Wednesday]
	,@Thursday = [Thursday]
	,@Friday = [Friday]
	,@Saturday = [Saturday]
	,@Weekday = [Weekday]
	,@WeekendDay = [WeekendDay]
	,@DAY = [DAY]
	,@MonthlyPeriod = [MonthlyPeriod]
	,@MonthlySpecificDay = [MonthlySpecificDay]
	,@ProcessAfterBusinessHours = [ProcessAfterBusinessHours]
	,@SpecificTime = [SpecificTime]
	,@RunNow = 1
FROM @Schedule
 
EXEC DataRefresh.PersistSchedule
	 @ItemID = @ItemID
	,@ItemName = @ItemName
	,@SPSiteID = @SPSiteID
	,@SPWebID = @SPWebID
	,@LastModifiedBy = @LastModifiedBy
	,@ScheduleLastUpdatedBy = @ScheduleLastUpdatedBy
	,@UserIdentity = @UserIdentity
	,@EmailNotification = @EmailNotification
	,@EmailList = @EmailList
	,@SecurityConfiguration = @SecurityConfiguration
	,@SSApplicationID = @SSApplicationID
	,@ProcessAllDataSources = @ProcessAllDataSources
	,@RV = @RV
		-- Schedule info
	,@Enabled = @Enabled
		-- Schedule details
	,@NextProcessDate = @NextProcessDate
	,@ScheduleStartDate = @ScheduleStartDate
	,@FrequencyKey = @FrequencyKey
	,@RepeatFrequency = @RepeatFrequency
	,@Sunday = @Sunday
	,@Monday = @Monday
	,@Tuesday = @Tuesday
	,@Wednesday = @Wednesday
	,@Thursday = @Thursday
	,@Friday = @Friday
	,@Saturday = @Saturday
	,@Weekday = @Weekday
	,@WeekendDay = @WeekendDay
	,@DAY = @DAY
	,@MonthlyPeriod = @MonthlyPeriod
	,@MonthlySpecificDay = @MonthlySpecificDay
	,@ProcessAfterBusinessHours = @ProcessAfterBusinessHours
	,@SpecificTime = @SpecificTime
	,@RunNow = @RunNow

The only parameters that don't come from GetSchedule are NextRunDate which we set to midnight of today and RunNow which is, of course, 1 and ScheduleLastModifiedBy (I just tossed my username in there, I assume you could use the system identity as well).

And then we set it up to refresh every 10 minutes via Agent job to test:

Happy refreshing! Let me know if you have any questions.

First, we assume that our dimension already exists (and we were using some other checksum or MERGE to maintain it). We have to add a column to store the MD5 hash:

ALTER TABLE dbo.DimCustomer ADD
	MD5 VARCHAR(34) NOT NULL DEFAULT ''

Second, we need a staging table to store updated/changed rows. Script out the current dimension as a CREATE, remove all unneeded constraints and indexes, and create a staging table as a heap:

CREATE TABLE [dbo].[Staging_DimCustomer_UpdatedRows](
	[CustomerKey] [INT] NOT NULL,
	[GeographyKey] [INT] NULL,
	[CustomerAlternateKey] [NVARCHAR](15) NOT NULL,
	[Title] [NVARCHAR](8) NULL,
	[FirstName] [NVARCHAR](50) NULL,
	[MiddleName] [NVARCHAR](50) NULL,
	[LastName] [NVARCHAR](50) NULL,
	[NameStyle] [BIT] NULL,
	[BirthDate] [DATETIME] NULL,
	[MaritalStatus] [NCHAR](1) NULL,
	[Suffix] [NVARCHAR](10) NULL,
	[Gender] [NVARCHAR](1) NULL,
	[EmailAddress] [NVARCHAR](50) NULL,
	[YearlyIncome] [MONEY] NULL,
	[TotalChildren] [TINYINT] NULL,
	[NumberChildrenAtHome] [TINYINT] NULL,
	[EnglishEducation] [NVARCHAR](40) NULL,
	[SpanishEducation] [NVARCHAR](40) NULL,
	[FrenchEducation] [NVARCHAR](40) NULL,
	[EnglishOccupation] [NVARCHAR](100) NULL,
	[SpanishOccupation] [NVARCHAR](100) NULL,
	[FrenchOccupation] [NVARCHAR](100) NULL,
	[HouseOwnerFlag] [NCHAR](1) NULL,
	[NumberCarsOwned] [TINYINT] NULL,
	[AddressLine1] [NVARCHAR](120) NULL,
	[AddressLine2] [NVARCHAR](120) NULL,
	[Phone] [NVARCHAR](20) NULL,
	[DateFirstPurchase] [DATETIME] NULL,
	[CommuteDistance] [NVARCHAR](15) NULL,
	[MD5] [VARCHAR](34) NOT NULL)

Now in to SSIS - We will be building:

1. Execute SQL Task to Truncate our Staging table(s)
2. Data Flow Task to Insert new rows and Stage updated rows
1. OLE DB Source to retrieve our source data
2. Script Component to Generate Row Numbers
3. Conditional Split to Evenly Distribute Rows
4. Script Component to Generate MD5 Hashes
5. Union All to Squish it all back together
6. Lookup to get the existing MD5 Hash (if it exists)
7. Conditional Split to separate Unchanged and Changed rows
8. RowCount Transformation
9. OLE DB Destination for Changed rows
10. OLE DB Destination for New rows
3. Execute SQL Task to Update changed rows

Completed Control Flow

Completed Data Flow

1: Execute SQL Task to Truncate our Staging table(s)

TRUNCATE TABLE Staging_DimCustomer_UpdatedRows;

2.1: OLE DB Source to retrieve our source data

Configure the Connection Manager - for the source we'll just select from the dimension minus the surrogate key and MD5 column:

You most likely won't need to exclude any columns - here we unselect the surrogate key and the MD5 column:

2.2: Script Component to Generate Row Numbers

Add one output column to Output 0 in your Row Number Script Transformation

The following code will assign the current row number and increment it for the next row through the buffer:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    private int _rowCount = 1;
 
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        Row.RowNumber = _rowCount++;
    }
}

2.3: Conditional Split to Evenly Distribute Rows

Now in the conditional split, we distribute rows across multiple outputs by using the modulo operator. Modify to suit the number of threads you need to remove any bottlenecks in the MD5 calculation (you may only need one thread):

2.4: Script Component to Generate MD5 Hashes

Add one output column to Output 0 in your Generate MD5 Script Transformation

Select all input columns or just select ones that are going to be in the hash. Since we explicitly specify columns in the script, the hash will only be generated with exactly what columns you tell it to use.

The following script will generate a string representation of a MD5 hash. We do this so that we can do direct comparisons with HASHBYTES converted output (and virtually everything likes dealing with strings better than binary). We do end up doubling the storage cost of the MD5 hash by storing it as a string, but we were fine with that:

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    private System.Security.Cryptography.MD5 _md5 =
                new System.Security.Cryptography.MD5CryptoServiceProvider();
 
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        try
        {
            string hashSource =
                (Row.GeographyKey_IsNull ? "" : Row.GeographyKey.ToString())
                + (Row.Title_IsNull ? "" : Row.Title)
                + (Row.FirstName_IsNull ? "" : Row.FirstName)
                + (Row.MiddleName_IsNull ? "" : Row.MiddleName)
                + (Row.LastName_IsNull ? "" : Row.LastName)
                + (Row.NameStyle_IsNull ? "" : Row.NameStyle.ToString())
                + (Row.BirthDate_IsNull ? "" : Row.BirthDate.ToString())
                + (Row.MaritalStatus_IsNull ? "" : Row.MaritalStatus)
                + (Row.Suffix_IsNull ? "" : Row.Suffix)
                + (Row.Gender_IsNull ? "" : Row.Gender)
                + (Row.EmailAddress_IsNull ? "" : Row.EmailAddress)
                + (Row.YearlyIncome_IsNull ? "" : Row.YearlyIncome.ToString())
                + (Row.TotalChildren_IsNull ? "" : Row.TotalChildren.ToString())
                + (Row.NumberChildrenAtHome_IsNull ? "" : Row.NumberChildrenAtHome.ToString())
                + (Row.EnglishEducation_IsNull ? "" : Row.EnglishEducation)
                + (Row.SpanishEducation_IsNull ? "" : Row.SpanishEducation)
                + (Row.FrenchEducation_IsNull ? "" : Row.FrenchEducation)
                + (Row.EnglishOccupation_IsNull ? "" : Row.EnglishOccupation)
                + (Row.SpanishOccupation_IsNull ? "" : Row.SpanishOccupation)
                + (Row.FrenchOccupation_IsNull ? "" : Row.FrenchOccupation)
                + (Row.HouseOwnerFlag_IsNull ? "" : Row.HouseOwnerFlag)
                + (Row.NumberCarsOwned_IsNull ? "" : Row.NumberCarsOwned.ToString())
                + (Row.AddressLine1_IsNull ? "" : Row.AddressLine1)
                + (Row.AddressLine2_IsNull ? "" : Row.AddressLine2)
                + (Row.Phone_IsNull ? "" : Row.Phone)
                + (Row.DateFirstPurchase_IsNull ? "" : Row.DateFirstPurchase.ToString())
                + (Row.CommuteDistance_IsNull ? "" : Row.CommuteDistance)
                ;
 
            byte[] hashBytes = _md5.ComputeHash(
                System.Text.UnicodeEncoding.Unicode.GetBytes(hashSource));
 
            System.Text.StringBuilder sb = new System.Text.StringBuilder();
 
            for (int i = 0; i < hashBytes.Length; i++)
            {
                sb.Append(hashBytes[i].ToString("X2"));
            }
 
            Row.MD5 = "0x" + sb.ToString();
        }
        catch (Exception e)
        {
            Row.MD5 = "Error";
        }
    }
}

To quickly generate the string concatenation statement:

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: + (Row.\1_IsNull ? "" : Row.\1)
6. Replace All, remove the leading +, and remove the garbage at the bottom, remove surrogate and business keys
7. Paste results in to the script
8. Add .ToString() to non-string columns and fix any column names (issues will be underlined in red)

Copy your completed script component once for each thread you'll be running. When you copy a script component, you need to actually open the script after copying it and click Edit Script before it will validate.

2.5: Union All to Squish it all back together

2.6: Lookup to get the existing MD5 Hash (if it exists)

Configure the Lookup component to redirect rows with no matches to the No Match output:

For your connection, select the surrogate key, business key(s), and MD5 hash from your existing dimension table:

Lookup based on the business key(s) and bring back the existing MD5 and surrogate key

2.7: Conditional Split to separate Unchanged and Changed rows

Name the output for rows where the ExistingMD5 is the same as the new MD5 "Ignore" and name the Default output "Update"

2.8: RowCount Transformation for Ignored Rows

For your Ignored Rows Row Count transformation, you'll have to create a variable to store the rowcount in (we don't use it, but that's just how the transformation works):

2.9: OLE DB Destination for Changed rows

We'll insert rows that need to be updated in to the previously created Staging_DimCustomer_UpdatedRows table:

2.10: OLE DB Destination for New rows

Finally, insert new rows directly in to the dimension:

3: Execute SQL Task to Update changed rows

The final step is to take those rows we staged for updating and actually perform the update:

UPDATE a SET
 [GeographyKey] = b.[GeographyKey]
,[CustomerAlternateKey] = b.[CustomerAlternateKey]
,[Title] = b.[Title]
,[FirstName] = b.[FirstName]
,[MiddleName] = b.[MiddleName]
,[LastName] = b.[LastName]
,[NameStyle] = b.[NameStyle]
,[BirthDate] = b.[BirthDate]
,[MaritalStatus] = b.[MaritalStatus]
,[Suffix] = b.[Suffix]
,[Gender] = b.[Gender]
,[EmailAddress] = b.[EmailAddress]
,[YearlyIncome] = b.[YearlyIncome]
,[TotalChildren] = b.[TotalChildren]
,[NumberChildrenAtHome] = b.[NumberChildrenAtHome]
,[EnglishEducation] = b.[EnglishEducation]
,[SpanishEducation] = b.[SpanishEducation]
,[FrenchEducation] = b.[FrenchEducation]
,[EnglishOccupation] = b.[EnglishOccupation]
,[SpanishOccupation] = b.[SpanishOccupation]
,[FrenchOccupation] = b.[FrenchOccupation]
,[HouseOwnerFlag] = b.[HouseOwnerFlag]
,[NumberCarsOwned] = b.[NumberCarsOwned]
,[AddressLine1] = b.[AddressLine1]
,[AddressLine2] = b.[AddressLine2]
,[Phone] = b.[Phone]
,[DateFirstPurchase] = b.[DateFirstPurchase]
,[CommuteDistance] = b.[CommuteDistance]
,[MD5] = b.[MD5] 
FROM DimCustomer a
	 INNER JOIN Staging_DimCustomer_UpdatedRows b
		ON a.CustomerKey = b.CustomerKey

To quickly generate the update statement:

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: ,[\1] = b.[\1]
6. Replace All, remove surrogate key and garbage at the bottom, remove leading comma
7. Add UPDATE a SET at the top
8. Add appropriate FROM clause at the bottom

And bob's your uncle.

In closing...

Why not write a reusable code block for the MD5 hash that you could cut and paste? Why hardcode columns?
Well, there's really only two ways to do it dynamically. One is to loop over each column in the InputBuffer and concatenate them. Man, that's a bunch of extra logic when we know very well what columns are, null handling is a little sketch, and proper datatype handling requires some special logic. Looping over the InputBuffer also just plain feels dirty. The second option is Reflection. Now we could approach reflection two ways - first we could iterate over the Row definition and concatenate the columns for each InputRow, but this is INCREDIBLY expensive (reflection is crazy slow). The second Reflection approach would be to construct a DynamicMethod at runtime and call that dynamic method for each row. While this could certainly be as fast as hardcoding, it's very obscure and difficult for most people to understand. Do you want code in your packages that's hard to understand?

We chose hardcode the columns because this is the absolute minimum amount of work that needs to be done to generate a reliable MD5 hash and is therefore the fastest possible solution.

Why not do that Reflection and DynamicMethod thing in a custom transformation?
Because one of the main goals here was to rid ourselves of 3rd party dependencies (all custom assembly dependencies really) and we wanted to do the minimum amount of work required to generated a reliable MD5 hash.

Why bulk update at the end rather than update each row via OLE DB Command?
Because the OLE DB Command isn't set based and it can block/be blocked by inserts. Also, parameter mapping to all those Param_XX parameters and counting question marks is just terrible. So, so terrible.

How do I know how many threads I need to distribute the MD5 calculation over?
First we'll usually watch the source query in the database engine to see what it's waiting on. PREEMPTIVE_OS_WAITFORSINGLEOBJECT tends to point at SQL Server waiting to send data to SSIS because it's not accepting rows fast enough. For a starting point, take your source query and see how long it takes to just insert the entire thing in to a heap - somewhere around there is what we're shooting for. Faster if you ignore many rows, slower if you update/insert many rows.

How does it perform?
It's ridiculously fast. One of our dimensions (55 columns, varying datatypes, ~5 million deep, ~4.5 million evaluated for changes 3 times a day) went from 50+ minutes per run to 5-9 minutes per run. More importantly, we were previously using the CRC32 algorithm in the Checksum transform and that wasn't producing reliable checksums so we would either update unchanged rows or even miss changes to rows - now the hashes are dead on and reproducible via HASHBYTES.

As a side note, we were able to get Checksum to keep up with sql server by spreading it across 6 threads but the lack of reliable output forced us to abandon it.

Is there a quick way to mimic the SSIS MD5 using HASHBYTES?

1. Script DimCustomer As SELECT to Clipboard and paste in to Notepad++
2. Search > Replace... (Ctrl+H)
3. Select Regular expression for Search Mode
4. Find: ^.*?\[(.*?)\].*$
5. Replace: +ISNULL(CAST([\1] AS NVARCHAR(255)),N'')
6. Replace All, remove surrogate key, business key(s), leading +, and garbage at the bottom
7. Add CONVERT(VARCHAR(34), HASHBYTES('md5', at the top
8. Add ),1) AS [Hashbytes] and an appropriate FROM clause at the bottom

Now if all your columns are chars, nchars, varchars, nvarchars, ints, and decimals - you're good. If some of your columns are the more unique datatypes (datetime, money, bit, etc) - then you've got more work to do. The issue lies in the difference between how sql server converts these datatypes to NVARCHAR and how C# converts them to strings. Here are a few examples:

The dates are the biggest debacle as they're sensitive to localization settings. If you actually had to generate the hashes using HASHBYTES on a regular basis, we would recommend modifying the date formatting when concatenating columns for the hashSource variable in the script so they match sql server. For us, we only generate MD5s using HASHBYTES for troubleshooting and updating the hash using purely dim columns rather than relying on the source as some business keys no longer exist in the source.

Here's the finished tsql for the above hash:

SELECT 
 CustomerKey
,CustomerAlternateKey
,MD5
,CONVERT(VARCHAR(34), HASHBYTES('md5',
ISNULL(CAST([GeographyKey] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Title] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FirstName] AS NVARCHAR(255)),N'')
+ISNULL(CAST([MiddleName] AS NVARCHAR(255)),N'')
+ISNULL(CAST([LastName] AS NVARCHAR(255)),N'')
+ISNULL(CASE WHEN [NameStyle] = 1 THEN N'True' ELSE N'False' END,N'')
+ISNULL(
CAST(DATEPART(MONTH,[BirthDate]) AS NVARCHAR(2))
+'/'+CAST(DATEPART(DAY,[BirthDate]) AS NVARCHAR(2))
+'/'+CAST(DATEPART(YEAR,[BirthDate]) AS NVARCHAR(4))
+' '+LEFT(RIGHT(CONVERT(NVARCHAR, [BirthDate], 109), 14),8)
+' '+RIGHT(CONVERT(NVARCHAR, [BirthDate], 109), 2)
,N'')
+ISNULL(CAST([MaritalStatus] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Suffix] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Gender] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EmailAddress] AS NVARCHAR(255)),N'')
+ISNULL(CAST(CAST(YearlyIncome AS FLOAT) AS NVARCHAR(255)),N'')
+ISNULL(CAST([TotalChildren] AS NVARCHAR(255)),N'')
+ISNULL(CAST([NumberChildrenAtHome] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EnglishEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([SpanishEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FrenchEducation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([EnglishOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([SpanishOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([FrenchOccupation] AS NVARCHAR(255)),N'')
+ISNULL(CAST([HouseOwnerFlag] AS NVARCHAR(255)),N'')
+ISNULL(CAST([NumberCarsOwned] AS NVARCHAR(255)),N'')
+ISNULL(CAST([AddressLine1] AS NVARCHAR(255)),N'')
+ISNULL(CAST([AddressLine2] AS NVARCHAR(255)),N'')
+ISNULL(CAST([Phone] AS NVARCHAR(255)),N'')
+ISNULL(
CAST(DATEPART(MONTH,DateFirstPurchase) AS NVARCHAR(2))
+'/'+CAST(DATEPART(DAY,DateFirstPurchase) AS NVARCHAR(2))
+'/'+CAST(DATEPART(YEAR,DateFirstPurchase) AS NVARCHAR(4))
+' '+LEFT(RIGHT(CONVERT(NVARCHAR, DateFirstPurchase, 109), 14),8)
+' '+RIGHT(CONVERT(NVARCHAR, DateFirstPurchase, 109), 2)
,N'')
+ISNULL(CAST([CommuteDistance] AS NVARCHAR(255)),N'')
),1) AS [Hashbytes]
FROM DimCustomer

And the results:

These are the queries used for this test:

Query A

SELECT 
	{Actual} ON 0
FROM [GL] 
CELL PROPERTIES VALUE

Query B

SELECT 
	{Actual} ON 0,
	NON EMPTY Hierarchize({DrilldownLevel({[Account].[Accounts].[All]},,,INCLUDE_CALC_MEMBERS)}) ON 1
FROM [GL] 
CELL PROPERTIES VALUE

Query C

SELECT 
	{Actual} ON 0,
	NON EMPTY Hierarchize({DrilldownLevel({[Date].[Fiscal].[All]},,,INCLUDE_CALC_MEMBERS)}) ON 1
FROM [GL] 
CELL PROPERTIES VALUE
GO

Results (Cache was cleared with ClearCache on the entire Database before each Series)

Series 1	Series 2	Series 3

So basically the Grand Total of this GL cube is a random number based on whoever sneaks the first query in after the cache is cleared (processing, sync, etc).

And for all of you that think the MDX script breaks everything everywhere - I did comment out my entire MDX script before running these tests.

Would you be OK with the following query as the source for a fact table?

SELECT DISTINCT *
FROM
(
  SELECT DISTINCT [c1],[c2] FROM [t1]
  UNION ALL
  SELECT DISTINCT [c1],[c2] FROM [t2]
) a

No? Then why are you OK with this?

SELECT [c1],[c2] FROM [t1]
UNION
SELECT [c1],[c2] FROM [t2]

Personally, I tend to use UNION to combine data from two different sources. For example, if I wanted to combine the sales transactions from two JDE instances - the query might look like:

SELECT [c1],[c2] FROM [JDE1].[dbo].[F42119]
UNION
SELECT [c1],[c2] FROM [JDE2].[dbo].[F42119]

Each line here in a sales detail represents a line on an order, so assuming you selected all the PK columns in your query, a UNION won't remove any rows within either set, but when combined - it's possible the same PK could exist in both systems. Now you could split hairs and say it's highly unlikely the rest of the row would be identical, but that's not the point - it's wrong to ever DISTINCT a set of transactions intended for creating a fact table. Sure, in this particular case you could add a "SourceSystem" column that would stop that, but then you still have to accept that the UNION is blocking and expensive. Just because a DISTINCT doesn't modify results doesn't mean it didn't do any work - it actually did a great deal of work. Now consider that we're usually not selecting 2 columns, we're selecting 10-20. Do you want to wait for it to figure out all the rows are distinct when you already knew that? Further, if they're not distinct you probably screwed up a JOIN somewhere.

As far as coding standards regarding UNION go, I would suggest never using UNION without the ALL clause and doing your DISTINCT elsewhere if that's actually what you intended. Does that seem silly? Yes - but I would wager you won't ever use the 2nd one outside of building Dimensions.

SELECT [c1],[c2] FROM [JDE1].[dbo].[F42119]
UNION ALL
SELECT [c1],[c2] FROM [JDE2].[dbo].[F42119]
 
--OR
 
SELECT DISTINCT * FROM
(
  SELECT [c1],[c2] FROM [JDE1].[dbo].[F42119]
  UNION ALL
  SELECT [c1],[c2] FROM [JDE2].[dbo].[F42119]
)

And don't get me started on unioning subqueries with GROUP BY's - Why don't you just throw half your processers in the garbage, light tempdb on fire and put all your data in heaps.

Connect to the database instance you want to filter the agent jobs out on
Browse to Databases > System Databases > msdb > Programmability > Stored Procedures > System Stored Procedures
Right-click on dbo.sp_help_category and select Modify...

At the top, change the definition of @where_clause to NVARCHAR(MAX)

19

  DECLARE @where_clause   NVARCHAR(MAX)

At the bottom, add in a few lines to append the @where_clause variable with a predicate that filters out the Report Server category when it's you from your workstation (so you can still see that category from another machine if you need to).

94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

  SELECT @cmd = @cmd + N'FROM msdb.dbo.syscategories '
 
  SET @where_clause += N'
  AND
	CASE
		WHEN 
			name = ''Report Server'' 
			AND (
				SELECT RTRIM(nt_username) + ''|'' + RTRIM(hostname) 
				FROM sys.sysprocesses 
				where spid = @@spid) = ''tlaqua|TIMSLAPTOP''  THEN 0
		ELSE 1
	END = 1 '
 
  -- Execute the query
  EXECUTE (@cmd + @where_clause + N'ORDER BY category_type, name')
 
  RETURN(@@ERROR) -- 0 means success
END

So, what on earth are we doing here? First, replace my nt_username with yours and replace my hostname with yours. From my less-than-exhaustive trial and error testing, it seems that when either the SSMS Jobs node is expanded or the Job Activity Monitor fetches jobs, two DB calls are made - one to fetch categories and another to fetch jobs. I tried filtering out the jobs portion originally and that yielded some errors. So I'm assuming it's trying to marry the categories and the jobs internally, and it expects there to be jobs for each category the first query returned. By not returning the Report Server category at all, the resulting merged list of jobs doesn't contain any jobs belonging to that category (logically an INNER JOIN).

Sure, this is a dirty hack, but I don't mind.

Update (2012-01-11)
Here is the predicate for hiding those jobs from ALL SSMS clients:

96
97
98
99
100
101
102
103
104
105
106

  SET @where_clause += N'
  AND
	CASE
		WHEN 
			name = ''Report Server'' 
			AND (
				SELECT program_name 
				FROM sys.sysprocesses 
				where spid = @@spid) = ''Microsoft SQL Server Management Studio''  THEN 0
		ELSE 1
	END = 1 '

You might even have yourself some logging that indicates an error like this:
[AS400 [19]] Error: ADO NET Destination has failed to acquire the connection {43E6AE37-24E8-46F6-8AB0-689DB6531167}. The connection may have been corrupted.

Assuming you are using a System DSN for your connection (this article is based on an ODBC DSN using the iSeries Access ODBC Driver to write to DB2, but it applies to any ODBC driver with 32bit and 64bit versions where a System DSN is used for the connection).

Before going on, you should understand and have checked the PackageProtectionLevel property in your package. To get any connection that requires a password to run in BIDS, you can't use DontSaveSensitive (unless you're using package configurations to enter the password or an explicit connection string). This is by far the most common reason connections fail to validate - but there's many resources out there already on that situation, so I won't go in to it.

Now that we know it SHOULD work - why doesn't it? In short, because you don't have a 64-bit version of the System DSN you selected. What sense does that make? Let me show you.

I'm sure you've all seen this before - which one do we select? I'm apparently pretty random with it as when I started this investigation, I found various different System DSNs in both places (and they didn't match). Maybe you just use the Control Panel to get to your ODBC Data Sources:

Interesting - we know there's two places to manage ODBC connections... Which one is this? It's the 64bit one. Which means if this is the only way you enter System DSNs, all you ever enter are 64bit System DSNs.

Now, on to how this happened in the first place. First, I'll create both a 64bit DSN and a 32bit DSN:

Then in SSIS, we create an ODBC connection, the dropdown lists our DSNs for us like so:

Notice that these are a mismash of both your User DSNs and 32bit System DSNs (and probably File DSNs... but who uses those anyway). No 64bit DSNs are listed. As a side note, this also presents itself when you create a "System DSN" (not knowing you're creating a 64bit specific DSN), then you go in to BIDS and can't find your shiny new DSN in the list. To get your DSN to show, you probably go back in and create a User DSN, and it will then show up. Until you try to run it on the server under a different username - which can be even more confusing.

To test if you are encountering this 32/64bit DSN issue, you can change your SSIS Solution properties to use the 32bit runtime:

Now run your package again and it should look a little more like this:

You can also take your compiled package and run it with the 32bit version of dtexec and the 64bit version. Here's what that should look like if you're having this issue:

Use the dtexec located under Program Files\Microsoft SQL Server\100\DTS\Binn

C:\>"C:\Program Files\Microsoft SQL Server\100\DTS\Binn\dtexec" /FILE "C:\Users\
tlaqua\Documents\Visual Studio 2008\Projects\AS400 ODBC Test\AS400 ODBC Test\bin
\ExportToAS400.dtsx" /CHECKPOINTING OFF  /REPORTING EWCDI
Microsoft (R) SQL Server Execute Package Utility
Version 10.50.2500.0 for 64-bit
Copyright (C) Microsoft Corporation 2010. All rights reserved.
 
Started:  12:29:04 PM
Info: 2011-11-18 12:29:04.24
   Code: 0x4004300A
   Source: Export to AS400 SSIS.Pipeline
   Description: Validation phase is beginning.
End Info
Error: 2011-11-18 12:29:05.21
   Code: 0xC0208452
   Source: Export to AS400 AS400 [19]
   Description: ADO NET Destination has failed to acquire the connection {43E6AE
37-24E8-46F6-8AB0-689DB6531167}. The connection may have been corrupted.
End Error
Error: 2011-11-18 12:29:05.21
   Code: 0xC0047017
   Source: Export to AS400 SSIS.Pipeline
   Description: component "AS400" (19) failed validation and returned error code
 0xC0208452.
End Error
Error: 2011-11-18 12:29:05.21
   Code: 0xC004700C
   Source: Export to AS400 SSIS.Pipeline
   Description: One or more component failed validation.
End Error
Error: 2011-11-18 12:29:05.21
   Code: 0xC0024107
   Source: Export to AS400
   Description: There were errors during task validation.
End Error
DTExec: The package execution returned DTSER_FAILURE (1).
Started:  12:29:04 PM
Finished: 12:29:05 PM
Elapsed:  1.029 seconds

Use the dtexec located under Program Files (x86)\Microsoft SQL Server\100\DTS\Binn

C:\>"C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\dtexec" /FILE "C:\
Users\tlaqua\Documents\Visual Studio 2008\Projects\AS400 ODBC Test\AS400 ODBC Te
st\bin\ExportToAS400.dtsx" /CHECKPOINTING OFF  /REPORTING EWCDI
Microsoft (R) SQL Server Execute Package Utility
Version 10.50.2500.0 for 32-bit
Copyright (C) Microsoft Corporation 2010. All rights reserved.
 
Started:  12:29:49 PM
Info: 2011-11-18 12:29:49.96
   Code: 0x4004300A
   Source: Export to AS400 SSIS.Pipeline
   Description: Validation phase is beginning.
End Info
Info: 2011-11-18 12:29:51.14
   Code: 0x4004300A
   Source: Export to AS400 SSIS.Pipeline
   Description: Validation phase is beginning.
End Info
Info: 2011-11-18 12:29:51.41
   Code: 0x40043006
   Source: Export to AS400 SSIS.Pipeline
   Description: Prepare for Execute phase is beginning.
End Info
Info: 2011-11-18 12:29:51.58
   Code: 0x40043007
   Source: Export to AS400 SSIS.Pipeline
   Description: Pre-Execute phase is beginning.
End Info
Info: 2011-11-18 12:29:51.94
   Code: 0x4004300C
   Source: Export to AS400 SSIS.Pipeline
   Description: Execute phase is beginning.
End Info
DataFlow: 2011-11-18 12:29:51.94
   Source: Export to AS400
   Component "AS400" (19) will receive 2 rows on input "ADO NET Destination Inpu
t" (22)
End DataFlow
DataFlow: 2011-11-18 12:29:52.11
   Source: Export to AS400
   Component "AS400" (19) was given end of rowset on input "ADO NET Destination
Input" (22)
End DataFlow
DataFlow: 2011-11-18 12:29:52.11
   Source: Export to AS400
   Component "AS400" (19) will receive 0 rows on input "ADO NET Destination Inpu
t" (22)
End DataFlow
Info: 2011-11-18 12:29:52.11
   Code: 0x40043008
   Source: Export to AS400 SSIS.Pipeline
   Description: Post Execute phase is beginning.
End Info
Info: 2011-11-18 12:29:52.14
   Code: 0x4004300B
   Source: Export to AS400 SSIS.Pipeline
   Description: "component "AS400" (19)" wrote 2 rows.
End Info
Info: 2011-11-18 12:29:52.14
   Code: 0x40043009
   Source: Export to AS400 SSIS.Pipeline
   Description: Cleanup phase is beginning.
End Info
DTExec: The package execution returned DTSER_SUCCESS (0).
Started:  12:29:49 PM
Finished: 12:29:52 PM
Elapsed:  2.574 seconds

So how do we avoid/fix this?

1. Ensure you setup the EXACT same System DSNs in both 64bit and 32bit on your development machine.
2. On your server(s), you most likely only need to setup the 64bit DSNs as all they do is execute via dtexec and it probably uses the 64bit dtexec by default.
3. If you plan on opening BIDS on your server - refer to point 1 for the server as well because if you run BIDS on it, it's a development box.
4. Avoid User DSNs. While they are kind of magic (they don't differentiate between the 64bit and 32bit administration tool - there's only ONE list of User DSNs), they are a formula for headaches on shared development machines, packages ran by the sql server agent, or packages run via scheduled task (all depending on what user context you have setup on each engine).
5. If you can't setup 64bit System DSNs and have to use 32-bit System DSNs, ensure you have your BIDS solution set to use the 32bit runtime, ensure when you run the packages using dtexec, you use the 32bit version, and when you execute the package via SQL Server Agent, configure it for 32bit execution:

   

The basic premise here is that we want our Data Collector Sets to always be running. Seems like a reasonable request, but for various reasons, Data Collector Sets just... stop. This script should be run regularly (every 5 minutes?) to ensure that the sets keep collecting data. In the event that the script finds some work to do, it will send you an email.

#======= START Config ========
$emailServer = "1.2.3.4";
$fromAddress = "you@yourcompany.com";
$toAddress = "you@yourcompany.com";
$ignorePattern = '^(RandomCollectorSet|OSTest)$';
#======== END Config =========
 
$ErrorActionPreference = "SilentlyContinue"
$serverName = Get-Content env:computername;
$emailBody = ""
 
$collectors = New-Object -COM Pla.DataCollectorSetCollection;
$collectors.GetDataCollectorSets($null, "Service\*");
$collectors._NewEnum | ?{$_.Status -ne 1 -and $_.Name -notmatch $ignorePattern} | %{
	$_.Start($true);
	if($_.Status -eq 1) {
		$emailBody += "Started $($_.Name)`n"
	} else {
		$emailBody += "Failed to start $($_.Name)`n"
	}
};
 
if($emailBody -ne "") {
  $smtp = new-object Net.Mail.SmtpClient($emailServer);
  $smtp.Send(
    "$serverName <$fromAddress>", 
    $toAddress, 
    "Started Data Collector Set(s)", 
    $emailBody);
}

If you want the script to keep any of the non-user created collectors running, modify the namespace parameter of the GetDataCollectorSets method call.

To do this, we only need one thing - a query that returns one row if the report should be sent and zero rows if the report should not be sent. Here is an example - Let's say I want to send a report out if a JDE Address Book record with an AT1 of 'DC' has been updated after a certain julian date.

First, write a query to return rows meeting that criteria:

SELECT *
FROM LIB.F0101
WHERE ABAT1 = 'DC'
	AND ABUPMJ >= 111292

That's great, but it can return multiple rows... So just select the first record (because all we care about is if we have one or more than one - exactly how many is irrelevant). We also don't care what the column values are, so just select some arbitrary value.

SELECT TOP 1 1 AS DoStuff
FROM LIB.F0101
WHERE ABAT1 = 'DC'
	AND ABUPMJ >= 111292

So now we have what we need in SQL - let me add one final note on this for those DB2 users out there. DB2 doesn't understand "TOP" so we have to use their syntax, FETCH FIRST (and likewise for MySQL, use LIMIT).

SELECT 1 AS DoStuff
FROM LIB.F0101
WHERE ABAT1 = 'DC'
	AND ABUPMJ >= 111292
FETCH FIRST 1 ROWS ONLY

Now, you're ready to create your subscription.

Go in to your report Subscriptions and click on the New Data Driven Subscription button

Enter meaningful name and select Shared Data Source

* If you don't use shared data sources - you should

Select Shared Data Source

* I really hope you don't have a shared data source called "local" - this is just an example

Enter query we developed above

Enter the usual options - all static values

Enter parameters - usually just "Use Default"

* This is one drawback to the pattern, you have to type in your parameter(s) - no dropdowns here if you're not using the default

Select your schedule type

And finally the usual timed schedule options

The schedule you specify will be both the time when the above query is executed to see if the report should be sent as well as the time the report will be sent out (assuming it should be sent out). As far as who the report is sent to, 99% of the time this should be a distribution list that can be managed elsewhere. You don't want to be wading through SSRS to update the list of subscribed users (unless you're the only subscribed user... then go ahead).

1. Execute the top (uncommented) part and identify which subscription you want to schedule.
2. Paste the SubscriptionId from that row in to the @id variable in the lower section
3. Run the entire lower (commented) section

DECLARE @ReportName VARCHAR(255) = 'Report Name'
 
SELECT
      a.PATH
      ,b.SubscriptionID 
      ,b.Description
      ,CAST(b.PARAMETERS AS XML) AS [PARAMETERS]
      ,b.LastRunTime
      ,b.LastStatus
      ,CAST(b.ExtensionSettings AS XML) AS [ExtensionSettings]
FROM ReportServer.dbo.[CATALOG] a
      INNER JOIN ReportServer.dbo.[Subscriptions] b
            ON b.Report_OID = a.ItemID
WHERE a.Name = @ReportName
 
/*
DECLARE 
       @id varchar(260) = ''
      ,@Type VARCHAR(32)                  = 'TimedSubscription'
 
EXEC [ReportServer].dbo.AddEvent @EventType=@Type, @EventData=@id
 
*/

A few things to check for right away to see if you're having the same issue we were:

• Check to see if the fuel pump runs during cranking - ours didn't
• Check for spark during cranking - we didn't have any
• Check to see if your check engine light is on (ours WASN'T)
• Check your voltage and fuel gauges with the ignition on - ours didn't register anything
• Check to see if your low fuel light is on - our was




Fancy tool time - we hooked up the Genisys EVO to the data link connecter (located under the steering wheel):


We first tried to connect to the PCM computer to view the Datastream (and check codes) - whatever it would tell us:


Aaaaand it said it didn't want to talk to us:


Then we moved on to the BUS monitor to see if the bus was up and what computers we could talk to:


Looks like the Body computer (BCM) is willing to talk to us:


Now, we see the bus is up, the PCM appears to be down - so we replaced the PCM. Unfortunately replacing the PCM did absolutely nothing for us other than lighten our wallets. At this point, we did some digging on AllDataDiy for an appropriate troubleshooting test to figure out what on earth was going on. While I can't re-publish their content due to the ToS agreement, I can tell you how we arrived at our final solution.

We started by testing the Throttle Position Sensor voltages:


Pin 3 should be above 4 volts:


Since it was under 4 volts, we unplugged the Camshaft Position Sensor (connector locations below - this is the passenger side of the engine compartment):


And checked again - still under 4 volts:


Then we unplugged the Crankshaft Position Sensor and checked voltage again:


Finally, a bad component! We replaced the Crankshaft Position Sensor and verified that all was well


Fuel Gauges now work, voltage works, and the low fuel light isn't on anymore


We started it up and checked the ECM datastream again - all systems go


Now, the troubleshooting procedure we started out with had us unplug the ECM at the beginning and then test the TPS pin 3 voltage - but these pictures are from us reproducing it after the fact and we forgot to unplug the ECM first... so we logged a code:


I mention this code just as a side note - we cleared the code with the EVO, but if you just ignore it long enough it'll go away after a few cycles.

]]> http://timlaqua.com/2011/07/troubleshooting-1998-jeep-grand-cherokee-zj-doesnt-start-cranks-no-fuel-no-spark/feed/ 2