Trending SQL Server Agent Job Duration by Hour

Tim — Wed, 23 Dec 2009 03:07:38 +0000

Earlier today I noticed a SQL Server Agent job taking a little longer than usual (or what I thought was longer than usual). Let's face it, we're not staring at the Job Activity monitor all day, so unless you've written a report to monitor job run times - on occasion you ask yourself "is that a normal run time for this thing?" The job I was curious about happened to be a job that runs throughout the day and should only have real work to do once or twice an hour - and it should run roughly the same amount of time on any given business day for a given hour (i.e. at 1:00 PM on any given business day, this thing should do the same amount of work).

So I came up with the following query to PIVOT the run duration on the hour the job executed:

DECLARE @JobName VARCHAR(255) = 'Agent Job Name'
 
;WITH JobHistory AS
(
  SELECT
     a.run_date
    ,a.run_time / 10000 AS [HOUR]
    ,(a.run_duration / 10000 * 60 * 60	-- Hours
     + a.run_duration % 10000 / 100 * 60	-- Minutes
     + a.run_duration % 100			-- Seconds
     ) / 60.0 AS [DurationMinutes]
  FROM
    msdb.dbo.sysjobhistory a WITH(NOLOCK)
    INNER JOIN msdb.dbo.sysjobs b WITH(NOLOCK)
    ON 
      a.[job_id] = b.[job_id]
      AND b.[name] = @JobName
      AND step_id = 0
      AND run_status = 1
)
SELECT *
FROM
  JobHistory	
  PIVOT
  ( SUM(DurationMinutes)
    FOR [HOUR] 
    IN  ([00],[01],[02],[03],[04],[05]
        ,[06],[07],[08],[09],[10],[11]
        ,[12],[13],[14],[15],[16],[17]
        ,[18],[19],[20],[21],[22],[23])
  ) AS p

UPDATE: Added AND run_status = 1

Then made a table (not sure why, but I can't resist doing the heatmap thing):

click the image to enlarge

And, of course, the chart so you can point at a picture should you have to explain something to somebody in the future:

click the image to enlarge

Estimating the Size of a Table in SQL Server 2008

Tim — Fri, 16 Oct 2009 02:35:00 +0000

I have read this (http://msdn.microsoft.com/en-us/library/ms178085.aspx) article at least 6, maybe 7 times in the past - and every time I say to myself "this is ridiculous - someone has to have written a script to do this by now" and every time, I google for hours and fail to find anything. So I finally gave up and wrote something to do it. Note, I've only verified it on 100% fixed width tables. I compared its output to a 600+ million row table and it came out somewhere around 3% higher - fine with me as I'd rather over-estimate space requirements than under-estimate.

First, we have to create a table type to hold our table column definitions. the datatype is required, length is required for anything that allows length (MAX is not handled for any types), and the worst case scenario is always used so if you know you won't be filling those VARCHAR(255)s, feel free to use a smaller, more average, number. IsKey is a BIT value that tells the proc to use those columns as part of the index key. The rest of it is all calculated.

CREATE TYPE TableColumnDefinition AS TABLE
	([DataType] VARCHAR(16) NOT NULL
	,[Length] INT
	,[IsKey] BIT
	,[DataLength] AS
		CASE 
			WHEN [DataType] = 'BIT' THEN 0.125
			WHEN [DataType] = 'TINYINT' THEN 1
			WHEN [DataType] = 'SMALLINT' THEN 2
			WHEN [DataType] IN ('INT', 'SMALLMONEY', 'REAL', 'SMALLDATETIME') THEN 4
			WHEN [DataType] IN ('BIGINT', 'DATETIME', 'MONEY') THEN 8
			WHEN [DataType] = 'UNIQUEIDENTIFIER' THEN 16
			WHEN [DataType] = 'NVARCHAR' THEN [Length] * 2 + 2
			WHEN [DataType] = 'VARCHAR' THEN [Length] + 2	-- DOESN'T SUPPORT MAX
			WHEN [Datatype] = 'DATE' THEN 3 
			WHEN [Datatype] = 'TIME' THEN 5 -- Worst case
			WHEN [Datatype] = 'DATETIME2' THEN 8 -- Worst case
			WHEN [Datatype] = 'DATETIMEOFFSET' THEN 10 -- Worst case
			WHEN [DataType] = 'CHAR' THEN [Length]
			WHEN [DataType] = 'NCHAR' THEN [Length] * 2 + 2
			WHEN [DataType] = 'BINARY' THEN [Length]
			WHEN [DataType] = 'VARBINARY' THEN [Length] + 2
			WHEN [Datatype] = 'TIMESTAMP' THEN 8
			WHEN [DataType] = 'XML' THEN [Length]	-- Estimate an average size in bytes, 2GB max
			WHEN [DataType] IN ('DECIMAL', 'NUMERIC') THEN 
				CASE 
					WHEN [Length] BETWEEN 1 AND 9	THEN 5
					WHEN [Length] BETWEEN 10 AND 19 THEN 9
					WHEN [Length] BETWEEN 20 AND 28 THEN 13
					WHEN [Length] BETWEEN 29 AND 38 THEN 17
				END
			WHEN [DataType] = 'FLOAT' THEN
				CASE
					WHEN [Length] BETWEEN 1 AND 24 THEN 4
					WHEN [Length] BETWEEN 25 AND 53 THEN 8
					WHEN [Length] IS NULL THEN 8	-- Default n is 53
				END
			ELSE 0
		END
	,[IsVariableLength] AS
		CASE 
			WHEN [DataType] IN ('VARCHAR', 'NVARCHAR', 'VARBINARY') THEN 1
			ELSE 0
		END)

Now for the proc that attempts to replicate the methodology outlined on the aforementioned MSDN article. The results appeared to be accurate or over the actual value, which was exactly what I needed, so I went with it and moved on. I'm completely open to corrections, suggestions, improvements, etc. Just make a comment and I'll integrate them (assuming they work .

/*************************************************************************************
***
*** Procedure:  		[Utility_EstimateClusteredIndexSize]
*** Purpose:		    Estimates the amount of space needed for a given clustered index
*** Methodology:		http://msdn.microsoft.com/en-us/library/ms178085.aspx
***			
***			
*** Author:		      Tim Laqua
*** Date Created:	  2009-10-15 
*** 
*** Revision History
*** Date		Author			Description
*** 2009-10-15	tl				Created
*************************************************************************************/
ALTER PROCEDURE [dbo].[Utility_EstimateClusteredIndexSize]
(
	 @Num_Rows FLOAT
	,@ClusteredIndexIsUnique BIT
	,@Fill_Factor FLOAT
	,@TableColumn TableColumnDefinition READONLY
)
AS 
 
SET NOCOUNT ON
 
DECLARE
	 @Num_Cols FLOAT = 0					-- total number of columns (fixed-length and variable-length)
	,@Fixed_Data_Size FLOAT = 0			-- total byte size of all fixed-length columns
	,@Num_Variable_Cols FLOAT	= 0			-- number of variable-length columns
	,@Max_Var_Size FLOAT	= 0				-- maximum byte size of all variable-length columns
	,@Variable_Data_Size FLOAT = 0
 
DECLARE @VAR TABLE
	([Name] VARCHAR(64)
	,[VALUE] FLOAT
	,[DisplayValue] AS
		CASE
			WHEN LOWER([Name]) LIKE '%size%' OR LOWER([Name]) LIKE '%space%' THEN
				CASE 
					WHEN [VALUE] < 1024 THEN CAST(CAST(ROUND([VALUE], 3) AS NUMERIC(16,0)) AS VARCHAR(16)) + ' B'
					WHEN [VALUE] < 1024*1024 THEN CAST(CAST(ROUND([VALUE]/1024, 3) AS NUMERIC(16,3)) AS VARCHAR(16)) + ' KB'
					WHEN [VALUE] < 1024*1024*1024 THEN CAST(CAST(ROUND([VALUE]/1024/1024, 3) AS NUMERIC(16,3)) AS VARCHAR(16)) + ' MB'
					ELSE CAST(CAST(ROUND([VALUE]/1024/1024/1024, 3) AS NUMERIC(16,3)) AS VARCHAR(16)) + ' GB'
				END
			ELSE CAST(FLOOR([VALUE]) AS VARCHAR(16))
		END)
 
 
-- Leaf level data
SELECT 
	 @Num_Variable_Cols = COUNT(1)
	,@Max_Var_Size = ISNULL(SUM([DataLength]), 0)
FROM @TableColumn WHERE [IsVariableLength] = 1
 
SELECT 
	 @Num_Cols = COUNT(1) + @Num_Variable_Cols
	,@Fixed_Data_Size = CEILING(ISNULL(SUM([DataLength]), 0))
FROM @TableColumn WHERE [IsVariableLength] = 0
 
 
INSERT INTO @VAR VALUES ('@Fixed_Data_Size', @Fixed_Data_Size)
 
IF @ClusteredIndexIsUnique <> 0
BEGIN
	SET @Num_Cols += 1
	IF @Num_Variable_Cols > 0
		SET @Num_Variable_Cols += 1
	IF @Max_Var_Size > 0
		SET @Max_Var_Size += 4
END
 
DECLARE @Null_Bitmap INT = CEILING(2 + ((@Num_Cols + 7) / 8))
INSERT INTO @VAR VALUES ('@Null_Bitmap', @Null_Bitmap)
 
-- if there are variable columns
IF @Num_Variable_Cols > 0
	SET @Variable_Data_Size = 2 + (@Num_Variable_Cols * 2) + @Max_Var_Size 
INSERT INTO @VAR VALUES ('@Num_Cols', @Num_Cols)
INSERT INTO @VAR VALUES ('@Num_Variable_Cols', @Num_Variable_Cols)
INSERT INTO @VAR VALUES ('@Max_Var_Size', @Max_Var_Size)
 
DECLARE @Row_Size FLOAT = CEILING(@Fixed_Data_Size + @Variable_Data_Size + @Null_Bitmap + 4 )
INSERT INTO @VAR VALUES ('@Row_Size', @Row_Size)
 
DECLARE @Rows_Per_Page FLOAT = FLOOR(8096 / (@Row_Size + 2))
INSERT INTO @VAR VALUES ('@Rows_Per_Page', @Rows_Per_Page)
 
DECLARE @Free_Rows_Per_Page FLOAT = FLOOR(8096 * ((100 - @Fill_Factor) / 100) / (@Row_Size + 2) )
INSERT INTO @VAR VALUES ('@Free_Rows_Per_Page', @Free_Rows_Per_Page)
 
DECLARE @Num_Leaf_Pages FLOAT = CEILING(@Num_Rows / (@Rows_Per_Page - @Free_Rows_Per_Page) )
INSERT INTO @VAR VALUES ('@Num_Leaf_Pages', @Num_Leaf_Pages)
 
DECLARE @Leaf_space_used FLOAT = 8192 * @Num_Leaf_Pages
INSERT INTO @VAR VALUES ('@Leaf_space_used', @Leaf_space_used)
 
-- Index Information
DECLARE
	 @Num_Key_Cols FLOAT = 0			-- total number of key columns (fixed-length and variable-length)
	,@Fixed_Key_Size FLOAT = 0			-- total byte size of all fixed-length key columns
	,@Num_Variable_Key_Cols FLOAT = 0	-- number of variable-length key columns
	,@Max_Var_Key_Size FLOAT = 0		-- maximum byte size of all variable-length key columns
 
 
SELECT 
	 @Num_Variable_Key_Cols = COUNT(1)
	,@Max_Var_Key_Size = ISNULL(SUM([DataLength]), 0)
FROM @TableColumn 
WHERE 
	[IsVariableLength] = 1 
	AND [IsKey] = 1
 
SELECT 
	 @Num_Key_Cols = COUNT(1) + @Num_Variable_Key_Cols
	,@Fixed_Key_Size = ISNULL(SUM([DataLength]), 0)
FROM @TableColumn 
WHERE 
	[IsVariableLength] = 0
	AND [IsKey] = 1
 
IF @ClusteredIndexIsUnique <> 0
BEGIN
	SET @Num_Key_Cols += 1
	IF @Num_Variable_Key_Cols > 0
		SET @Num_Variable_Key_Cols += 1
	IF @Max_Var_Key_Size > 0
		SET @Max_Var_Key_Size += 4
END
INSERT INTO @VAR VALUES ('@Num_Variable_Key_Cols', @Num_Variable_Key_Cols)
INSERT INTO @VAR VALUES ('@Max_Var_Key_Size', @Max_Var_Key_Size)
INSERT INTO @VAR VALUES ('@Num_Key_Cols', @Num_Key_Cols)
 
-- Who would seriously have a nullable key column?  ridiculous.
DECLARE @Index_Null_Bitmap FLOAT = 2 + ((@Num_Key_Cols + 7) / 8) 
INSERT INTO @VAR VALUES ('@Index_Null_Bitmap', @Index_Null_Bitmap)
 
DECLARE @Variable_Key_Size FLOAT = 0
 
IF @Max_Var_Key_Size > 0
	SET @Variable_Key_Size = 2 + (@Num_Variable_Key_Cols * 2) + @Max_Var_Key_Size 
INSERT INTO @VAR VALUES ('@Variable_Key_Size', @Variable_Key_Size)
 
DECLARE @Index_Row_Size FLOAT = @Fixed_Key_Size + @Variable_Key_Size + @Index_Null_Bitmap + 1 + 6
INSERT INTO @VAR VALUES ('@Index_Row_Size',@Index_Row_Size )
 
DECLARE @Index_Rows_Per_Page  FLOAT = FLOOR(8096 / (@Index_Row_Size + 2))
INSERT INTO @VAR VALUES ('@Index_Rows_Per_Page', @Index_Rows_Per_Page)
 
DECLARE @Non_Leaf_Levels FLOAT = 1
 
-- Log base @Non_Leaf_Levels hack
WHILE (@Num_Leaf_Pages / @Index_Rows_Per_Page) > POWER(@Index_Rows_Per_Page,@Non_Leaf_Levels)
	SET @Non_Leaf_Levels += 1
 
SET @Non_Leaf_Levels += 1
INSERT INTO @VAR VALUES ('@Non_Leaf_Levels', @Non_Leaf_Levels)
 
-- Summation hack
DECLARE 
	 @Num_Index_Pages FLOAT = 0
	,@Summand INT = @Non_Leaf_Levels
 
WHILE @Summand > 0
BEGIN
	SET @Num_Index_Pages += CEILING(@Num_Leaf_Pages/POWER(@Index_Rows_Per_Page, @Summand))
	SET @Summand -= 1
END
INSERT INTO @VAR VALUES ('@Num_Index_Pages', @Num_Index_Pages)
 
DECLARE @Index_Space_Used FLOAT = 8192 * @Num_Index_Pages 
INSERT INTO @VAR VALUES ('@Index_Space_Used', @Index_Space_Used)
 
DECLARE @Clustered_Index_Size FLOAT = @Leaf_Space_Used + @Index_Space_used
INSERT INTO @VAR VALUES ('@Clustered_Index_Size', @Clustered_Index_Size)
 
SELECT * FROM @VAR

Now for an example of how to run this doodad. This particular estimate was for a staging table - I needed the UNIQUEIDENTIFER as the key and the CreatedTime field had to be part of the clustered index for the partition scheme.

DECLARE @TableColumn TableColumnDefinition
 
INSERT INTO @TableColumn
VALUES 
	 ('UNIQUEIDENTIFIER', NULL, 1)
	,('DATETIME', NULL, 1)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
	,('INT', NULL, 0)
 
EXEC [dbo].[Utility_EstimateClusteredIndexSize]	
 @Num_Rows = 750000		-- number of rows in the table
,@ClusteredIndexIsUnique = 1 	-- is it?
,@Fill_Factor = 90			-- page fill factor
,@TableColumn = @TableColumn	-- definition of table columns

As for what the output looks like, I wanted it to output all the variables used so I could validate along the way with th emanual method. The results look something like this

Name                                                             Value                  DisplayValue
---------------------------------------------------------------- ---------------------- -------------------
@Fixed_Data_Size                                                 60                     60 B
@Null_Bitmap                                                     5                      5
@Num_Cols                                                        12                     12
@Num_Variable_Cols                                               0                      0
@Max_Var_Size                                                    0                      0 B
@Row_Size                                                        69                     69 B
@Rows_Per_Page                                                   114                    114
@Free_Rows_Per_Page                                              11                     11
@Num_Leaf_Pages                                                  7282                   7282
@Leaf_space_used                                                 59654144               56.891 MB
@Num_Variable_Key_Cols                                           0                      0
@Max_Var_Key_Size                                                0                      0 B
@Num_Key_Cols                                                    3                      3
@Index_Null_Bitmap                                               3.25                   3
@Variable_Key_Size                                               0                      0 B
@Index_Row_Size                                                  34.25                  34 B
@Index_Rows_Per_Page                                             223                    223
@Non_Leaf_Levels                                                 2                      2
@Num_Index_Pages                                                 34                     34
@Index_Space_Used                                                278528                 272.000 KB
@Clustered_Index_Size                                            59932672               57.156 MB

So now you may ask "but what about estimating Non-Clustered Indexes and Heaps?" Well, good question as this is the methodology for estimating the size of a Clustered Index. I ran this for a fact table with two non-clustered indexes in addition to the clustered index by basically defining the index (so pretending the index was a table for the purposes of this proc - included columns would just be non-key columns). All in all, it came out +3.31% on 600+ million rows, so I called that methodology good enough

Determining when RESTORE DATABASE command will complete (SQL Server 2008)

Tim — Fri, 25 Sep 2009 17:48:22 +0000

Ah, I see you just started restoring that 1TB monster and now everyone wants to know when it's going to be finished, where you're at in the process, etc. Fear not, Microsoft is very good at making up fictional numbers for us to use as rough estimates! I usually add 10-20% on top of these estimates just incase the database gremlins wander by to ruin your day again. Or incase you encounter "storage issues."

SELECT 
	 percent_complete AS [PercentComplete]
	,estimated_completion_time/1000.0/60.0 AS [RemainingMinutes]
	,total_elapsed_time/1000.0/60.0 AS [ElapsedMinutes]
	,(estimated_completion_time+total_elapsed_time)/1000.0/60.0 AS [TotalMinutes]
	,DATEADD(MILLISECOND, estimated_completion_time, GETDATE()) AS [EstimatedTimeOfCompletion]
	,st.TEXT AS [CommandSQL]
FROM sys.dm_exec_requests r
	cross apply sys.dm_exec_sql_text(r.sql_handle) st
WHERE command LIKE '%RESTORE DATABASE%'

Determining how long a database will be IN RECOVERY (SQL Server 2008)

Tim — Fri, 25 Sep 2009 14:16:16 +0000

So, your MSSQL service crashed in the middle of a big transaction? Or you bumped the service while it was rolling back some gigantic schema change (like say a column add on a 800 million row table)? Well, as you prepare your resume in preparation for the fallout from this debockle, you can use the following query to see how much time you have left. Or, I should say, how much time it thinks you have left... which seems to swing wildly up and down... microsoft math ftw.

 
DECLARE @DBName VARCHAR(64) = 'databasename'
 
DECLARE @ErrorLog AS TABLE([LogDate] CHAR(24), [ProcessInfo] VARCHAR(64), [TEXT] VARCHAR(MAX))
 
INSERT INTO @ErrorLog
EXEC sys.xp_readerrorlog 0, 1, 'Recovery of database', @DBName
 
SELECT TOP 5
	 [LogDate]
	,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
	,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
	,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining
	,[TEXT]
 
FROM @ErrorLog ORDER BY [LogDate] DESC

tim laqua dot com » sql server 2008

Trending SQL Server Agent Job Duration by Hour

Estimating the Size of a Table in SQL Server 2008

Determining when RESTORE DATABASE command will complete (SQL Server 2008)

Determining how long a database will be IN RECOVERY (SQL Server 2008)