Product:

Microsoft SQL server 2016

Problem:

How is my Index used?

Solution:

List the index in the database:

SELECT OBJECT_NAME(IX.OBJECT_ID) Table_Name
   ,IX.name AS Index_Name
   ,IX.type_desc Index_Type
   ,SUM(PS.[used_page_count]) * 8 IndexSizeKB
   ,IXUS.user_seeks AS NumOfSeeks
   ,IXUS.user_scans AS NumOfScans
   ,IXUS.user_lookups AS NumOfLookups
   ,IXUS.user_updates AS NumOfUpdates
   ,IXUS.last_user_seek AS LastSeek
   ,IXUS.last_user_scan AS LastScan
   ,IXUS.last_user_lookup AS LastLookup
   ,IXUS.last_user_update AS LastUpdate

 

Show if index are fragmented:

SELECT  OBJECT_NAME(IDX.OBJECT_ID) AS Table_Name,
IDX.name AS Index_Name,
IDXPS.index_type_desc AS Index_Type,
IDXPS.avg_fragmentation_in_percent  Fragmentation_Percentage
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) IDXPS
INNER JOIN sys.indexes IDX  ON IDX.object_id = IDXPS.object_id
AND IDX.index_id = IDXPS.index_id
ORDER BY Fragmentation_Percentage DESC

 

List index usage:

SELECT *
FROM
sys.dm_db_index_usage_stats
INNER JOIN sys.objects ON dm_db_index_usage_stats.OBJECT_ID = objects.OBJECT_ID
INNER JOIN sys.indexes ON indexes.index_id = dm_db_index_usage_stats.index_id AND dm_db_index_usage_stats.OBJECT_ID = indexes.OBJECT_ID
WHERE
indexes.is_primary_key = 0 --This line excludes primary key constarint
AND
indexes. is_unique = 0 --This line excludes unique key constarint
AND 
dm_db_index_usage_stats.user_updates <> 0 -- This line excludes indexes SQL Server hasn’t done any work with
AND
dm_db_index_usage_stats. user_lookups = 0
AND
dm_db_index_usage_stats.user_seeks = 0
AND
dm_db_index_usage_stats.user_scans = 0
ORDER BY
dm_db_index_usage_stats.user_updates DESC

 

 

More Information:

Gathering SQL Server indexes statistics and usage information

https://codefibershq.com/blog/useful-sqlserver-queries-commands-and-snippets

Product:

Microsoft SQL server 2016

Issue:

What happens on the SQL server now?

Solution:

Run this query:

SELECT r.start_time [Start Time],session_ID [SPID],
DB_NAME(database_id) [Database],
SUBSTRING(t.text,(r.statement_start_offset/2)+1,
CASE WHEN statement_end_offset=-1 OR statement_end_offset=0
THEN (DATALENGTH(t.Text)-r.statement_start_offset/2)+1
ELSE (r.statement_end_offset-r.statement_start_offset)/2+1
END) [Executing SQL],
Status,command,wait_type,wait_time,wait_resource,
last_wait_type
FROM sys.dm_exec_requests r
OUTER APPLY sys.dm_exec_sql_text(sql_handle) t
WHERE session_id != @@SPID -- don't show this query
AND session_id > 50 -- don't show system queries
ORDER BY r.start_time

or install and run sp_WhoIsActive from Adam Machanic (https://github.com/amachanic/sp_whoisactive/):

 

More Information:

https://www.sqlmatters.com/Articles/See%20what%20queries%20are%20currently%20running.aspx

https://www.brentozar.com/archive/2010/09/sql-server-dba-scripts-how-to-find-slow-sql-server-queries/

Product:

TM1

Issue:

How create a rule?

Solution:

Check out this from https://exploringtm1.com/3-types-of-tm1-rules-all-developers-should-know/

 

 

3 types of rules which every consultant should know:

  • Allocation/Phase/Spread Rule – e.g. Allocate/phase/spread our budgeted sales across States based on the Actual Sales ratio.
  • Rolling Value Rule – e.g. Opening (Measure) is equal to the Closing of the prior period. Often used in the Balance Sheet or Depreciation rules.
  • Averaging Rule (C Level) – e.g. Averaging Percentages or Rates up all hierarchies within the cube.

As with TM1, and Platform Software in general, there are a million ways to do anything, so don’t worry if we don’t follow the methodology you are familiar with. That being said, these 3 TM1 rules are a great guide for any developer!

Allocation/Phase/Spread Rule – How to Spread a value Across Periods

This is a common requirement often seen in budgeting rules to allocate/phase/spread an annual budget across months based on Calendar Days, Working Days or Last Year’s Actual Values for the given account.

Here is a sample rule which will phase an annual budget across months based on the number of working days in each month.

[{'Jul','Aug','Sep','Oct','Nov','Dec','Jan','Feb','Mar','Apr','May','Jun'},'Budget','$'] = N:
   IF( ['Annual'] <> 0 
     ,IF( DB('General Ledger', !Year, 'Annual', !Scenario, !Department, !Account, 'Phasing') @= 'Even Phasing' 
       ,['Annual'] \
          DB('Assumption', !Year, 'All Months', 'Actual', 'Unspecified Department', 'Working Days') *
          DB('Assumption', !Year, !Month, 'Actual', 'Unspecified Department', 'Working Days')
       , CONTINUE )
     , CONTINUE ) ;

There is a number of different ways to write this. For example, I could exclude the {} Months from my rule filter (scope) and filter using an ELISANC within an IF Statement to check that the month element of the current cell being calculated is a descendant of the ‘All Months’ Element but that would clutter the rule tracer when/if used later on the ‘Annual’ Element.


Feeder for Allocation/Spread Rule

Don’t you need a complex feeder with a rule like this? No. We only have to calculate a month if there is a value in the ‘Annual’ month (which is a posting element for annualised data) within the same year. Which means our feeder can simply be:

[‘Annual’,’Budget’,’$’] => [‘All Months’];

If you were writing this longhand, it would look like this:

[‘Annual’,’Budget’,’$’] => [‘Jan’], [‘Feb’], [‘Mar’], [‘Apr’] … [‘Dec’];

Rolling Value TM1 Rule

Calculating a Balance Sheet, Net Book Value or Depreciation? This rule logic is bound to come up. This methodology is going to be slightly different depending on how you have your Time Dimension(s) set up within your cube.

Firstly there are even more possible solutions here, but we are aiming for a sustainable example. This means we will be avoiding DIMNM(DIMIX()-1) in favour of using attributes to help move around periods.

We’ll take a customer subscription calculation as our example. We’ll assume the cube for this rule has a separate Year and Month dimension.

Measures Dimension for Rolling Rule

Given a measures dimension which looks like this:

  • Closing Subscriptions
    • Opening Subscriptions
    • New Subscribers
    • Subscriber Churn (Displayed as a positive sign, aggregated with a -1 Weighting)

Attribute used on Month Dimension

We then have a clever little attribute table on the Month Dimension:

Attributes on the Month dimension for a rolling TM1 Rule
Sample Attribute Table to assist Rolling Value Rules.

Rolling Rule

A TM1 rule can then be written which looks like this:

[{'Jul','Aug','Sep','Oct','Nov','Dec','Jan','Feb','Mar','Apr','May','Jun'},'Forecast','Opening Subscriptions'] = N: DB('Subscription'
    ,STR(NUMBR(!Year) - ATTRN('Month',!Month,'Prior Year Component'),4,0)
    ,STR(NUMBR(ATTRS('Month',!Month,'Month Number')) - ATTRN('Month',!Month,'Prior Month Component'),2,0)
    ,!Scenario,!Department,!Product,'Closing Subscriptions');

This is what I would class as a bare-bones rule for Rolling a Value. This should go back as far as the Year dimension’s elements will go and has not potential to create a circular reference like DIMNM(DIMIX()-1) methodology.

If you want to post an opening amount into the first month and first year within your TM1 cube you can use an additional check to see if the generated Year exists using the DIMIX function, if it doesn’t a STET will make the cell editable.

[{'Jul','Aug','Sep','Oct','Nov','Dec','Jan','Feb','Mar','Apr','May','Jun'},'Forecast','Opening Subscriptions'] = N:
     IF( DIMIX('Year', STR(NUMBR(!Year) - ATTRN('Month',!Month,'Prior Year Component'),4,0)) = 0 
         ,STET
         ,DB('Subscription'
            ,STR(NUMBR(!Year) - ATTRN('Month',!Month,'Prior Year Component'),4,0)
            ,STR(NUMBR(ATTRS('Month',!Month,'Month Number')) - ATTRN('Month',!Month,'Prior Month Component'),2,0)
                 ,!Scenario,!Department,!Product,'Closing Subscriptions')
     );

Feeder for Rolling Rule

Then, the feeders for this involve the same amount of coding, but the theory may be daunting for people still learning. This is because where the rule went back across time periods to get the value, the feeder has to go forwards across time periods to push the value into the rule calculation cell.

[{'Jul','Aug','Sep','Oct','Nov','Dec','Jan','Feb','Mar','Apr','May','Jun'},'Forecast','Closing Subscriptions'] =>
      DB('Subscription'
          ,STR(NUMBR(!Year) + ATTRN('Month',STR((NUMBR(ATTRS('Month',!Month,'Month Number'))-13)-1,2,0),'Prior Year Component'),4,0)
          ,STR(NUMBR(ATTRS('Month',!Month,'Month Number')) + ATTRN('Month',STR((NUMBR(ATTRS('Month',!Month,'Month Number'))-13)-1,2,0),'Prior Month Component'),2,0)
              ,!Scenario,!Department,!Product,'Opening Subscriptions');

This could be written simpler if we didn’t piggyback the same “Prior Year” Attributes and instead added new “Next Year” Attributes. The above example feeder has a minimalistic approach to attributes but is paying for it in rule complexity.

I am also using a filter of each month because I have other N level elements in my month dimension which I don’t want this rule applied to.


Averaging Rule (C Level)

C  Level (Consolidation Level) TM1 rules which do averaging are very similar to normal rules but the reason we have them listed is that people don’t realize until they have to write one that a (non-zero value) countermeasure is needed and you need to use a separate measure to perform the calculation in most cases.

Legacy Method

For averaging a value based on a counter.

['Average Price'] = C:
  ['Price'] \ ['Product Count'];
  ['Product Count'] = N:
  IF( ['Price'] > 0 , 1 , 0 );

Feeder:

['Price'] =>
  ['Product Count'],
  ['Average Price'];

I’m using price and not units for my product counter as I want to average all prices regardless of if the product is sold in a specific period. However, if I wanted an average price pro-rata units sold I would back solve my revenue equation like so.

['Average Price'] = C: ['Subscription Revenue'] \ ['Closing Subscriptions'];

Feeder:

['Closing Subscriptions'] => ['Average Price'];

New Function Method

Averaging based on the data within a measure. Here we use the ConsolidatedAvg function.

['Average Price'] = ConsolidatedAvg (2, 'Subscription', !Year, !month, !Scenario, !Department, !Product, 'Price');

Feeder:

[‘Price’] => ['Average Price'];

The first argument of the ConsolidatedAvg function can be set as either:

  • 0 – consider all cells while averaging.
  • 1 – use weightings
  • 2 – ignore blank cells (zero values) while averaging
  • 3 – use weights and ignore blanks

 

To learn more about TM1 – go to https://tm1explorers.com/webinars/

Product:

TM1

Issue:

Set security with code?

Solution:

Check out this from https://exploringtm1.com/cell-security-in-tm1-the-complete-guide/

TM1’s security can be as simple or as complex as you need. We tend to start with the broadest possible definition of security and then refine it down to the specific, to the cell if required. TM1 cell security places an overhead for your administrator to manage as it can get complex, not only within a cube but also with the interaction of it with element or dimension security. This guide will take you through how to create Cell Security the right way, that minimises the overhead on your server and administrator.


Standard “Create Cell Security Cube” Method

When you right-click on a cube and select Security, you are prompted to Create a Cell Security Cube. If you do this, it will create the cube that replicates the primary cube, but with the addition of the }Groups dimension. Let’s say you have a GL cube with Time, Version, Entity, Cost Centre, Account, and Measure. Using the automated method will give you those plus }Groups.

This is a dead-simple way to create a Cell Security cube and from it you can assign cell-level security. However, if you have, say six dimensions in the underlying cube, then you’ll have seven dimensions in the resulting cube. Great flexibility, because you can assign security to any corresponding intersection to the primary cube. Huge overhead through because you have to maintain all those intersections and if you want to go down the path of having rules manage the cell security, then it could have a big hit on performance.


Customised Cell Security Cube

So what do we do? We want a cell security cube that only has the dimensions you need to assign security for the primary cube. If, from our GL cube above, we only need Time, Version, and Account for administering security, then we create a security cube with only those plus the }Groups dimension. Administering a 4 dimension cube is very much easier than a 7 dimension cube!


Create a Custom Cell Security Cube

The only way two create a customised cell security cube is via a special TI. This TI contains just two lines, namely:

CubeName = 'General Ledger';
CellSecurityCubeCreate ( CubeName, '1:1:0:0:1:0');

This has a set of simple binary switches that enable or disable a dimension from the primary cube. So obviously, our primary cube, the General Ledger cube, here has six dimensions, they are referred to here in the exact order they are in the primary cube and are separated by a colon (one of these “:”). Finally, the zero and one switches are contained inside a single inverted comma.

Running this TI will then create a cell security cube with only the required dimensions. So, with our dimensions above, we would end up with Time, Version, Account, and }Groups in the new cube. This corresponds with the 1’s in the command in the TI.


Rules

Once the Cell Security cube has been created, then we can assign rules to it. In the rule below we have six blocks. YOU can read the annotation in the rules. Note the last one is a catch-all with the scope of []. This sets it to be for all remaining intersections not caught by the rules above.

# Set Actuals to have Write access to future Weeks only
['Actual'] = S: IF ( ELLEV ( 'Time', !Time) = 0
	,IF ( ATTRN ('Time', !Time, 'FY Week No') <= DB('System Control','Current Week','Value'), STET,'WRITE')
	,CONTINUE
	);

# Set Actuals to be Read for historic Periods (months) up until the most recent completed Month End
['Actual'] = S: IF ( ATTRS ('Time', !Time, 'Monthend Completed') @= 'Yes', STET,'WRITE');

# Set Budget to be all Read only
['Budget'] = S: 'READ';

# Set WEEKS for Active Forecast Versions to Write from Forecast Start Week onwards
[] = S: IF ( ELISANC ( 'Version', 'Active Forecast Versions', !Version ) = 1 & ELLEV ( 'Time', !Time) = 0 
	,IF ( ATTRN ('Time', !Time, 'FY Week No') < DB('System Control','Forecast Start Year-Week','Value'), STET,'WRITE')
	,CONTINUE 
	);

# Set PERIODS for Active Forecast Versions to Write from Forecast Start Week onwards
[] = S:  IF ( ELISANC ( 'Version', 'Active Forecast Versions', !Version ) = 1
	,IF ( ATTRN ('Time', !Time, 'FY Period No') < ATTRN ( 'Time' ,ATTRS ( 'Time', DB('System Control','Forecast Start Year-Week','String'), 'Current Period'), 'FY Period No'), STET,'WRITE')
	,CONTINUE 
	);

# Set ALL else to READ
[] = S: 'READ';

Deleting a Control Cube

Like other Control Objects, these cubes are special and cannot be elated by the normal right-click method. They must be deleted via a TI process as well. The TI just needs to contain the following

CubeName = '}CellSecurity_General Ledger';
CubeDestroy ( CubeName );

Obviously, if you run this, it will delete any data in the cube and any rules you have written against it. So copy the rules out first if you want to re-use them!

Product:
Microsoft SQL Server 2019
Issue:
How active encryption on SQL servers databases?

Solution:

You need a folder on the SQL server to store the certificate, create a folder like e:\key and only give local administrators and the SQL service account access there.

You can use the same certificate for a group of SQL servers. Then it is possible to restore a database backup to one of the others server in that group – that use the same certificate.

One the first SQL server:

USE Master;
GO
CREATE MASTER KEY ENCRYPTION
BY PASSWORD='InsertStrongPasswordHere12!';
GO

CREATE CERTIFICATE TDE_Cert
WITH
SUBJECT='Database_Encryption';
GO

BACKUP CERTIFICATE TDE_Cert
TO FILE = 'e:\key\TDE_Cert.cer'
WITH PRIVATE KEY (file='e:\key\TDE_CertKey.pvk',
ENCRYPTION BY PASSWORD='InsertStrongPasswordHere12!')

 

Then on every other SQL server in the group , copy above files to the e:\key folder on the next server, and do this to activate TDE:

USE Master;
GO
CREATE MASTER KEY ENCRYPTION
BY PASSWORD='InsertStrongPasswordHere12!';
GO

USE MASTER
GO
CREATE CERTIFICATE TDE_Cert
FROM FILE = 'e:\key\TDE_Cert.cer'
WITH PRIVATE KEY (FILE = 'e:\key\TDE_CertKey.pvk',
DECRYPTION BY PASSWORD = 'InsertStrongPasswordHere12!' );

 

Then to enable the encryption, you need to run this on every database:

USE <DB>
GO
CREATE DATABASE ENCRYPTION KEY
WITH ALGORITHM = AES_256
ENCRYPTION BY SERVER CERTIFICATE TDE_Cert;
GO

ALTER DATABASE <DB>
SET ENCRYPTION ON;
GO

Replace <DB> with your database name.

 

Then the database and its coming backup files are encrypted. The Backup can only be restored on a server with the same certificate.

 

If you get an error like this:

The certificate, asymmetric key, or private key file is not valid or does not exist; or you do not have permissions for it.

Try by change the path from file=‘e:\key\TDE_CertKey.pvk’ to file=’e:/key/TDE_CertKey.pvk’

 

To see what databases are encrypted:

SELECT name,is_encrypted,* FROM sys.databases WHERE is_encrypted = 1

To check if the certificate is installed:

SELECT * FROM sys.certificates WHERE name = 'TDE_Cert'

 

Important: Keep your password and backup of the certificates files in a secure location. In case you need to restore a database to a new SQL server, this keys need to be restored first.

To remove encryption from a database:

ALTER DATABASE [RecoveryWithTDE]
  SET ENCRYPTION OFF;
GO
USE [RecoveryWithTDE]
GO 
DROP DATABASE ENCRYPTION KEY;

To backup the master key:

USE Master ;
Open Master Key Decryption by password = 'InsertStrongPasswordHere12!'
Backup master key to file = 'e:\key\MasterKeyName.key'
        ENCRYPTION BY PASSWORD = 'InsertStrongPasswordHere12!';
    GO

To restore the master key to the database server:

Use master 
    restore master key
    FROM FILE = 'e:\key\MasterKeyName.key'
    DECRYPTION BY PASSWORD = 'InsertStrongPasswordHere12!'
    ENCRYPTION BY PASSWORD = 'InsertStrongPasswordHere12!'

 

The TEMPDB database will be encrypted when you start using TDE, then it is always encrypted.

Cannot encrypt a system database. Database encryption operations cannot be performed for ‘master’, ‘model’, ‘tempdb’, ‘msdb’, or ‘resource’ databases.

More Information:

https://www.databasejournal.com/ms-sql/suspending-and-resuming-transparent-data-encryption-tde/

https://www.sqlshack.com/how-to-configure-transparent-data-encryption-tde-in-sql-server/

https://www.sqlservertutorial.net/sql-server-administration/sql-server-tde/

https://learn.microsoft.com/en-us/sql/relational-databases/security/encryption/transparent-data-encryption?view=sql-server-ver16