Product:

Microsoft SQL server Azure

Issue:

How to sum the amount in one column?

Solution:

Inside SSMS enter this, to sum column [amount] all rows that have active=”y’ and date = 20221201.

SELECT SUM(amount)
FROM [DM].[table1]
where [Active] = 'Y'
and [sDate] = '20221201'

 

To list all rows from two identical tables, including duplicates enter:

SELECT column_name(s) FROM table1
UNION ALL
SELECT column_name(s) FROM table2

 

List number of rows from a table, from a selection:

select count(*)
FROM [DM].[table1] WHERE [Active] = 'Y'

 

How copy a number of rows into a new table:

SELECT *
INTO [DM].[table1]
FROM [DM].[table2]
WHERE [active] = 'N';

 

More information:

https://www.w3schools.com/sql/sql_sum.asp

https://www.w3schools.com/sql/sql_union.asp

https://www.w3schools.com/sql/sql_update.asp

Product:
Planning Analytics Workspace Version 88
PAA Agent version 2.0.88.1607  ( version found in file D:\Program Files\ibm\cognos\tm1_64\paa_agent\wlp\usr\servers\kate-agent\version.txt)

Issue:

Planning Analytics Workspace Administration page can not display information about the TM1 instances.

You need to check the PA Administration Agent log files to get more information.

The message.log files are found in folder D:\Program Files\ibm\cognos\tm1_64\paa_agent\wlp\usr\servers\kate-agent\logs on the TM1 server.

You have some TM1 instance that there name is shown in garbage character – you are probably using a character only used in your language, and not in English.

Even that you stop your TM1 instance with the none-english name, the PAW administration page does not give you access to the other TM1 instances.

Error message in log file:

[10/6/23 9:30:58:542 CEST] 00000069 SystemErr R ERROR:status.py:Exception while getting windows registry details:
Traceback (most recent call last):
File “kateagent/scripts\status.py”, line 904, in get_win_registry_detail_batch
p = subprocess.Popen([“powershell.exe”, service_cmd],
File “D:\Program Files\ibm\cognos\tm1_64\paa_agent\wlp\usr\servers\kate-agent\apps\expanded\PA_KATE_AGENT.war\WEB-INF\lib\jython-standalone-2.7.2.jar\Lib\subprocess.py”, line 892, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File “D:\Program Files\ibm\cognos\tm1_64\paa_agent\wlp\usr\servers\kate-agent\apps\expanded\PA_KATE_AGENT.war\WEB-INF\lib\jython-standalone-2.7.2.jar\Lib\subprocess.py”, line 1361, in _execute_child
args = [fileSystemDecode(arg) for arg in args]
UnicodeDecodeError: ‘utf-8’ codec can’t decode bytes in position 437-438: invalid data
[10/6/23 9:30:58:542 CEST] 00000069 com.ibm.pa.kate.agent.scheduler.ScheduledTask E Error occurred while updating the server info json file
Traceback (most recent call last):
File “kateagent/scripts\status.py”, line 1025, in <module>
configPaths = get_win_registry_detail_batch(allServers)
File “kateagent/scripts\status.py”, line 922, in get_win_registry_detail_batch
return mapped
UnboundLocalError: local variable ‘mapped’ referenced before assignment

 

Suggested Solution:

The PAA agent will run a python script status.py to check for what Tm1 instance exist and other information, on line 922 it will read the windows registry to find the started TM1 instances – in key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services – check at that location in REGEDIT.EXE program. If there is any non-english character, like in service name or “DisplayName”=”IBM Cognos TM1 Server … then this can be the issue.

The error say it found a character that it can not understand with UTF-8, and therefor have problem scanning for more information.

Inside Cognos Configuration for Planning Analytics, remove the application with the strange name, and register it again with a name that only use English characters.

It can be that you have created the TM1 instance from the command prompt and not from the Cognos Configuration program, then it can be registered with wrong codepage.

Like;

tm1sd.exe -install -n”TM1 Production Server” -z”C:\Program Files\Cognos\TM1\Custom\TM1Data\PData”

Check that the values in registry is readable, and then restart the kate-agent (IBM Planning Analytics Administration Agent)

More information:

https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/ 

https://www.ibm.com/docs/en/planning-analytics/2.0.0?topic=rtso-setting-up-tm1-server-run-as-windows-service 

https://www.ibm.com/support/pages/setting-tm1-server-run-service 

The python code to find the information;

def get_win_registry_detail_batch(service_names):
paths = []
for n in service_names:
paths.append("'HKLM:\SYSTEM\CurrentControlSet\Services\{}'".format(n))
service_cmd = "Get-ItemPropertyValue -Path {} -Name ConfigPath".format(", ".join(paths))
try:
p = subprocess.Popen(["powershell.exe", service_cmd],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
out, err = p.communicate()
vettedResults = [line.strip()
for line in out.replace('\r', '').split('\n')
if line != '' and re.match(r'^.:\\', line)]
mapped = zip(service_names, vettedResults)
if p is not None and p.poll() == None:
p.kill()
if err and out is None:
logger.error('Error while getting windows registry details: {}'.format(err))
except Exception as e:
logger.exception('Exception while getting windows registry details: {}'.format(e.message))
return mapped

 

 

Product:

Planning Analytics Workspace

Problem:

List of versions that match other parts like the server and the workspace.

Solution:

Check the web for a list of matches.

https://www.ibm.com/support/pages/node/6519826

 

 

*IBM recommends that those who use Custom Reports and Dynamic Reports upgrade to Planning Analytics for Excel 2.0.89 immediately. For more details, see this Flash Alert.

*IBM recommends that customers upgrade to Planning Analytics Local 2.0.9.18. For more details, see this Flash Alert.

The version of PAW to use with PAX is the same version number or one above or one below. For example, if you have PAX version 2.0.50 installed, then use PAW version 2.0.49, 2.0.50 or 2.0.51.

New version for Planning Analytics Workspace, Planning Analytics for Excel comes out every 15-40 days, for Planning analytics new version comes out every 3-6 months.

 

More Information:

https://exploringtm1.com/planning-analytics-version-management/ 

https://www.tm1forum.com/viewtopic.php?t=16128 

https://blog.octanesolutions.com.au/pa-paw-pax-version-conformance

Product:
Planning Analytics workspace version 88
Microsoft Windows 2019 server

Issue:

OS report that all memory is used on the Windows server.

Solution:

First check the task manager to find what application is eating the memory.

It can be the anti-virus program is using all memory. If registry or msmpeng.exe is using all page pools then that can be that (anti-virus) program who is using all memory.

Exclude the TM1 PAW folders from the anti-virus scanning and restart the windows server.

D:\Docker

D:\PAW88

C:\ProgramData\docker

C:\Program Files\docker

 

The docker program is listed as dockerd.exe in task manager.

More information:

https://www.minitool.com/news/microsoft-defender-antivirus-service-high-memory.html 

https://www.ibm.com/docs/en/SSD29G_2.0.0/com.ibm.swg.ba.cognos.tm1_prism_gs.2.0.0.doc/tm1_prism_gs.pdf 

https://exploringtm1.com/planning-analytics-workspace-scripts/ 

Product:

Microsoft SQL Azure

Issue:

Want a new column on a table, that is needed only for the SP.

Want two column to be a new one, so we do not need to join on the two columns later.

Suggested Solution:

In a separate query , that you run before your SP,  check if the column exist and if not create it with SQL like this:

IF COL_LENGTH('[Stage].[tablename]','cDate') IS NOT NULL
PRINT 'Column Exists';
ELSE
BEGIN;
ALTER TABLE [Stage].[tablename] ADD cDate AS ( Year + '-' + Month + '-01') PERSISTED;
END;

Or even this will work:

IF COL_LENGTH('[Stage].[tablename]','cDate') IS NULL
BEGIN;
ALTER TABLE [Stage].[tablename] ADD cDate AS ( Year + '-' + Month + '-01') PERSISTED;
END;

You can find out better SQL to this problem.

Should give a new column with data:  2023-09-01, if the Year contain 2023 and Month contain 09.

Then you can compare a date formatted column with your cDate column.

PERSISTED will store the value of the calculation in the table, so it does not need to be calculated when asked for later.

More Information:

https://learn.microsoft.com/en-us/sql/relational-databases/tables/specify-computed-columns-in-a-table?view=sql-server-ver16 

https://www.sqlshack.com/an-overview-of-computed-columns-in-sql-server/ 

SQL Server Computed Columns

https://database.guide/add-a-computed-column-to-an-existing-table-in-sql-server/ 

https://www.sqlservercentral.com/articles/using-computed-columns 

https://www.tsql.info/ex/sql-check-if-column-exists-in-a-table.php 

https://www.geeksforgeeks.org/how-to-check-if-a-column-exists-in-a-sql-server-table/ 

https://blog.sqlauthority.com/2017/07/29/sql-server-check-column-exists-sql-server-table/ 

https://www.sqlshack.com/sql-if-statement-introduction-and-overview/ 

Product:
Microsoft SQL Azure

Issue:

The process need more CPU, can i change the DTU on the AZURE SQL server from inside the SQL store procedure?

Solution:

Yes, use command:

ALTER DATABASE [databasename] MODIFY (EDITION = 'Standard', MAXSIZE = 250 GB, SERVICE_OBJECTIVE = 'S4');

You have to test what level is needed for your SQL query’s.

 

DTU Model is a preconfigured blend of compute, storage, and IO resources at a fixed price. This is the simpler option that is great for paying a fixed amount each month for a preconfigured bundles of resources.

DTU stands for Database Transaction Unit. DTUs give you a way to compare database performance across the service tiers offered by Azure. DTUs roughly measure performance as a combination of CPU, Memory, Reads, and Writes.

100 DTU S3 is less than one CPU core in a on-prem database.

To see current Azure level in a database:

SELECT Edition = DATABASEPROPERTYEX('databasename', 'EDITION'),
ServiceObjective = DATABASEPROPERTYEX('databasename', 'ServiceObjective'),
MaxSizeInBytes = DATABASEPROPERTYEX('databasename', 'MaxSizeInBytes');

More Information:

https://legacysupport.timextender.com/hc/en-us/articles/360026418592-What-Azure-SQL-Database-Service-Tier-and-Performance-Level-Should-I-Use- 

Azure SQL Database DTU Versus vCore

https://sqlperformance.com/2017/03/azure/what-the-heck-is-a-dtu 

https://www.fmsinc.com/microsoft-azure/sql-server/monitoring.htm

https://techcommunity.microsoft.com/t5/azure-sql-blog/real-world-azure-sql-db-unexpected-database-maximum-size-limit/ba-p/305363 

 

Product:

Microsoft SQL Azure

Issue:

What processes are running on the SQL database server?

Solution:

In SSMS enter this:

select 
r.session_id, 
s.login_name, 
c.client_net_address, 
s.host_name, 
s.program_name, 
st.text, s.status 
from sys.dm_exec_requests r 
inner join sys.dm_exec_sessions s 
on r.session_id = s.session_id 
left join sys.dm_exec_connections c 
on r.session_id = c.session_id 
outer apply sys.dm_exec_sql_text(r.sql_handle) st 
where client_net_address is not null and text is not null and s.status = 'running'

Copy the content of the text column out to notepad to see the query running in detail.

To see if it is CPU or I/O that is used most the last hour, enter in SSMS this:

SELECT
database_name = DB_NAME()
, UTC_time = end_time
, 'CPU Utilization In % of Limit' = rs.avg_cpu_percent
, 'Data IO In % of Limit' = rs.avg_data_io_percent
, 'Log Write Utilization In % of Limit' = rs.avg_log_write_percent
, 'Memory Usage In % of Limit' = rs.avg_memory_usage_percent 
, 'In-Memory OLTP Storage in % of Limit' = rs.xtp_storage_percent
, 'Concurrent Worker Threads in % of Limit' = rs.max_worker_percent
, 'Concurrent Sessions in % of Limit' = rs.max_session_percent
FROM sys.dm_db_resource_stats AS rs --past hour only
ORDER BY rs.end_time DESC;

Data IO show 100%, here is the bottleneck, maybe the SQL QUERY PLAN is using more TEMPDB than it should.

 

To see what SQL plan is in use, install, and use:

sp_blitzwho @GetLiveQueryPlan=1

Click on live-query plan, and you will get the SQL plan in used shown inside SSMS.

More Information:

https://learn.microsoft.com/en-us/azure/azure-sql/database/query-performance-insight-use?view=azuresql 

https://learn.microsoft.com/en-us/azure/azure-sql/database/monitoring-with-dmvs?view=azuresql

https://www.brentozar.com/archive/2018/07/cxconsumer-is-harmless-not-so-fast-tiger/ 

https://github.com/amachanic/sp_whoisactive/releases 

https://www.brentozar.com/first-aid/sp_blitzwho/ 

https://www.mssqltips.com/sqlservertip/4132/correct-sql-server-tempdb-spills-in-query-plans-caused-by-outdated-statistics/

Product:
Microsoft SQL 2016 server

Issue:

How create a new database from a BACPAK file in SSMS?

Solution:

Start SSMS and connect to the target SQL server.

On the database select “Import Data-Tier Application”

Click Next

Select the BACPAC file to import and click Next

Enter the name of the new database and click Next

Click Finish

Wait during the import.

When Operation Complete click Close.

You have your new database in the new server, including data.

 

More Information:

https://4sysops.com/archives/dacpac-and-bacpac-in-sql-server/ 

https://blogs.msmvps.com/deborahk/deploying-a-dacpac-with-sql-server-management-studio/ 

https://www.sqlshack.com/importing-a-bacpac-file-for-a-sql-database-using-ssms/ 

A DAC is a logical database management entity that defines all of the SQL Server objects which associates with a user’s database. A BACPAC includes the database schema as well as the data stored in the database.

Product:
Microsoft SQL server 2016 standard
Microsoft Windows 2016 server

Problem:
How load data from csv file into SQL server, where the table contains more columns than the text file?

The BULK INSERT command will fill the not used target columns with the next row of data, and you get a inconsistent table.

Solution:

Create a view of the table, where you have less columns shown, and then bulk insert to the view.

From inside SSMS script out a select of the table, and add create view to first line, remove the columns you do not want.

CREATE VIEW StorageView AS
SELECT [Organisation]
,[Version]
,[Belopp]
FROM StorageTable

Then in your BULK INSERT us the view instead, so the csv files number of columns match the (view) target tables number of columns.

BULK INSERT [StorageView]
FROM 'C:\temp\storagefile.csv'
WITH (
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\r',
FIRSTROW = 2
);

 

FIELDTERMINATOR = set the separator for the columns in the csv file
ROWTERMINATOR = set the character to skip to next row/record
FIRSTROW = tell that first line in the csv file is headers and should not be read

If you get a error like this:

Msg 4832, Level 16, State 1, Line 16
Bulk load: An unexpected end of file was encountered in the data file.
Msg 7399, Level 16, State 1, Line 16
The OLE DB provider “BULK” for linked server “(null)” reported an error. The provider did not give any information about the error.
Msg 7330, Level 16, State 2, Line 16
Cannot fetch a row from OLE DB provider “BULK” for linked server “(null)”.

Then the csv file have a blank line as the last line, edit your csv file to not contain any empty lines.

 

More Information:

https://www.sqlservertutorial.net/sql-server-administration/sql-server-bulk-insert/ 

https://www.w3schools.com/sql/sql_view.asp

https://www.w3schools.com/sql/sql_update.asp

https://www.mssqltips.com/sqlservertip/6109/bulk-insert-data-into-sql-server/

Product:
Planning Analytics 2.0.9.18
Microsoft Windows 2019 Server

Issue:

When running TI process that have worked before we get error random like this;

Error in process completion: Unable to save subset “computed subsetname here” in dimension “Version”

Possible Solution:

That the a new anti-virus software is installed, that blocks the TM1 TI process from delete old subset files in data folder.

If you have MS Defender, ask to exclude the data folder and subfolders from the scanning. E.g. d:\tm1\budget\data

The anti-virus software is running from folder C:\ProgramData\Microsoft\Windows Defender\Platform\4.18.23080.2006-0

More Information:

https://answers.microsoft.com/en-us/windows/forum/all/windows-defender-real-time-protection-service/fda3f73e-cc0a-4946-9b9d-3c05057ef90c

https://www.softeng.es/en/blog/microsoft-defender-for-endpoint-the-solution-to-protect-detect-and-respond-to-the-most-advanced-attacks/ 

https://code.cubewise.com/bedrock