📚 Based on 28 Portfolio Management ETL Scripts
Purpose: Loads data from various sources into QlikSense data model
Syntax:
LOAD FieldName1, FieldName2, ...
FROM source;
Project Example (from Transactions.txt, line 68):
LOAD
APPLYMAP('mAccFramework',ACCOUNTING_FRAMEWORK_KEY,'-') AS "Accounting Framework Transaction",
TEXT("Bus Trans No") AS "Bus Trans No",
"Current Value Myr",
"Current Value Qc"
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions/Transactions.qvd] (qvd);
Explanation:
When to Use:
Purpose: Creates a mapping table (key-value pair) for efficient lookups using APPLYMAP
Syntax:
[MapName]:
MAPPING LOAD KeyField, ValueField
FROM source;
Project Example (from Mappings.txt, line 12-14):
[mBusTransCode]:
MAPPING LOAD BUS_TRANS_CATEGORY_KEY, "Business Transaction Category Bus Trans Code"
RESIDENT TmpBCat;
[mBusTransCodeName]:
MAPPING LOAD BUS_TRANS_CATEGORY_KEY, "Business Transaction Category Bus Trans Code Name"
RESIDENT TmpBCat;
Explanation:
When to Use:
Best Practice from Project:
// Step 1: Load temporary table with all fields
TmpBCat:
LOAD BUS_TRANS_CATEGORY_KEY, "Field1", "Field2", "Field3"
FROM [Source.qvd] (qvd);
// Step 2: Create multiple mappings from same temp table
[mMap1]: MAPPING LOAD KEY, "Field1" RESIDENT TmpBCat;
[mMap2]: MAPPING LOAD KEY, "Field2" RESIDENT TmpBCat;
[mMap3]: MAPPING LOAD KEY, "Field3" RESIDENT TmpBCat;
// Step 3: Drop temp table to save memory
DROP TABLE TmpBCat;
Purpose: Loads data from a table already loaded into QlikSense memory
Syntax:
LOAD Field1, Field2
RESIDENT ExistingTableName;
Project Example (from Transactions.txt, line 5-7):
[Transactions]:
LOAD *
, ((MATCH([Instrument Type], 'Equity','GDR/ADR','Right','Warrant', 'Covered warrant')
OR ([Instrument Type]='Fund certificate' AND MATCH([Security Type], 'ETF', 'ETP', 'ETN', 'REIT'))
) and MATCH([Buy or Sell], 'Buy','Sell')) AS IsS2
;
LOAD *,
num([Trade Date]) as MAIN_DATE_KEY,
...
FROM [Transactions.qvd] (qvd);
Explanation:
When to Use:
Project Example (from Grouped_Total.txt):
[Instrument Name]:
LOAD DISTINCT
"Security ID",
"Instrument" AS "Instrument Name"
RESIDENT [Transactions];
CONCATENATE ([Instrument Name])
LOAD
"Security ID",
'TOTAL' AS "Instrument Name"
RESIDENT [Transactions];
Purpose: Loads data from QlikView Data files (optimized binary format)
Syntax:
LOAD Field1, Field2
FROM [FilePath.qvd] (qvd);
Project Example (from Transactions.txt, line 189-192):
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions/Transactions.qvd] (qvd)
WHERE 1=1
AND [Transaction Cancellation No] = 0
AND [Transaction Status Level No] >= 30;
Explanation:
When to Use:
Purpose: Creates small tables with hardcoded data directly in script
Syntax:
LOAD * INLINE [
Field1, Field2
Value1, Value2
Value3, Value4
];
Project Example (from Monthly_Aggregations.txt):
LEFT KEEP ([Transactions]) LOAD * INLINE
[Month Name w Total
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
DEC
TOTAL
];
Explanation:
When to Use:
Purpose: Generates rows automatically without a data source
Syntax:
LOAD Expression
AUTOGENERATE NumberOfRows;
Project Example (from Master_Calendar.txt, line 36-39):
LOAD
Date(MinDate + IterNo()) AS TempDate
WHILE MinDate + IterNo() <= MaxDate;
LOAD
min(FieldValue('Trade Date', recno()))-1 AS MinDate,
max(FieldValue('Trade Date', recno())) AS MaxDate
AUTOGENERATE FieldValueCount('Trade Date');
Explanation:
When to Use:
Purpose: Joins all rows from left table with matching rows from right table (keeps all left rows)
Syntax:
LEFT JOIN (LeftTableName)
LOAD Field1, Field2
FROM source;
Project Example (from Transactions.txt, line 239-243):
LEFT JOIN (Transactions)
[Open Payments]:
LOAD DISTINCT
[Transaction Number Ik] AS "Transaction Ik",
[Outstanding Amount Pc] AS "Outstanding Amount Pc",
[Outstanding Amount Qc] AS "Outstanding Amount Qc"
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions Open Payments/Transactions Open Payments.qvd] (qvd);
Explanation:
When to Use:
Purpose: Joins all rows from right table with matching rows from left table (keeps all right rows)
Syntax:
RIGHT JOIN (LeftTableName)
LOAD Field1, Field2
FROM source;
Project Example (from Capital_Info.txt, line 9-13):
RIGHT JOIN ([Capital Info])
LOAD
SECURITY_KEY,
MAX("Share Date Full") AS "Share Date Full"
RESIDENT [Capital Info]
WHERE "Share Date Full" <= "Trade Date"
GROUP BY SECURITY_KEY;
Explanation:
When to Use:
Purpose: Keeps only rows that exist in BOTH tables
Syntax:
INNER JOIN (LeftTableName)
LOAD Field1, Field2
FROM source;
Project Example (from PFC_at_Lodgement.txt, line 79-82):
INNER JOIN LOAD QUOTATION_CURRENCY_KEY,
[Quotation Currency] AS [Quotation Currency PFC]
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Portfolio Calculation Results/Quotation Currency.qvd] (qvd);
Explanation:
When to Use:
Purpose: Appends rows from one table to another (union)
Syntax:
CONCATENATE (TargetTable)
LOAD Field1, Field2
FROM source;
Project Example (from Grouped_Total.txt):
[Instrument Name]:
LOAD DISTINCT
"Security ID",
"Instrument" AS "Instrument Name"
RESIDENT [Transactions];
CONCATENATE ([Instrument Name])
LOAD
"Security ID",
'TOTAL' AS "Instrument Name"
RESIDENT [Transactions];
Explanation:
When to Use:
Purpose: Reduces left table to only rows that have matches in right table (association, not join)
Syntax:
LEFT KEEP (LeftTableName)
LOAD Field1, Field2
FROM source;
Project Example (from Transactions.txt, line 219-227):
LEFT KEEP (Transactions)
[Transactions Profit Loss]:
LOAD DISTINCT
APPLYMAP('mAccFramework',ACCOUNTING_FRAMEWORK_KEY,'-') AS "Accounting Framework PL",
"Booked Current Value Pc",
"Cost Value Myr",
"Transaction Ik"
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions Profit Loss/Transactions Profit Loss.qvd] (qvd)
WHERE "Accounting Framework Ik" > 0;
Explanation:
When to Use:
Purpose: Reduces right table to only rows that have matches in left table
Syntax:
RIGHT KEEP (RightTableName)
LOAD Field1, Field2
FROM source;
Project Example (from Monthly_Aggregations.txt):
LEFT KEEP ([Transactions]) LOAD * INLINE
[Month Name w Total
JAN
FEB
...
TOTAL
];
When to Use:
Purpose: Filters rows during data load based on conditions
Syntax:
LOAD Field1, Field2
FROM source
WHERE Condition;
Project Example (from Transactions.txt, line 189-192):
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions/Transactions.qvd] (qvd)
WHERE 1=1
AND [Transaction Cancellation No] = 0
AND [Transaction Status Level No] >= 30;
Explanation:
WHERE 1=1 : Always true, makes adding AND conditions easierWhen to Use:
Purpose: Aggregates data by grouping rows with same key values
Syntax:
LOAD
KeyField,
SUM(ValueField) as Total
RESIDENT SourceTable
GROUP BY KeyField;
Project Example (from Capital_Info.txt, line 9-13):
RIGHT JOIN ([Capital Info])
LOAD
SECURITY_KEY,
MAX("Share Date Full") AS "Share Date Full"
RESIDENT [Capital Info]
WHERE "Share Date Full" <= "Trade Date"
GROUP BY SECURITY_KEY;
Explanation:
When to Use:
Purpose: Sorts data during load
Syntax:
LOAD Field1, Field2
RESIDENT SourceTable
ORDER BY Field1 DESC, Field2 ASC;
Project Example (from Transactions.txt, line 325-328):
[TD2]:
LOAD
TDF AS [Trade Date Date Full],
RowNo() as RNo
RESIDENT TD
ORDER BY TDF;
Explanation:
When to Use:
Purpose: Removes duplicate rows from result
Syntax:
LOAD DISTINCT Field1, Field2
FROM source;
Project Example (from Capital_Info.txt, line 3-4):
LOAD DISTINCT SECURITY_KEY, [Trade Date]
RESIDENT [Transactions];
Explanation:
When to Use:
Purpose: Converts text to uppercase or lowercase
Syntax:
UPPER(TextField)
LOWER(TextField)
Project Example (from Transactions.txt, line 110-111):
UPPER(APPLYMAP('mSecId',SECURITY_KEY,'-')) AS "Security ID",
UPPER(APPLYMAP('mSecName',SECURITY_KEY,'-')) AS "Security Name",
When to Use:
Purpose: Converts any value to text/string format
Syntax:
TEXT(AnyValue)
Project Example (from Corporate_Actions.txt, line 3-4):
LEFT JOIN ([Transactions]) LOAD DISTINCT
TEXT("Bus Trans No") AS "Bus Trans No",
'Yes' AS "Corporate Action Flag",
Explanation:
When to Use:
Purpose: Replaces all occurrences of a substring with another string
Syntax:
REPLACE(SourceText, SearchString, ReplaceString)
Project Example (from Sheet_01.txt, line 59):
LET xSHEET01_Entitlement = REPLACE('Sum({$(xFILTER_SHEET01)} [Nominal Basis])','@','$');
Explanation:
When to Use:
Purpose: Extracts portion of string from left, right, or middle
Syntax:
LEFT(Text, NumberOfChars)
RIGHT(Text, NumberOfChars)
MID(Text, StartPosition, NumberOfChars)
Project Example (from Transactions.txt, line 152):
UPPER(LEFT(MONTHNAME(DATE("Trade Date Full",'$(DateFormat)')),3)) AS "Trade Date Month Name Short",
Explanation:
When to Use:
Purpose: Concatenates values from multiple rows into single delimited string
Syntax:
CONCAT(FieldName, Delimiter)
Project Example (from PFC_at_Lodgement.txt, line 18-22):
[LastPFC]:
LOAD CONCAT(PORTFOLIO_CALCULATION_KEY & '_' & NUM([Valuation Date Full]) & '-' & NUM([Calculation Date Full]),
Chr(39) & ',' & Chr(39)) as PFCDList
RESIDENT mLastPFC;
LET vPFCDList = Chr(39) & peek('PFCDList') & Chr(39);
Explanation:
When to Use:
Purpose: Returns first non-NULL value from list of fields
Syntax:
COALESCE(Value1, Value2, Value3, ...)
Project Example (from Transactions.txt, line 96):
COALESCE([Free Comment],'') AS "Free Comment",
Explanation:
When to Use:
Purpose: Returns character corresponding to Unicode/ASCII value
Syntax:
CHR(Number)
Project Example (from PFC_at_Lodgement.txt, line 19):
LOAD CONCAT(..., Chr(39) & ',' & Chr(39)) as PFCDList
Explanation:
Common CHR Codes:
Chr(9) → Tab
Chr(10) → Line feed
Chr(13) → Carriage return
Chr(39) → Single quote
Chr(44) → Comma
When to Use:
Purpose: Formats numeric date value as text date
Syntax:
DATE(NumericDate, FormatString)
Project Example (from Transactions.txt, line 135):
DATE("Trade Date Full",'$(DateFormat)') AS "Trade Date",
Format Codes:
YYYY → 4-digit year (2024)
MM → 2-digit month (01-12)
DD → 2-digit day (01-31)
When to Use:
Purpose: Extracts year, month, or day from date
Syntax:
YEAR(DateValue)
MONTH(DateValue)
DAY(DateValue)
Project Example (from Master_Calendar.txt):
Year(TempDate) AS Year,
Month(TempDate) AS MonthName,
Day(TempDate) AS Day,
When to Use:
Purpose: Returns day of week or week number
Syntax:
WEEKDAY(DateValue)
WEEK(DateValue)
Project Example (from Transactions.txt):
IF(WEEKDAY("Trade Date Full") >= 4,
DATE("Trade Date Full"+3-(WEEKDAY("Trade Date Full")-4),'$(DateFormat)'),
DATE("Trade Date Full"+1,'$(DateFormat)')) AS "Trade Date TD+1BD",
Explanation:
When to Use:
Purpose: Returns current date or date-time
Syntax:
TODAY()
NOW()
Project Example (from Main.txt):
LET Var_Today = Today();
When to Use:
Purpose: Returns month and year as formatted text
Syntax:
MONTHNAME(DateValue)
Project Example (from Transactions.txt):
MONTHNAME(DATE("Trade Date Full",'$(DateFormat)'))
When to Use:
Purpose: Returns first or last day of year
Syntax:
YEARSTART(DateValue)
YEAREND(DateValue)
When to Use:
Purpose: Returns first or last day of month
Syntax:
MONTHSTART(DateValue)
MONTHEND(DateValue)
When to Use:
Purpose: Returns first or last day of week
Syntax:
WEEKSTART(DateValue)
WEEKEND(DateValue)
When to Use:
Purpose: Adds specified number of years or months to date
Syntax:
ADDYEARS(DateValue, NumberOfYears)
ADDMONTHS(DateValue, NumberOfMonths)
When to Use:
Purpose: Formats number with specified format pattern
Syntax:
NUM(NumericValue, FormatPattern)
Project Example (from Capital_Info.txt):
NUM("Nominal Per Share",'#,##0.000000') AS "Nominal Per Share"
Format Patterns:
#,##0 → 1,234
#,##0.00 → 1,234.56
#,##0.000000 → 1,234.567890
When to Use:
Purpose: Rounds down (FLOOR) or up (CEIL) to nearest integer
Syntax:
FLOOR(NumericValue)
CEIL(NumericValue)
Project Example (from Dividend.txt):
Floor("Ex Dividend Date Full") AS ExDivDateNum,
Explanation:
When to Use:
Purpose: Returns absolute value (removes negative sign)
Syntax:
FABS(NumericValue)
Project Example (from Sheet_05.txt):
FABS(ALT(Sum({$(xFILTER_SHEET05)} ...) / Sum({$(xFILTER_SHEET05)} [Nominal]),0))
When to Use:
Purpose: Returns one value if condition is true, another if false
Syntax:
IF(Condition, TrueValue, FalseValue)
Project Example (from Transactions.txt):
IF("Bus Trans Code"<>'BalBook' AND [Current Value Qc]>0, 'Sell',
IF(("Bus Trans Code"<>'BalBook' AND [Current Value Qc]<0), 'Buy')
) AS "Buy or Sell",
When to Use:
Purpose: Returns first non-NULL value from list of expressions
Syntax:
ALT(Expression1, Expression2, Expression3, ...)
Project Example (from Transactions.txt):
IF(ALT([Dividend Reinvestment Amount Pc],0) = 0,
[Payment Amount Pc],
ALT([Interest Dividend Capital Event Pc],0) - ALT([Dividend Reinvestment Amount Pc],0)
) AS "Net Cash Received Pc",
When to Use:
Purpose: Checks for NULL values
Syntax:
NULL()
ISNULL(Expression)
Project Example (from Mappings.txt):
MAPPING LOAD SECURITY_KEY, "Security Dividend Registrar"
RESIDENT TmpSec
WHERE NOT IsNull("Security Dividend Registrar");
When to Use:
Purpose: Treats empty strings as NULL
Syntax:
EMPTYISNULL(Expression)
Project Example (from PFC_at_Lodgement.txt):
WHERE NOT ISNULL(EmptyIsNull([CA Record Date]))
When to Use:
Purpose: Looks up value from mapping table
Syntax:
APPLYMAP('MappingTableName', LookupKey, DefaultValue)
Project Example (from Transactions.txt):
APPLYMAP('mAccFramework',ACCOUNTING_FRAMEWORK_KEY,'-') AS "Accounting Framework Transaction",
APPLYMAP('mBroker',SECURITY_KEY,'-') AS "Broker",
Full Pattern:
// Create mapping
[mBroker]:
MAPPING LOAD SECURITY_KEY, BrokerName FROM Brokers.qvd (qvd);
// Use mapping
APPLYMAP('mBroker', SECURITY_KEY, 'Unknown') AS "Broker"
When to Use:
Purpose: Checks if value exists in list (returns position or 0)
Syntax:
MATCH(Expression, Value1, Value2, Value3, ...)
Project Example (from Transactions.txt):
MATCH([Instrument Type], 'Equity','GDR/ADR','Right','Warrant', 'Covered warrant')
When to Use:
Purpose: Checks if value matches wildcard pattern
Syntax:
WILDMATCH(Expression, Pattern1, Pattern2, ...)
Wildcards:
* = any characters? = single characterProject Example (from Transactions.txt):
IF(WILDMATCH([Bus Trans Code], '*Buy*'), 'Buy',
IF(WILDMATCH([Bus Trans Code], '*Sell*'), 'Sell','-'))
When to Use:
Purpose: Adds up all values in an expression
Syntax:
SUM(Expression)
SUM({SetExpression} Expression)
Project Example (from Sheet_01.txt):
LET xSHEET01_Entitlement = REPLACE('Sum({$(xFILTER_SHEET01)} [Nominal Basis])','@','$');
When to Use:
Purpose: Counts number of values (non-NULL)
Syntax:
COUNT(Expression)
COUNT(DISTINCT Expression)
Project Example (from Sheet_04.txt):
LET xSHEET04_BuyTrades = REPLACE('Count({...} DISTINCT [Bus Trans No])','@','$');
When to Use:
Purpose: Returns maximum or minimum value
Syntax:
MAX(Expression)
MIN(Expression)
Project Example (from Capital_Info.txt):
MAX("Share Date Full") AS "Share Date Full"
When to Use:
Purpose: Calculates average (mean) value
Syntax:
AVG(Expression)
When to Use:
Purpose: Returns value from a previous row of loaded table
Syntax:
PEEK(FieldName)
PEEK(FieldName, RowNumber)
PEEK(FieldName, Offset, TableName)
Project Example (from PFC_at_Lodgement.txt):
LET vPFCDList = Chr(39) & peek('PFCDList') & Chr(39);
When to Use:
Purpose: Returns value from previous row (in chart expressions)
Note: Used in chart expressions, not load script
Purpose: Returns current row number during load
Syntax:
ROWNO() // Row number in result
RECNO() // Row number in source
Project Example (from Transactions.txt):
LOAD
TDF AS [Trade Date Date Full],
RowNo() as RNo
RESIDENT TD
ORDER BY TDF;
When to Use:
RECNO() → 1, 3, 7 (original row numbers) ROWNO() → 1, 2, 3 (result row numbers)
---
## 10.4 FIELDVALUE / FIELDVALUECOUNT
**Purpose:** Returns specific value from a field by position / counts distinct values
**Syntax:**
```qlik
FIELDVALUE(FieldName, RowNumber) // Get Nth value (1-based)
FIELDVALUECOUNT(FieldName) // Count distinct values
Project Example (from Master_Calendar.txt, line 41-43):
LOAD
min(FieldValue('Trade Date', recno()))-1 AS MinDate,
max(FieldValue('Trade Date', recno())) AS MaxDate
AUTOGENERATE FieldValueCount('Trade Date');
Explanation:
FIELDVALUECOUNT('Trade Date') : How many distinct trade dates exist?AUTOGENERATE: Create that many rowsFIELDVALUE('Trade Date', recno()): Get Nth distinct trade dateWhen to Use:
Example:
// Get 5th distinct customer
FIELDVALUE('CustomerID', 5)
// Count distinct products
FIELDVALUECOUNT('ProductName')
// Loop through all distinct years
LOAD
FIELDVALUE('Year', IterNo()) as Year
WHILE IterNo() <= FIELDVALUECOUNT('Year');
// Create dimension from existing values
LOAD
FIELDVALUE('Status', ROWNO()) as Status
AUTOGENERATE FIELDVALUECOUNT('Status');
Important Notes:
Purpose: Returns number of rows in a table
Syntax:
NOOFROWS('TableName')
Note: Not directly found in project, but commonly used pattern
When to Use:
Example:
// Check if table loaded successfully
IF NOOFROWS('Sales') > 0 THEN
TRACE Table Sales loaded with $(NOOFROWS('Sales')) rows;
ELSE
TRACE WARNING: Sales table is empty!;
END IF
// Store row count in variable
LET vRowCount = NOOFROWS('Transactions');
// Conditional processing
IF NOOFROWS('Errors') > 0 THEN
STORE Errors INTO ErrorLog.qvd (qvd);
END IF
// Loop through all rows
FOR i = 0 TO NOOFROWS('TempTable')-1
LET vValue = PEEK('FieldName', $(i), 'TempTable');
TRACE Row $(i): $(vValue);
NEXT i;
Purpose: Returns timestamp when QVD file was created
Syntax:
QVDCREATETIME('QVDFilePath')
Use Case (for incremental load):
// Check if QVD exists and get last load time
LET vQVDPath = 'Data/Transactions.qvd';
IF NOT IsNull(QvdCreateTime('$(vQVDPath)')) THEN
// QVD exists - do incremental load
[TempLastLoad]:
LOAD MAX([Trade Date]) AS vMaxDate
FROM [$(vQVDPath)] (qvd);
LET vLastLoadDate = Peek('vMaxDate', 0, 'TempLastLoad');
DROP TABLE TempLastLoad;
TRACE Incremental load from $(vLastLoadDate);
// Load only new records
[Transactions]:
LOAD * FROM Source.qvd (qvd)
WHERE [Trade Date] > '$(vLastLoadDate)';
// Append historical data
CONCATENATE ([Transactions])
LOAD * FROM [$(vQVDPath)] (qvd);
ELSE
// QVD doesn't exist - full load
TRACE Full load - QVD not found;
[Transactions]:
LOAD * FROM Source.qvd (qvd);
END IF
// Store updated QVD
STORE [Transactions] INTO [$(vQVDPath)] (qvd);
When to Use:
Example:
// Check QVD age
LET vFileAge = Now() - QvdCreateTime('Data.qvd');
IF vFileAge > 1 THEN
TRACE QVD is more than 1 day old - refreshing;
// Reload from source
ELSE
TRACE Using cached QVD ($(vFileAge) days old);
LOAD * FROM Data.qvd (qvd);
END IF
Purpose: Creates field with both numeric and text representation
Syntax:
DUAL(TextValue, NumericValue)
Project Example (from Master_Calendar.txt, line 11-12):
Dual(Month(TempDate) & ' - ' & Year(TempDate),
Year(TempDate) & Num(Month(TempDate),'00')) AS YearMonthName,
Dual(Week(weekstart(TempDate)) & '-' & WeekYear(TempDate),
Year(TempDate) & Num(Week(weekstart(TempDate)),'00')) as WeekYear,
Explanation:
Visual Result:
Without DUAL: With DUAL:
Month (sorted) Month (sorted)
Apr 2024 Jan 2024
Aug 2024 Feb 2024
Dec 2024 Mar 2024
Feb 2024 Apr 2024
Jan 2024 ...
When to Use:
Example:
// Month with proper sorting
DUAL('January', 1) as Month
DUAL('February', 2) as Month
// Status with readable text
DUAL('New', 1) as Status
DUAL('In Progress', 2) as Status
DUAL('Completed', 3) as Status
// Displays text but sorts numerically
// Priority levels
DUAL('Low', 1) as Priority
DUAL('Medium', 2) as Priority
DUAL('High', 3) as Priority
DUAL('Critical', 4) as Priority
// Quarter display
DUAL('Q1 2024', 20241) as Quarter
DUAL('Q2 2024', 20242) as Quarter
How DUAL Works:
Purpose: Filters data in expressions (aggregations) independent of current selections
Syntax:
SUM({SetExpression} Field)
Project Example (from General_Filters.txt, line 11-26):
LET xFILTER_Equity = '<[Instrument Type] *= {"Equity","GDR/ADR","Right","Warrant", "Covered warrant", "Bond"}>';
LET xFILTER_Brokerage = '<[Cost Type] = {"BROKER","COMMISSION"}>';
LET xFILTER_StampDuty = '(<[Cost Type] = {"*STMP"}> + <[Cost Class] = {"STAMP_DUTY"}>)';
Project Example (from Sheet_01.txt, line 59-63):
LET xSHEET01_Entitlement = REPLACE('Sum({$(xFILTER_SHEET01)} [Nominal Basis])','@','$');
LET xSHEET01_GrossAmountMyr = REPLACE('Sum({$(xFILTER_SHEET01)} [Interest Dividend Capital Event Pc])','@','$');
// Where xFILTER_SHEET01 is defined earlier as:
LET xFILTER_SHEET01 = '<[Bus Trans Code] *= {"DRP Dividend","Dividend", "CapRed"}
, [Trade Date]={"<=@(=GetFieldSelections([Trade Date S1]))"}
, [Payment Date Transaction]={">=@(=GetFieldSelections([Trade Date S1]))"}
> + <[Bus Trans Code] *= {"DRP Dividend","Dividend", "CapRed"}
, [Trade Date]={"<=@(=GetFieldSelections([Trade Date S1]))"}
, [PaymentClose] = {"0"}
>';
Explanation:
{...} : Set analysis modifier<FieldName = {Value}> : Set modifier syntax+ combines sets (OR logic)Basic Syntax Components:
SUM({<Field = {Value}>} Amount)
│ │ │ │ │
│ │ │ │ └─ Field to sum
│ │ │ └─ Value to filter
│ │ └─ Field to filter
│ └─ Set modifier
└─ Set expression brackets
Purpose: Applies filters within set analysis
Operators:
= : Equal to+= : Add to current selection-= : Exclude from current selection*= : Keeps intersection with current selection/= : Replaces current selection (symmetric difference)Project Example (from General_Filters.txt, line 11-24):
// Exact match
LET xFILTER_Brokerage = '<[Cost Type] = {"BROKER","COMMISSION"}>';
// Wildcard match (contains "STMP")
LET xFILTER_StampDuty = '<[Cost Type] = {"*STMP"}>';
// Exclusion (NOT equal)
LET xFILTER_NotBrokerage = '<[Cost Type] -= {"BROKER","COMMISSION"}>';
// Exclusion of multiple
LET xFILTER_FEES_SHEET05 = '<[Cost Type] -= {"BROKER", "COMMISSION", "MY_CLRG", "*STMP", "BROKER_SST", "SCAN_SST"},
[Cost Class] -= {"STAMP_DUTY"}>';
// Wildcard match with *= operator
LET xFILTER_Equity = '<[Instrument Type] *= {"Equity","GDR/ADR","Right","Warrant"}>';
Explanation:
= : Include only these values-= : Exclude these values (show everything else)*= : Wildcard match ("*STMP" matches "ADD_STMP", "SELL_STMP", etc.)Examples:
// Equal (exact match)
SUM({<Year = {2024}>} Sales)
// Multiple values (OR)
SUM({<Status = {'Approved', 'Completed'}>} Amount)
// Exclude values
SUM({<Status -= {'Cancelled', 'Rejected'}>} Amount)
// Wildcard match
SUM({<Product *= {'*Phone*'}>} Sales)
// Matches: 'iPhone', 'Android Phone', 'Phone Case'
// Multiple conditions (AND)
SUM({<Year = {2024}, Region = {'North'}>} Sales)
// Comparison operators
SUM({<Amount = {">1000"}>} Amount)
SUM({<Date = {">=01/01/2024"}>} Sales)
Purpose: Inserts variable values into expressions
Syntax:
$(VariableName) // Expands variable value
$(=Expression) // Evaluates expression
Project Example (from Sheet_01.txt, line 59):
LET xSHEET01_Entitlement = REPLACE('Sum({$(xFILTER_SHEET01)} [Nominal Basis])','@','$');
// ^
// Variable expansion: $(xFILTER_SHEET01)
Explanation:
$(xFILTER_SHEET01) gets replaced with the filter definitionSum({<[Bus Trans Code] *= {"DRP Dividend",...}>} [Nominal Basis])Project Pattern - Why REPLACE('@', '$')?
// PROBLEM: Dollar sign in string triggers immediate expansion
LET Var = 'Sum({$(Filter)} Amount)';
// $(Filter) expands WHEN defining Var
// SOLUTION: Use @ as placeholder, replace with $ later
LET Var = REPLACE('Sum({@(Filter)} Amount)', '@', '$');
// @(Filter) stays literal, $ inserted later when Var is used
When Variable Expands:
// Definition time (SET - no expansion)
SET vYear = 2024;
SET vFormula = 'SUM(Sales)';
// Definition time (LET - evaluates)
LET vToday = Today(); // Stores numeric date value
LET vCount = COUNT(Customer); // Error! Can't aggregate during script
// Usage time (in expression)
SUM({<Year = {$(vYear)}>} Sales) // Expands to: Year = {2024}
$(vFormula) // Expands to: SUM(Sales)
Dynamic Expressions:
// Current vs Prior Year
SET vCurrentYear = '=Year(Today())';
SET vPriorYear = '=Year(Today())-1';
// In chart
SUM({<Year = {$(=vCurrentYear)}>} Sales) // Current year sales
SUM({<Year = {$(=vPriorYear)}>} Sales) // Prior year sales
// Conditional formatting
=$(=IF(SUM(Sales) > 1000000, 'Green', 'Red'))
// Complex variable
SET vYTDSales = 'SUM({<Date = {">=$(=YearStart(Today()))<=$(=Today())"}>} Sales)';
// Usage: $(vYTDSales)
Project Example (from Sheet_02.txt, line 32-33):
LET xSHEET02_Quantity = REPLACE('Sum({<@(xFILTER_SHEET02)>} [Nominal])','@','$');
// ^
// @(var) will become $(var) after REPLACE
Advanced: Nested Expansion
LET vFilter = 'Year = {2024}';
LET vExpression = 'SUM({<$(vFilter)>} Sales)';
// When used: $(vExpression)
// Step 1: Expands to: SUM({<$(vFilter)>} Sales)
// Step 2: Expands to: SUM({<Year = {2024}>} Sales)
Purpose: Define variables in script
Syntax:
SET VariableName = Expression; // Stores as literal text
LET VariableName = Expression; // Evaluates then stores
Project Example (from Main.txt, line 19):
LET Var_Today = Today();
// LET evaluates Today() → stores numeric value (e.g., 44941)
SET DateFormat = 'DD-MM-YYYY';
// SET stores literal text 'DD-MM-YYYY'
Project Example (from General_Filters.txt, line 11-30):
LET xFILTER_Equity = '<[Instrument Type] *= {"Equity","GDR/ADR","Right","Warrant"}>';
LET xFILTER_Brokerage = '<[Cost Type] = {"BROKER","COMMISSION"}>';
LET xFILTER_StampDuty = '(<[Cost Type] = {"*STMP"}> + <[Cost Class] = {"STAMP_DUTY"}>)';
All use LET to store text values (even though they look like formulas)
Project Example (from Variables.txt, line 3):
LET vSHEET05_ShowTransNo = 'Yes';
SET vNrUser = "1";
Key Differences:
// SET - stores literal text
SET vFormula = '=SUM(Sales)';
TRACE $(vFormula); // Output: =SUM(Sales)
// LET - evaluates first
LET vFormula = '=SUM(Sales)';
TRACE $(vFormula); // Output: =SUM(Sales) (same in script, different in UI)
// SET with calculation
SET vYear = Year(Today());
TRACE $(vYear); // Output: Year(Today()) [literal text]
// LET with calculation
LET vYear = Year(Today());
TRACE $(vYear); // Output: 2024 [evaluated]
// SET with equals sign
SET vExpression = '=SUM(Sales)/1000';
// In chart: $(vExpression) → Evaluates as expression
// LET with equals sign
LET vExpression = '=SUM(Sales)/1000';
// Same as SET in this case
When to Use Each:
// Use SET for:
SET DateFormat = 'DD-MM-YYYY'; // Constants
SET vExpression = '=SUM(Sales)'; // Chart expressions
SET vFilter = '<Year = {2024}>'; // Set analysis filters
// Use LET for:
LET vToday = Today(); // Evaluated values
LET vRowCount = NoOfRows('Table'); // Calculations
LET vMaxDate = Peek('MaxDate'); // Function results
LET vPath = '$(AppPath)/Data/'; // Concatenation with variables
Project Pattern:
// Define filter pattern with SET (literal text)
SET Filter = '<Field = {Value}>';
// Create specific measure with LET (can use variable expansion)
LET Measure = 'SUM({$(Filter)} Amount)';
// In chart expression:
=$(Measure) // Expands to: SUM({<Field = {Value}>} Amount)
Purpose: Insert variable value into expression
Methods:
$(VarName) : Basic expansion$(=Expression) : Evaluate expressionProject Example (throughout Sheet scripts):
// Define filter variable
LET xFILTER_SHEET01 = '<[Bus Trans Code] = {"Dividend"}>';
// Use in measure
LET xSHEET01_Entitlement = REPLACE('Sum({$(xFILTER_SHEET01)} [Nominal Basis])','@','$');
// ^
// Variable expansion $(xFILTER_SHEET01)
Expansion Timing:
// Script time expansion (during load)
LET vPath = '$(AppPath)/Data/';
LOAD * FROM [$(vPath)File.qvd] (qvd);
// Expands immediately: [C:/MyApp/Data/File.qvd]
// Expression time expansion (in chart)
SET vSales = 'SUM(Sales)';
// In chart: $(vSales) → Expands when chart renders
// Evaluated expansion
$(=Today()) // Evaluates Today() function
$(=MAX(Year)) // Evaluates MAX(Year) from current data
Project Example (from Transactions.txt, line 189):
FROM [$(LIB_FOLDER)/$(DWHMODEL)/Transactions/Transactions.qvd] (qvd)
// ^ ^
// Variable Variable
// expansion expansion
Example: Dynamic Date Range
// In script
LET vStartDate = Date(YearStart(Today()), 'YYYY-MM-DD');
LET vEndDate = Date(Today(), 'YYYY-MM-DD');
// In expression
SUM({<Date = {">=$(=vStartDate)<=$(=vEndDate)"}>} Sales)
// What it becomes:
// SUM({<Date = {">=2024-01-01<=2024-03-15"}>} Sales)
Example: Conditional Formula
// Define conditional measure
SET vMeasure = IF(GetSelectedCount(Year) = 1,
'SUM(Sales)',
'SUM(Sales) / COUNT(DISTINCT Year)');
// In chart
=$(vMeasure)
// If 1 year selected: SUM(Sales)
// If multiple years: SUM(Sales) / COUNT(DISTINCT Year)
Purpose: Removes table from data model to free memory
Syntax:
DROP TABLE TableName;
DROP TABLES Table1, Table2, Table3; // Multiple tables
Project Example (from Mappings.txt, line 19, 40, etc.):
// Load temp table
TmpBCat:
LOAD BUS_TRANS_CATEGORY_KEY, "Field1", "Field2", "Field3"
FROM [Source.qvd] (qvd);
// Create mappings
[mField1]: MAPPING LOAD KEY, "Field1" RESIDENT TmpBCat;
[mField2]: MAPPING LOAD KEY, "Field2" RESIDENT TmpBCat;
// Drop temp table to save memory
DROP TABLE TmpBCat;
Project Example (from PFC_at_Lodgement.txt, line 22):
DROP TABLES mLastPFC, LastPFC;
Project Example (from Transactions.txt, line 334):
Drop Tables TD2;
When to Use:
Example:
// Create dimension from fact
[Products]:
LOAD DISTINCT ProductID, ProductName
RESIDENT Sales;
// Don't need full sales anymore
DROP TABLE Sales;
// Multiple temporary tables
[Temp1]: LOAD * FROM Source1.qvd (qvd);
[Temp2]: LOAD * FROM Source2.qvd (qvd);
// ... process ...
DROP TABLES Temp1, Temp2;
Best Practice Pattern:
// 1. Load temporary data
[TempData]:
LOAD * FROM Source.qvd (qvd);
// 2. Extract what you need
[Final]:
LOAD Key, Value
RESIDENT TempData
WHERE Condition;
// 3. Drop temporary table
DROP TABLE TempData;
Purpose: Removes specific field(s) from table
Syntax:
DROP FIELD FieldName;
DROP FIELDS Field1, Field2 FROM TableName;
Project Example (from Master_Calendar.txt, line 53):
DROP FIELDS RangeStart, RangeEnd;
Removes temporary calculation fields no longer needed
When to Use:
Example:
// Create calculated field
LOAD
*,
TempValue1 + TempValue2 as FinalValue
FROM Data.qvd (qvd);
// Drop intermediate fields
DROP FIELDS TempValue1, TempValue2;
// Drop from specific table
DROP FIELDS InternalKey, TechnicalFlag FROM Transactions;
// Drop system fields
DROP FIELDS %Key, %TableID;
Purpose: Saves table to QVD file for future loads
Syntax:
STORE TableName INTO FilePath.qvd (qvd);
Project Example (from PFC_at_Lodgement.txt - implied but not shown):
// Would be used like:
STORE [PFC] INTO [$(QVD_PATH)/PFC_Latest.qvd] (qvd);
STORE [PFC_Holdings] INTO [$(QVD_PATH)/PFC_Holdings_Latest.qvd] (qvd);
Use Case: Incremental Loading
// Load new data
[Transactions]:
LOAD * FROM Source.qvd (qvd)
WHERE [Date] > '$(vLastLoadDate)';
// Append historical data
CONCATENATE ([Transactions])
LOAD * FROM Transactions_History.qvd (qvd)
WHERE [Date] <= '$(vLastLoadDate)';
// Store combined data
STORE [Transactions] INTO Transactions_History.qvd (qvd);
Use Case: Caching
// Check if cached QVD exists and is recent
IF NOT IsNull(QvdCreateTime('Cache.qvd')) THEN
LET vAge = Now() - QvdCreateTime('Cache.qvd');
IF vAge < 1 THEN
// Use cache
LOAD * FROM Cache.qvd (qvd);
EXIT SCRIPT;
END IF;
END IF;
// Calculate fresh data
[Data]:
LOAD * FROM [Complex calculation...];
// Store for next time
STORE [Data] INTO Cache.qvd (qvd);
When to Use:
Example: Multi-Stage ETL
//=== Stage 1: Extract ===
[RawData]:
LOAD * FROM Database USING Connection;
STORE [RawData] INTO Stage1_Raw.qvd (qvd);
//=== Stage 2: Transform ===
[TransformedData]:
LOAD *, [Calculation] as NewField
FROM Stage1_Raw.qvd (qvd);
STORE [TransformedData] INTO Stage2_Transformed.qvd (qvd);
//=== Stage 3: Load ===
LOAD * FROM Stage2_Transformed.qvd (qvd);