Combiner functions in Power Query

In Power Query "Table.CombineColumns" function is used to combine values from several columns (1) into one column (2).  This function takes 4 arguments. First argument is source table. Second argument is list of columns to combine. Last argument is name of newly created column.

Table.CombineColumns( Source, { "C1","C2","C3","C4" }, Combiner.CombineTextByDelimiter("<>"), "CombineByDelimiter" )

Third argument is what interests us. That argument explains how to combine column values. That argument is given by using one of this five functions:

  • Combiner.CombineTextByDelimiter
  • Combiner.CombineTextByEachDelimiter
  • Combiner.CombineTextByLengths
  • Combiner.CombineTextByPositions
  • Combiner.CombineTextByRanges

Some of these functions are simple, but others are not so obvious.

Combiner.CombineTextByDelimiter

This version is simple. It concatenate all values by placing delimiter between them. Combiner. CombineTextByDelimiter function is used just to give that delimiter. In thise example, delimiter is "–".

Table.CombineColumns( Source, { "C1","C2","C3","C4" }, Combiner.CombineTextByDelimiter("-"), " CombinedByDelimiter" )
C1C2C3C4CombinedByDelimiter
ABBBBBCDDDD=>ABBBBCDDDD

This is similar to TextJoin function in Excel:

TextJoin
=TEXTJOIN(" – ";TRUE;A2:D2)=>ABBBBCDDDD

Combiner.CombineTextByDelimiter can have one more argument. That argument could take value QuoteStyle.None or QuoteStyle.Csv. Quote.Csv will change every column which has quotes, so that column content will be delimited with quotes and all original quotes will be doubled.

ABB"BBCDDDD=> ABB"BBCDDDD QuoteStyle.None
ABB"BBCDDDD=> A"BB""BB"CDDDD QuoteStyle.Csv

Combiner.CombineTextByDelimiter( "-", QuoteStyle.Csv or QuoteStyle.None)

Combiner.CombineTextByEachDelimiter

This version is similar to previous. This time we give a list of delimiters. Those delimiters are used sequentially to separate column values.

Table.CombineColumns( GiveType, { "C1","C2","C3","C4" }, Combiner.CombineTextByEachDelimiter( { "_1_", "_2_", "_3_" }, QuoteStyle.Csv), "CombineByDelimiter" )
C1C2C3C4CombinedByEachDelimiter
ABBBBCDDDD=>A_1_BBBB_2_C_3_DDDD

The question is what if have too many or too less of delimiters. If we have five delimiters, but four columns, the result will be unchanged "A_1_BBBB 2_C_3_DDDD". Surplus delimiters will not be used. If we have only two delimiters, but four columns, the result be "A_1_BBBB_2_CDDDD". There will be no delimiters between "C" and "DDDD".

This function also accepts QuoteStyle second argument.

Combiner.CombineTextByLengths

We define list of numbers. From each column only so many characters will be presented.

Table.CombineColumns( GiveType, { "C1","C2","C3","C4" }, Combiner.CombineTextByLengths( { 2, 3, 3, 3 } ), "CombinedByLenghts" )

From first column we will take only two characters, from second column only three characters and so on. If we don't have enough characters in our column, column content will be padded with spaces to reach reserved space. If we have too many characters in column, rest of characters will be trimed. If our argument is { 2, 3, 3 } then the last column will not be presented.

C1C2C3C4 CombinedByLenghts
ABBBBCDDDD=> A␣BBBC␣␣DDD

This function has another argument called "template". Template is "background" text that we can use to create mask for our text.

Table.CombineColumns( GiveType, { "C1","C2","C3","C4" }, Combiner.CombineTextByLengths( { 2, 3, 3, 6 }, "*******" ), "CombineByDelimiter" )

If our "background" mask is "*******" then this will combine with previous result "A␣BBBC␣␣DDD" to create "A*BBBC*DDD".

Combiner.CombineTextByPositions

In this case, start of every column content is determined by positions list.

Table.CombineColumns( GiveType, { "C1","C2","C3","C4" }, Combiner.CombineTextByPositions( { 2, 4, 6, 9, 12 } ), "CombineByPositions" )

So, content of first column will start on position (2), of second column on position (4), third column on position (6), last column on position (9). If our list contains only numbers { 2, 4, 6 } then there will be no last column values in final result.

0123456789101112
A…B…C…D…

Then, we'll add rest of letters to see what is happening. Columns will fill as much space as possible. If column content is not enough, reserved space will be filled with spaces. If our list only has values { 2, 4, 6, 9 }, last column would not be trimed to three "DDD".

0123456789101112
ABBCDDD

This is final result.

C1C2C3C4 CombinedByPositions
ABBBBCDDDD=>␣A␣BBC␣␣DDD

This function also accepts Template second argument.

Combiner.CombineTextByRanges

This version is combination of Positions and Lengths versions. Our argument is sequence of pairs. First value in pair is position. The second is length.

Table.CombineColumns( GiveType, { "C1","C2","C3","C4" }, Combiner.CombineTextByRanges( { { 2,2 },{ 4, 4},{ 5,2 },{ 7,3 } } ), "CombineByPositions" )

We can see bellow that content of every column starts on positions 2, 4, 5, 7 respectively. Each columns content has maximal length 2, 4, 2, 3 characters respectively. Latter columns will overwrite former columns. For example, column 3 will overwrite column 2 on position 5 ("C" instead of "B").

0123456789
C1A
C2BBBB
C3C
C4DDD
resultABCBDDD

If we only use pairs { { 2,2 },{ 4, 4},{ 5,2 } }, then last column will not be presented. Length of range can not be zero.

This function also accepts Template second argument.

Excel sample file:

How to Refresh Only Subset of Tables

Often, we do not want to refresh all the tables in Power Pivot or Power BI Desktop. We can choose subset of tables to refresh, both in Power Pivot and Power BI Desktop, as a permanent settings. It is also possible to select tables to refresh with mouse and only those tables will be refreshed. For Power Pivot we can use VBA to select tables to be refreshed.

Power BI Desktop

In Power BI Desktop, all you have to do is to uncheck the "Include in report refresh" option, for some of the tables, and those tables will not be part of refresh. This option is located in the contextual menu for each query, in Power Query window (1). We can see that only tables SampleTable1 and SampleTable2 will be refreshed (2) if we click on Refresh button in Ribbon. This is because we unchecked this option for SampleTable3 and SampleTable4.

Beside "Include in report refresh" option, which is permanent, we can choose which tables to refresh by selecting them in model view. Holding Ctrl key and clicking on the table headers in model view, we will select subset of tables. On any of those tables we can open contextual menu with right click. Refresh button is in this menu. Now only, tables SampleTable2 and SampleTable4 will be refreshed, because only those tables are selected.

Power Pivot and Power Query

Similar to Power BI Desktop, Power Pivot also has option to permanently disable refreshing of some tables (1). This option is placed in Properties for Queries and Connections (2). Queries are made by Power Query and for them we can control Refresh status for each query.

If the data is retreived using Power Pivot directly, without Power Query, we can only set refresh setting for each connection. This setting is valid for all tables that are loaded by using that connection. If we want to control refreshing of each table separatelly, we have to create separate connection for each of the tables. On the image bellow we can see that we have two connections made to the same Excel file. This is solution for having full control.

In Excel, we can also select tables in Diagram view (1). After this we should click on "Refresh" (not Refresh All), and only selected tables will be refreshed (3).

VBA

If name of query in Power Query is "SampleTable" then by using this line of code we can refresh only that query with VBA.

ThisWorkbook.Connections("Query - SampleTable").OLEDBConnection.Refresh

For refreshing a connection we can use this line of code:

ThisWorkbook.Connections("WorksheetConnection_New Microsoft Excel Worksheet.xlsx!SampleTable").Refresh

In this case it is important whether this query will be executed asynchronously. This article explains how to change that setting.

Excel and Power BI Desktop sample file:

Volatile, Reference, Case Sensitive Functions

Volatile functions

Volatile functions are recalculated each time any cell in the spreadsheet is changed. Some other actions can also cause recalculation, such as renaming sheets, inserting columns, deleting rows, etc. Any formula that contains a volatile function will also become volatile. Any cell that depends on a cell that contains volatile function will also become volatile. Too many cells that contain a volatile function in a spreadsheet will significantly slow down that spreadsheet.

Volatile functions are RAND, RANDBETWEEN, RANDARRAY, NOW, TODAY, OFFSET, INDIRECT. INFO and CELL functions can be volatile depending on which arguments they use.

UDF functions can become volatile if the statement "Application.Volatile" is placed in front of them. The function below is nonvolatile.

Function UnVolatileFunction()
UnVolatileFunction = ActiveCell.Offset(-2, 0).Value + ActiveCell.Offset(-1, 0).Value
End Function

In contrast, this function is volatile because it contains an "Application.Volatile" statement..

Function VolatileFunction()
Application.Volatile
VolatileFunction = ActiveCell.Offset(-2, 0).Value + ActiveCell.Offset(-1, 0).Value
End Function

Excel Functions That Return References

Excel functions that return references are OFFSET, INDEX, XLOOKUP, CHOOSE, SWITCH, IF, IFS, INDIRECT. This means that the result of the function is neither a scalar nor an array. The result is the address of the range in the table.

Although the reference refers to multiple cells, Excel 365 or 2021 can directly display the result of these functions as a spill (1). In older versions of Excel, it was necessary to convert these functions into array functions with Ctrl + Shift + Enter in order to display their result directly (2). If a given range contains only one cell, all versions of Excel will return the contents of that cell (3). If a range contains more than one cell, it has always been possible to wrap that range in an aggregate function that will result in a scalar that can be displayed in a spreadsheet (4).

Case Sensitive Excel Functions

In Excel, there are three functions that are case sensitive. Those are EXACT, FIND and SUBSTITUTE. There are also XLOOKUP and XMATCH functions that have arguments that can be used to control whether the functions are case sensitive. Those functions are available in Excel 365 and 2021.

Direct comparison of A1 and B1 cells will return TRUE (1), but if we use EXACT function, result will be FALSE (2).

How to unpivot Excel tables

Unpivot by using pivot table

This technique is only useful for unpivoting tables with smaller number of columns. Power Query is preferable for larger tables.

We have a table with a pivoted column of office products. We want to unpivot that column.

All we have to do is to create a normal Pivot table. In this pivot table all unchanged columns should be placed into Rows section (1). All other columns, that should be unpivoted, should be placed in Values section (2). Notice that in Columns section we have a special column "∑ Values" (3), that Excel created itself.

Now drag that special "∑ Values" column into Rows section (4). We will immediately get unpivoted table (5). This table asks for some maintenance so is best to make a copy of it in spreadsheet and then fix its header, and filter rows with no values.

This approach could will not be optimal when there are many columns that should be unpivoted. In that case we have to move all those columns into Values section manually. This could be tedious. In that case it is much easier to use Power Query unpivot tool.

Unpivot by using Power Query

Get your unpivoted table into Power Query. Select all columns that should be unpivoted (1). In Transform tab (2) find command "Unpivot Columns" (3). This command will do all the work. Notice that we also have command "Unpivot Other Columns" (3). This allow us to select only columns that shouldn't be unpivoted, and then all the other columns will be unpivoted. When we have really big number of columns to unpivot, this command make it even more easier to accomplish our task.

At the end we get unpivoted table. Again, we have to fix column names (1). Rows without vales will be automatically filtered (2).

Excel sample file:

Instructional video:

PQ execute queries asynchronously

Power Query is great tool for preparing data to be written into CSV file. CSV files are usually used to import data into databases but the requirement is that they must be formatted in specific way. We can export formatted data to a spreadsheet, and then we can use VBA to write already formatted data to a CSV file.

Data connections in Power Query are by default set to be asynchronous. If we start a query with "Workbook.RefreshAll" from VBA procedure, the rest of VBA procedure won't wait for the query to complete. Instead of writing new data from the spreadsheet to a CSV file, VBA procedure will write the old ones, because there will be no time for the query to complete.

The solution is to change the execution of the query to synchronous. This is done in Excel Connection Properties. We can find our connection properties in Data tab (1) > Workbook Connections Properties (2). There is a checkbox "Enable background refresh". This checkbox should be disabled.

The checkbox can be disabled directly from the VBA. That possibility allow us to fully control the execution of a query in procedure. This procedure will temporarily change BackgroundQuery properties to False so that our Procedure will wait for the query to complete.

Sub Refresh_All_Data_Connections()
    For Each objConnection In ThisWorkbook.Connections
        'Get current background-refresh value
        bBackground = objConnection.OLEDBConnection.BackgroundQuery
        'Temporarily disable background-refresh
        objConnection.OLEDBConnection.BackgroundQuery = False
        'Refresh this connection
        objConnection.Refresh
        'Set background-refresh value back to original value
        objConnection.OLEDBConnection.BackgroundQuery = bBackground
    Next
End Sub

Excel sample table: