Introduction

Merge is used when we want to use records from one table to decide what records to modify in another table. Let's assume that we want to keep two tables synchronized. In that case, every change on the first table should be reflected on the second table.

For full reflection, all of the updates, deletes or inserts that are done on the A table should also be done on the B table. For partial reflection, we can do just the part of the full reflection. For example, new rows from the A table will also be added to the B table. Records updated or deleted in the table A, will not be updated or deleted in the table B.

MERGE statement cannot synchronize updates, deletes and inserts at the same time. Merge can synchronize ( INSERT and UPDATE ), or ( INSERT and DELETE ). Image bellow give us an explanation for INSERT and UPDATE. MERGE will create left outer join between tables. It will match rows based on our condition. Merge will then add and update records in the table B, so that table B is the same as the table A.

For INSERT and DELETE, we do the similar thing. New rows will be added to the table B, but matched rows will be deleted from the table B.

Sample Tables

We will create two tables. We will try to propagate all the changes in the table A to the table B. For start, we will enter only one row in the table A.

CREATE TABLE A ( Letter CHAR, Number INTEGER ); INSERT INTO A ( Letter, Number ) VALUES ( 'A', 1 ); CREATE TABLE B ( Letter CHAR, Number INTEGER );

Initial Synchronization

All the rows from the table A that do not exist in table B will be inserted into table B.

MERGE INTO B      USING A         ON B.Letter = A.Letter       WHEN NOT MATCHED THEN INSERT ( Letter, Number ) VALUES ( A.Letter, A.Number );

MERGE INTO targetTable USING sourceTable ON matchingCondition WHEN NOT MATCHED THEN INSERT RECORDS

Now, table B, has one row, the same as table A.

Merge with INSERT

First, we will insert another row into the table A. INSERT INTO A ( Letter, Number ) VALUES ( 'B', 2 );

Then we will repeat the same MERGE statement as above. This new row will then appear in the table B.

MERGE INTO B      USING A         ON B.Letter = A.Letter       WHEN NOT MATCHED THEN INSERT ( Letter, Number )                            VALUES ( A.Letter, A.Number );

Merge with UPDATE

First, we will update the table A. We will change A1 to A4.

UPDATE A SET Number = 4 WHERE Letter = 'A';

Then we'll push that change to the table B. Notice that we don't use 'WHEN NOT MATCHED'. Now we use 'WHEN MATCHED'.

MERGE INTO B           USING A                   ON B.Letter = A.Letter and A.Letter = 'A'       WHEN MATCHED THEN UPDATE SET Number = A.Number;

Merge with DELETE

We will update one record in the table A.

UPDATE A SET Number = 8 WHERE Letter = 'B';

Now, we will delete all the rows from the table B, that do exist in the table A. Row A4 exists in both tables, so that row will be deleted.

MERGE INTO B      USING A         ON B.Letter = A.Letter and B.Number = A.Number       WHEN MATCHED THEN DELETE;

Merge with INSERT and UPDATE

The real reason for existence of MERGE statement is because we can do two things in one statement. This time we will do INSERT and UPDATE at the same time.

MERGE INTO B           USING A                 ON B.Letter = A.Letter             WHEN NOT MATCHED THEN INSERT ( Letter, Number )                                                          VALUES ( A.Letter, A.Number )             WHEN MATCHED THEN UPDATE SET Letter = A.Letter,                                    Number = A.Number;

Merge with INSERT and DELETE

Currently, tables A and B are equal. We will add a row to table A to show how to use merge with INSERT and DELETE.

INSERT INTO A ( Letter, Number ) VALUES ( 'E', 5 );

MERGE INTO B      USING A                 ON B.Letter = A.Letter       WHEN NOT MATCHED THEN INSERT ( Letter, Number )                             VALUES ( A.Letter, A.Number )       WHEN MATCHED THEN DELETE;

Things That Don't Work in Merge

INSERT with Subquery

Subqueries are not supported in INSERT clause, inside of Merge statement. MERGE INTO B           USING A                 ON B.Letter = A.Letter             WHEN NOT MATCHED THEN INSERT SELECT * FROM A;

Several MATCHED, or NOT MATCHED sentences

We can only use one MATCHED clause and/or one NOT MATCHED clause. MERGE INTO B      USING A         ON B.Letter = A.Letter       WHEN MATCHED THEN DELETE       WHEN MATCHED THEN UPDATE SET Letter = A.Letter,                                    Number = A.Number;

Multiple matches

I will add row "E6" in the table "A" with a statement "INSERT INTO A VALUES ( 'E', 6 )". Now this table has rows E5 and E6. If we now apply MERGE statement, both rows E5 and E6 will try to update row E5 in the table B. This is not allowed and will fail.

MERGE INTO B USING A ON B.Letter = A.Letter WHEN MATCHED THEN UPDATE SET Letter = A.Letter, Number = A.Number;

First, we will create sample tables.

The Definition of Subquery

Subquery is SELECT statement nested within some other SQL statement. Subquery can be nested within an UPDATE, DELETE, or other SELECT statement.

We can use subquery in three ways.
1) As a source table. Here we use a subquery in the FROM clause.
2) As an expression that will become a new column. Here we use a subquery in the SELECT clause.
3) As an operand in Boolean expressions. This is when we use subquery in WHERE or HAVING clauses.

The SELECT Statement

SELECT statement is special. Only in the SELECT statement we can subquery as a source table, or an expression column.

Subquery as a Source Table

We can read our data from subquery. In this case subquery replaces the source table.   SELECT * FROM innerQuey;    SELECT * FROM ( SELECT * FROM innerQuery ) subqueryAlias;    The distinguishing element of each subquery is parentheses. They are always mandatory. If we are using subquery as the source table, then we must also use the alias.

SELECT * FROM outerQuery INNER JOIN ( SELECT * FROM innerQuery ) subqueryAlias     ON outerQuery.Letter = SubqueryAlias.Letter;

It is possible to use a subquery within a JOIN clause.

SELECT * FROM ( SELECT * FROM outerQuery UNION SELECT * FROM innerQuery ) subqueryAlias;

We can also create a union between two subqueries and then use that union in the FROM clause. This means that we can combine several subqueries into one, using set operators.

Subquery as a Column Expression

If a subquery returns a scalar, we can use that value directly as a column value.

SELECT 3, ( SELECT 5 + 5 );

SELECT *, ( SELECT 5 + 2 ) columnAlias FROM outerQuery;

Subquery as an Operand of a Boolean Expression

In SQL expressions we can work with scalars and with tables. These are the two data structures that exist in SQL.

SELECT '7';

This will return the scalar.

SELECT * FROM Table1;

This will return the table.

Once we get the result of a SELECT statement, we can use that result as an operand in Boolean expression. We need operators for expressions. Let's see what kind of operators we can use.

Comparison Operators

If the SELECT query returns a scalar, we can use that value with the comparison operators "=, <>, >, >=".

SELECT ( 2 = ( SELECT Number FROM innerQuery WHERE Letter = 'B' ) ) AS Result; This is the same as the statement: SELECT ( 2 = 2 ) AS Result; The inner query will return the number 2, which we will use in our expression.

For all operators we can negate the result with the NOT prefix.

SELECT ( NOT 2 = ( SELECT Number FROM innerQuery WHERE Letter = 'B' ) ) AS Result;   This prefix will transform a "true" result into a "false" result.

IN Operator

The IN operator will check if a value belongs to a set. We can negate IN with NOT IN.

SELECT 2 IN ( SELECT Number FROM innerQuery ) AS Result; The statement above is similar to the statements below. They all would get the same result. SELECT 2 IN ( 2, 2 ) AS Result; SELECT 2 IN ( VALUES (2),(2) ) AS Result;

The statement "VALUES (2), (2)" will return a column. The statement "VALUES ( 2, 2 )" will return a row. When we use the IN operator, we always have to use only one column. In some databases this statement below will work but this will not work in MonetDB. We can only use one column with the IN operator. SELECT ( 2, 2 ) IN ( VALUES ( 2, 2 ) );

ALL and ANY

Comparison operators can be made more powerful by combining them with the ALL or ANY operators.

The "Number" column from the innerQuery table has the values 2 and 3. In the example (1), we check whether ( 1 < 2 ) AND ( 1 < 3 ). The result is TRUE. SELECT 1 < ALL ( SELECT Number FROM innerQuery ) AS Result; In example (2), we check whether the number 3 is smaller than all the numbers from the innerQuery table, ( 3 < 2 ) AND ( 3 < 3 ). It is not, this expression returns false. SELECT 3 < ALL ( SELECT Number FROM innerQuery ) AS Result;

ANY is less demanding than ALL. With ANY, we ask "is there some number in the innerQuery table that will make our comparison TRUE?".

SELECT 2 < ANY ( SELECT Number FROM innerQuery ) AS Result; ANY is using OR logic, ( 2 < 2 ) OR ( 2 < 3 ). SELECT 2 < ALL ( SELECT Number FROM innerQuery ) AS Result; ALL is using AND logic ( 2 < 2 ) AND ( 2 < 3 ).

EXISTS, or NOT EXIST

Sometimes we just want to test whether a query is empty or not. We use the EXISTS operator for that.

Our table does not have the letter 'Z', so the subquery will be empty. That's why the EXISTS operator returns FALSE. SELECT EXISTS ( SELECT * FROM innerQuery WHERE Letter = 'Z' ) AS Result;

Conclusion

1) We can only compare compatible data types. We cannot compare apples and pears (numbers and strings).
2) The result of all expressions involving a subquery is Boolean.
3) When using subquery operators, the subquery can be on both the left and right sides. Some databases do not allow subqueries to be on the left. MonetDB is not like that.

This subquery is on the left side of operator. It is custom to place subquery on the right side, and I will follow that convention.

Subquery in WHERE or HAVING clauses

Now that we know how to use subquery in a Boolean expression, we can use it in WHERE or HAVING clauses. We can use HAVING in SELECT statement, but WHERE can be used in SELECT, UPDATE or DELETE statements. Let's look at some examples:

SELECT Letter, SUM( Number ) AS Total FROM ( SELECT * FROM outerQuery UNION ALL SELECT * FROM innerQuery ) subqueryAlias GROUP BY Letter; We will create a union of our two tables to group them together.

This SELECT statement will be used as an operand. It will return a result of 2.5. SELECT AVG( Number ) FROM innerQuery;

SELECT Letter, SUM( Number ) AS Total FROM ( SELECT * FROM outerQuery UNION ALL SELECT * FROM innerQuery ) subqueryAlias GROUP BY Letter HAVING SUM( Number ) > ( SELECT AVG( Number )                          FROM innerQuery );

We will combine the two statements above into one. We will filter only those groups in which the SUM(number) values are greater than 2.5.

These are two examples with WHERE.

UPDATE outerQuery SET Number = 5 WHERE Number IN ( SELECT Number FROM innerQuery );

The subquery will return the numbers 2 and 3. Therefore, the number 2 from the outer query will be updated to the number 5.

We can return that number using the following statement.

UPDATE outerQuery SET Number = 2 WHERE Number > ALL  ( SELECT Number FROM innerQuery );

The number 5 is now greater than all numbers from innerQuery. So we're going to move it back to number 2.

Now it's easy to understand why I named my tables outerQuery and innerQuery. We used outerQuery for the main query, and innerQuery table for the subquery.

Correlated Subquery

A correlated subquery is a special type of subquery. It is extremely powerful, but its performance is dismal. In real life, where we have large tables, using correlated subqueries will kill performance. We can still use it successfully on small tables.

SELECT oq.Letter, oq.Number, ( SELECT iq.Number FROM innerQuery iq WHERE iq.Number = oq.Number ) AS siblingNumber FROM outerQuery oq

In a correlated subquery, we execute the subquery once for each row of the outer query. The subquery uses the value from the outer query as its parameter. This can be better understood in the table below.

For each row, we execute the inner query once. For the WHERE parameter of the inner query, we take the value from the outer query. This is how a correlated subquery works.

outerQuery.Letter	outerQuery.Number		siblingNumber
A	1	SELECT Number FROM innerQuery WHERE Number = 1	=> Returns null
B	2	SELECT Number FROM innerQuery WHERE Number = 2	=> Returns 2

Correlated subquery can be used in SELECT, HAVING, WHERE clauses. A subquery is a correlated subquery if it uses a value from an outer query as a parameter.

Order By

In MonetDB, subqueries can use the ORDER BY clause.

SELECT * FROM ( SELECT * FROM innerQuery ORDER BY Number ) Alias;

In some other databases this is only allowed if the subquery uses LIMIT or OFFSET specifiers.

Where VS Having

Where and Having are two clauses that we will best explain using examples. This time we will create only one table.

1) CREATE TABLE WhereOrHaving ( Letter CHAR DEFAULT 'D', Number TINYINT ); 2) INSERT INTO WhereOrHaving VALUES ( 'A', 1 ), ( 'A', 2 ),                                    ( 'B', 3 ), ( 'B', 4 ),                                    ( 'C', 5 ), ( 'C', 6 );

Where

Where is simple. It is a filter that will exclude all the rows which don't satisfy the condition. SELECT * FROM WhereOrHaving WHERE Number > 3;

Having

Having is more complex. First, we will group our table by the first column.   SELECT Letter, SUM( Number ) As sumOfGroup FROM WhereOrHaving GROUP BY Letter;

Then we will remove all the rows where sumOfGroup is smaller than 3. This time we do not use WHERE clause. We are not filtering detail rows, this time we are filtering groups. For that we use HAVING clause. SELECT Letter, SUM( Number ) As sumOfGroup FROM WhereOrHaving GROUP BY Letter HAVING SUM( Number ) > 3

Let's combine these two clauses to better understand them.

1) This is our starting table. We will transform it step by step.

2) Next, we will filter only those rows where Number is bigger than 3.

3) We will now group the table from the previous step by the "Letter" column.

4) And finally, we will filter only groups that are bigger than 6.

So, first we have to use WHERE. WHERE is always used before grouping, because it works on the detail rows. After this comes grouping. When we get our groups then is the time to pick only some of them by using HAVING condition.

INSERT

INSERT statement is used to enter data into table. We will insert some more rows to table WhereOrHaving. Basic way of using INSERT is to make a list of values. These values will be placed in the table as a new row. Our table WhereOrHaving has two columns of CHAR and TINYINT types, so we should be careful to insert data of that data types.

INSERT INTO WhereOrHaving VALUES ( 'E', 36 );

It is also possible to enter several rows at once. INSERT INTO WhereOrHaving VALUES ( 'X', 36 ), ( 'Y', 46 ), ( 'Z', 56 );

IF we don't provide values for all the columns, insert will fail. INSERT INTO WhereOrHaving VALUES ( 'M' );
	We have to provide values for all the columns, even if some of values are nulls. INSERT INTO WhereOrHaving VALUES ( 'M', null );

Sometimes we want to be specific into which columns we want to enter values. Now we can provide values in any order we want.

INSERT INTO WhereOrHaving ( Number, Letter ) VALUES ( 13, 'P'); We will first provide value for Number and then for Letter.

If we use syntax where some of the columns are specified, but other are not, then all other columns will be filled with nulls.

INSERT INTO WhereOrHaving ( Letter ) VALUES ( 'G' );

If we enter value for Number, and not for Letter, then the DEFAULT value will be used for the Letter.

INSERT INTO WhereOrHaving ( Number ) VALUES ( 77 );

This image will revise all of our results.

INSERT With SELECT

We will take small sample from the table above.

SELECT * FROM WhereOrHaving WHERE Letter = 'D';

We can now add this sample into any other table. Presumption is that destination table should have the same structure as our sample. For simplicity we will not add this sample to same other table, but we will add it back to our WhereOrHaving table. Then, our table will have two ( 'D', 77 ) rows.

INSERT INTO WhereOrHaving SELECT * FROM WhereOrHaving WHERE Letter = 'D';

UPDATE

We saw that we now have two rows in WhereOrHaving table with nulls. We have "M, null" and "G, null".

UPDATE WhereOrHaving SET Number = 101 WHERE Letter = 'G';

We have updated "G, null" to "G, 101".

We can update values "M, null" to "Q, null".

UPDATE WhereOrHaving SET Letter = 'Q' WHERE Number IS null;

To find rows where column value is null, we use operator IS, and not =.

DEFAULT value for Letter columns is "D". We can set any field in Letter column to this default value.

UPDATE WhereOrHaving SET Letter = DEFAULT WHERE Number = 101;

Now we have default letter "D" beside number 101.

DELETE

We use DELETE for deletion of rows. We can return our table, WhereOrHaving, to its original state, by deleting all the added rows.     DELETE FROM WhereOrHaving WHERE NOT Letter in ( 'A', 'B', 'C' );

Built-In Variables

MonetDB has a collection of built-in variables. Those variables can provide us with valuable information about our current session and current time.

CURRENT_SCHEMA	CURRENT_DATE	CURRENT_TIMESTAMP	LOCALTIME
CURRENT_USER	CURRENT_TIME	CURRENT_TIMEZONE	LOCALTIMESTAMP

SELECT CURRENT_SCHEMA AS CurrentSchema, CURRENT_USER AS CurrentUser, CURRENT_DATE AS CurrentDate
     , CURRENT_TIME AS CurrentTime, CURRENT_TIMESTAMP AS CurrentTimestamp, CURRENT_TIMEZONE AS CurrentTimezone
     , LOCALTIME AS LocTime, LOCALTIMESTAMP AS LocTimeStamp;

SET OPERATORS

Let's say we have a purple and a green data set. The data set is the result of a query. Using the SET operators, we can treat the rows in those two data sets as mathematical sets.

With joins, we concatenate tables horizontally. If we want to fuse tables vertically then we would use a union. For union to work we need to meet three conditions: – Two data sets must have the same number of rows. – Order of columns in one data set must be the same as order of the columns in another data set. – Columns at the same position must have similar data type.   There are two kinds of unions, we have "UNION" and we have "UNION ALL". "UNION ALL" would just glue two data sets together. If the first data set has X rows, and the second one Y rows, final data set would have X + Y rows.   "UNION" will return the same thing, but without duplicates.   "INTERSECT" means that we are looking for the rows that belong both to the first and the second data set. "INTERSECT" will return only distinct rows. "EXCEPT" returns only rows from the first dataset, that do not have equivalents in the second data set. Again, only distinct rows will be returned.

Sample tables

We will create two sample tables. Notice that columns in these tables are using compatible data types ( CHAR vs VARCHAR(1), TINYINT vs SMALLINT ).

UNION ALL

Our tables are meeting conditions for a UNION ALL. SELECT * FROM aboveTable UNION ALL SELECT * FROM  belowTable;

UNION

UNION will remove duplicate rows. SELECT * FROM aboveTable UNION SELECT * FROM belowTable;

INTERSECT

Intersect will give us overlap between two tables. SELECT * FROM aboveTable INTERSECT SELECT * FROM belowTable;

EXCEPT

Let's see rows that exist in the aboveTable, but not in the belowTable.   SELECT * FROM aboveTable EXCEPT SELECT * FROM belowTable;

INTERSECT and EXCEPT with duplicates

If we can use ALL with UNION, is it possible to use it with INTERSECT and EXCEPT? It is possible. This way, all of the duplicates will remain.

SELECT * FROM aboveTable INTERSECT ALL SELECT * FROM  belowTable;	SELECT * FROM aboveTable EXCEPT ALL SELECT * FROM  belowTable;

ORDER BY and SET operators

When we use ORDER BY, it must be applied on the last query.   SELECT * FROM aboveTable UNION SELECT * FROM belowTable ORDER BY Number DESC;

Column names and SET operations

The above SELECT is determining names of columns. Names of columns in above and below SELECT don't have to be the same. SELECT Letter as L, Number as N FROM aboveTable UNION SELECT * FROM belowTable ORDER BY N DESC;

Connecting More Than 2 Tables with Set Operators

How EXCEPT works

SELECT * FROM aboveTable UNION ALL SELECT * FROM belowTable EXCEPT SELECT * FROM belowTable;

Chained operations are conducted in sequence manner. First we make an UNION, and then we apply EXCEPT.   Let's see below steps that are leading to our results.

1) In the first step, UNION ALL will unite our two tables. Next we have to do EXCEPT vs belowTable.

2) In the second step, all duplicates from both tables will be removed. Then we would do EXCEPT operation.   3) The only row that exist in the first table, and not in the second table is a row "A-1". That is our result.

How EXCEPT ALL works

SELECT * FROM aboveTable UNION ALL SELECT * FROM belowTable EXCEPT ALL SELECT * FROM belowTable;

When we use EXCEPT ALL, we would get 4 rows as a result. Let's see below steps that have leaded to this outcome.

1) In the first step, UNION ALL will unite our two tables. Next, we have to do EXCEPT ALL vs belowTable.

2) This time there is no DISTINCT. Rows from the two tables will destroy each other like matter and antimatter. What remains in the left table will be our final result.

EXCEPT ALL with DISTINCT

Statement below will return table on the right image. Now we have 6 rows in the result data set. SELECT * FROM aboveTable UNION ALL SELECT * FROM belowTable EXCEPT ALL SELECT DISTINCT * FROM belowTable;

This time only two rows from the UNION ALL tables will be excluded so we will be left with 6 remaining rows.

UNION AND INTERSECT

Here is one example with INTERSECT. Again, first two data sets will be connected with UNION ALL. Their result will be than intersected with the last data set. SELECT * FROM aboveTable UNION ALL SELECT * FROM belowTable INTERSECT ALL SELECT DISTINCT * FROM belowTable;

1) We will start two intermediate data sets. First one is the result of UNION and it will have 8 rows. The other one is created with DISTINCT from the belowTable, and it has 2 rows. 2) Intersection between these two data sets is this:

Two Conclusions

1) SET operators don't have priority among them. They are applied from the top to the bottom. This is different than in some other databases where INTERSECTION has the top priority. This would be the result if we enforce priority of an INTERSECT operator.

2) SET operators without ALL specifiers will first remove duplicates from their operands. SET operators with ALL specifiers will leave their operands untouched. This is true not only for EXCEPT (like in previous examples), but also for UNION and INTERSECT. The message here is that removal of the duplicates is not conducted on the result, but on the operands, and after that SET operators will apply their logic.

Corresponding

THIS IS NOT WORKING IN MONETDB, although documentation claims that it works.

When working with SET operators we need to be careful to order our columns correctly. Their position has to match between tables. This makes things harder. We will face errors many times.

CORRESPONDING clause can fix this problem. CORRESPONDING clause will correctly pair columns so that UNION is successful. Unfortunately, statement on the right side will work in some other databases, but not in the MonetDB, so I will just stop explaining it. You can read more about this clause on this blog: https://blog.jooq.org/a-rarely-seen-but-useful-sql-feature-corresponding/

SELECT Number, Letter FROM aboveTable UNION ALL CORRESPONDING SELECT Letter, Number FROM belowTable;