Werner Daehn
1 min readDec 12, 2021

--

I see a couple of contradictions in the text I am having a hard time to resolve:

1. Git rid of the ETL process that loads the DWH. You can use dbt to build models on top of an existing DWH. (I know, you want to copy the source database model into the DWH staging area and do all transformations from there)

2. Do not duplicate the data. Duplicate your data into Pre-Aggregated tables.

3. Use incremental models with dbt yet how do you build them when there is no delta information, records can be deleted, tables change constantly and users need subsecond response times?

4. Express all in SQL yet many transformations cannot be expressed in SQL easily, e.g. compare with last year's revenue? Some can be done with window functions, some cannot and for sure with a performance penalty.

5. Do not use ETL tools (which generate SQLs to join, aggregate,... the source data) and use dbt instead, because it generates SQLs.

Essentially what is described here is a semantic layer like Business Objects ("BO") had 20 years ago (except in dbt it is code, BO had a nice UI) plus database optimization techniques, primarily materialized views, except that databases are much more powerful in that regards.

Am I missing the point?

--

--

Werner Daehn
Werner Daehn

Written by Werner Daehn

Data Integration expert for Big Data and SAP

No responses yet