Werner Daehn
Oct 14, 2021

--

What was surprising to me is how often the data can be partitioned for parallel processing. Just as you described it.

If order 1 with data in the master and detail tables was created, after that order 2 and finally order 1 is updated, can it be loaded in parallel?

Yes! It is important that header is created first and item second and that the update happens after the order creation.

But this can be achieved by partitioning by order number. Then one topic contains header and detail records for all changes of order 1, the other topic the order 2 data.

The orders themselves must be consistent but if your or my order is created first, that is secondary.

And this happens all over the place. This is one of the reasons Kafka can be used so widely.

--

--

Werner Daehn
Werner Daehn

Written by Werner Daehn

Data Integration expert for Big Data and SAP

No responses yet