Data and analytics

In September, I wrote about whether data mindsets are hurting integration. Since then, I feel like it may not be a broad problem, but it does happen. It caused me to think a but more about what the difference is between a data problem and an integration/software problem.

After some thought, I think I’ve arrived at a hopefully cogent answer. Integration is about message passing (units of work). Integration folks, much like traditional application developers, focus on coding, architecting, deploying, and monitoring. Data folks tend to think more about collecting, analyzing, and reporting.

The mindset is obviously different, as are the tools. Integration tends to involve services, API calls, and transactions (sort of). All of these things, when done properly, involve thinking about encapsulation and abstraction. In a modern SAAS world, you will not (and should not) even be able to access the data store underneath these applications. The tools (languages, frameworks) and approaches like TDD/CI/CD have mostly carried over well from the traditional application development world. The data world, in my experience, given the emphasis on collecting and analyzing after things have happened, does not share all of the TDD and CI/CD mindsets. I have, however seen indications, with things like Azure Data Factory’s pipeline/environment support, that this is changing. I’m sure there are other examples as well.

What makes this all the more interesting is the desire for a shorter feedback loop between analyzing data and making decisions based on that analysis. As the pace of rapid/automated decision making increases, which turns analytics decisions into system integration actions/units of work, solving these problems gets more interesting.