As a listened more of the Ask me anything with Udi Dahan session organised by Virtual DDD, more points, which Udi was making, especially about Event Sourcing, made my fingers itch to wrote some more.
When talking about the natural progression in system evolution once it’s in production, Udi gave an example of adding columns in the read-model as a consequence of more properties to some domain objects. That particular example was about the search feature. Although the example might be superficial, I really love search examples in context of Event Sourcing.
So, Udi concludes:
The problem is: there’s nothing in CQRS and Event Sourcing that makes any of that any easier. In many ways, the technical design choices exacerbate the pain of those types of changes.
Further on, Udi says that developers often underestimate the future volatility and complexity of the data model, and how it would impact the previously made choice of an underlying database technology.
Well, what if I tell you that using Event Sourcing and proper domain modelling will save the day when you hit a massive requirement to search everything by everything?
Models
Let’s start with the model aspect. It’s quite naive to assume that all the attributes for a product, which your customers will be using as a search criteria, would ever come from a single bounded context, or a single aggregate. Having a gigantic model, which combines the product catalog (which is almost static), the pricing model, the inventory management, is the road to hell. All those things operate on different concerns and could be cleanly separate to their own models, which live independently (more or less) from one another.
But then again, what happens if you, indeed, need a single search model? In an event-sourced system the solution is obvious. You can create a read-model, which will consolidate all the necessary information about a single product in one database element. It could be a table (not sure if it’s a good idea), or a document. You can start by experimenting with one document per product, then you start separating them to individual variant, as it makes the search easier (that particular design, one colour, one size).
Moving from one reporting model structure to another can be done perfectly without affecting the currently running system. After you decided that the one product per document model won’t work in a long run, you build a new set of projections. Those projections will have the new model, but the current one will still be running and know nothing about its sibling, which is going to replace it. You can run load tests, user testing, A/B testing, whatever you want, at any time, in production, and no one will ever notice.
Further on, you can later decide that your current database, where the reporting model resides, doesn’t really fit the purposes of search any longer. Here we have a choice.
One option would be to subscribe to the MongoDB change feed stream and shovel documents from there to another database, as-is. Say, we decided to use ElasticSearch for the search model. I’ve done exactly this before, and it worked, until we found out that in MongoDB the change feed has a limited size and time to live. If your replication component gets down for a prolonged period of time, some changes will not propagate to the search model, making it essentially inconsistent. It can be fixed by introducing a new field on all the documents stored in MongoDB, a timestamp of the latest update, for example. All the services, which write to MongoDB need to get it implicitly implemented. Then, the replication component would use a query to get the latest changes. Once again, I’ve done it before, and it worked splendidly. But it only makes sense, if you intend to keep both models for some reason. For example, you use MongoDB for most of the queries, but not for search. Both reporting models must be identical, or almost identical.
If you don’t want to keep two identical reporting models in two different databases, as the whole model only aims to solve the search issue, you can just build a new set of projections, which will project directly to Elastic.
Here again, this new set of projections can be built on a side to the current production system, deployed to production, run in production, be user-tested in production, without having any impact on the currently running system.
Under the normal circumstances, replacing a database for a particular data model would be a huge undertaking. People normally speak about such a change in terms of multi-months project. I argue that in an event-sourced system it’s way easier to implement such a change, as you don’t even need to consider it as a change. It’s a new set of components, which are completely independent of anything in the running system.
When those new projections catch up with all the historical events, all you’d need to do is to move the search API to use this new database. The UI part wouldn’t even notice the change. If the API scope is identical, both can run side-by-side for quite some time, until you ensure that the new model is indeed better than the existing one. If not - you can just scrap it.
Conclusion
I’ve heard quite a few times the statement that Event Sourcing sucks for reporting. It is quite the opposite. I’ve worked with many production systems, which suffered from having a single store for transactions and reporting. Separating the reporting model is not a new idea, but in an event-sourced system it’s much easier to implement.