Our partners and customers are building forward looking planning applications. They are usually building these systems on top of existing transactional data systems and data warehouses. The problem is they are trying to build forward looking planning systems with every piece of data they have. This is a computational nightmare.
Forward looking planning systems are mostly like looking for directions on Google or on your GPS system. When you want to go from your house to the mall, you enter the address of the mall and within seconds you have the fastest route. Forward looking planning systems use data from a variety of systems and have to build the map before they can find the fastest route.
The issue is that if you decide to build the map with every detail, it takes longer than you can wait before you have the “map” and can find the route. At some point, these systems need data that is grouped or aggregated into logical blocks so that you can effectively plan.
Example
We are building one system that involves logistics between manufacturers and their retailers. The “raw” data comes in as a listing of every product in every container type. This is literally thousands of SKUs and makes our “map” very big. To generate the first version of the map and find the best route, our first cut of the system took 22 hours. As soon as we finished, the partner asked to change a price on one SKU and re run the model. Oops there goes another 22 hours. After looking at the situation and applying some discipline, we were able to come up with logical groups or containers of many SKUs based on their characteristics as related to shipping, packaging and storage. This aggregation allowed the model run time to be reduced to less than 2 minutes and allowed this system to be used in real time planning exercises and negotiations which was its intended purpose.
In our experience all forward looking planning involves looking at dozens of versions of the plan or “scenarios”. The good news is that technology has evolved to support this kind of planning. Within minutes, you can tweak a plan, find the best route and push the data into any kind of a reporting structure. The key however is having a model that turns over quickly. This happens when you aggregate the data.
How do you aggregate?
You aggregate by thinking about the problem and finding the common characteristics in the different data so you can logically group in a manageable size. One of the main issues with not being able to aggregate data is that you are trying to solve too many problems at once. Take the time to go through a design exercise where you look at the problem and describe what the ideal data for the system is. From there think about what transformations to your data are needed and possible to get to that “ideal” state.
Data aggregation is not easy. It requires discipline and an understanding of the problem you are trying to solve with your forward looking planning system. This understanding of the problem only comes with hard won domain expertise and the ability to synthesize the data, the process and the goal into a coherent, working model.
Also, most people think that aggregation is a one way trip. It is not very often we have created data transformations that aggregate data for planning purposes, then use it for planning, and after the forward looking planning is done, they “blow the data back out” into its detailed form for reporting and post solve analysis. This is especially useful if you are doing dimensional analysis with your data.
In Conclusion
I once heard comedian Steven Wright tell a joke about how hard it is to fold a map of the world scale 1:1. Data aggregation is a very important step in the planning process. Good aggregation is the difference between a good planning system and something unusable.
