Data and Energy: Parallel worlds, both on shifting ground
I’ve been living in the data world for many years, and in the last three years I’ve done a lot of work at the intersection of data and energy. At this intersection, data is fundamental to the many transformations happening in energy (i.e. the 4 D’s: digitalization, democratization, decarbonization and decentralization). You simply can’t handle the latter three D’s without a fundamental shift toward harnessing data to drive faster, smarter, more automated decisions and operations, i.e. digital transformation. Much has been written about this so I won’t belabor it here (for those who are less familiar with the 4 D’s, this article that provides an overview and updates on the topic.) But lately, prompted my current work in the data realm with my friends at StreamSets, I’ve been thinking more about the commonalities between the two sectors as they both go through tectonic shifts in how they fundamentally operate.
Both energy and data are highly complicated networks of connectivity and flow (whether electrons or bytes) that are going through multiple-orders-of-magnitude steps up in pace and complexity. In the energy world, we have:
Exponentially increasing distributed energy resources, whether behind-the-meter or distribution-level-- many of them not in the direct control of the utility or energy company which needs to use and connect them
New technologies across many areas: storage, renewable generation, EVs and EV charging equipment, smart home/building management, distributed energy resource management systems (DERMS), etc.
New use cases: vehicle electrification; building electrification; microgrids; virtual power plants; non-wires alternatives; and more
Changing markets, regulations and incentives
In the data world, we have:
New data sources everywhere, from web and mobile apps; sensors, equipment and other IoT devices; social media; data-as-a-service’s-- many of them not in the control of the company/organization which needs to use and connect them
New technologies: AI/machine learning; computing languages and platforms; data platforms; data processing, movement and wrangling tools, etc.
New use cases: genomic research; cybersecurity; real-time pricing optimization; personalized offer management; predictive maintenance. Te list goes on and on.
Changing regulations, especially with respect to data privacy
But these exponential changes are colliding head-long into the old way of doing things that simply wasn’t designed for the pace and complexity of modern times. In both worlds, the traditional approach for managing these complex systems has been a command-and-control structure:
Build centralized sources (power plants in energy; enterprise data warehouse and large enterprise applications such as ERP and CRM in data) that are highly controlled and designed to last for many years, even decades
Set up stable and secure ways to distribute the resource to downstream users (the electrical grid sends energy; integration tools move data). Those distribution systems include large networks of relatively dumb pipes just designed to get things from point A to point B (poles and wires in energy; pipelines in data, which don’t have much brains built into them.
Establish an overall architecture that is designed for stability and security, and assumes that changes are incremental.
Staff a team dedicated to controlling the system (the control room in energy; the data management team and/or integration center of excellence in data)
Address changes via a regulated process where users or project teams must justify the need for the change, and processes are in place to minimize and mitigate impact on existing operations.
In short, the way that organizations historically ensured the safety and stability of their business operations was to build rigid systems that were highly controlled and rigorously hardened to ensure they would function as needed. Such a rigid system works fine when things aren’t changing much, just like a brick building can stand for hundreds of years as long as the ground is stable.
But in both data and energy, the sheer number of things has grown by at least three to five orders of magnitude versus just a decade ago, whether that is petabytes of data or gigawatts of distributed energy resources. The new sources of data and energy are proliferating, and many of them are controlled by others-- thousands and millions of other people, companies and organizations. Most importantly, things are now constantly, unpredictably, uncontrollably changing, and the pace of that change is only accelerating.
The old, rigid architectures and practices that were designed for stasis simply break down in the face of such torrential change. Unexpected changes and events, often caused by a third party, lead to breaks and outages that are difficult to discover and diagnose, much less fix. Existing change management processes cannot keep up with rapidly evolving business demands. Architectures become so complex with layers upon layers of incremental changes (often done as band-aids) that they become incomprehensible and risk collapsing under their own weight, like a brick building when the ground shifts due to an earthquake.
Those who try to fight the inevitable changes are engaged in a losing battle. The changes are simply inexorable and unavoidable (unless you simply go out of business). The only path forward is to design your architecture and practices to anticipate and gracefully handle change, in as automated a manner as possible, so that the system can evolve with the times. Just like modern earthquake-resistant buildings are designed to flex, sway and absorb the roll and pitch of ground moving beneath them, and still stay standing.
Design principles that go into a both a modern energy and a modern data system are:
Massive scalability. The need is obvious. But it takes modern technology to scale to today’s processing requirements. And as things get even bigger, faster and just plain “more” in the future, new technologies will be required-- and you will need to be able to adopt those, too, quickly. Hadoop was the end-all and be-all technology for big data for about a decade, and now it’s already considered outdated. If your technology, be it your data management tools or your DMS (distribution management system in energy) is over a decade old, it’s safe to assume it’s going to hit the wall, hard, with respect to scale.
Automation, automation, automation. If every little thing needs human intervention, you’re just not going to make it. New connections need to be created and changes made in an automated fashion with intelligent tooling. Systems should be managed and optimized leveraging artificial intelligence. This doesn’t mean humans are out of the equation-- but they need to focus on the most critical things that require human creativity, and let automation take care of the mundane tasks.
Instrumentation and embedded intelligence throughout. Things that were formerly dumb like electric poles and wires, or data pipelines, need to be instrumented with sensors to that it’s clear what is happening in them in real time. Better yet, they should be continuously monitoring, self-sensing and self-healing, with built-in intelligence to make adjustments, handle problems or issue alerts for additional intervention.
Abstraction: In a complex network of flows, end points need to be abstracted, whether it’s a battery storage system or a web application. Because you know it’s going to change, and if it changes, you don’t want everything else connected to it breaking. Abstraction is a critical, although sometimes esoteric, tool in an architect’s arsenal.
Technology neutrality. Technology languages, platforms, standards, software and hardware will continue to proliferate and change. You need to be able to work across different technologies because again, you can’t control where it all comes from. And if you’re neutral to specific technologies, you can much more easily adopt new ones that advance your practice or serve a growing business need. Of course, no one can be 100% neutral-- you need to make some platform-level bets-- but look for key signifiers of neutrality such as abstraction layers (per the point above), open standards, APIs and potentially even open source technologies. This is one area where the data world is far ahead of the energy world, where open standards are significantly lagging and need a major boost.
Pervasive policies. Just because things are changing fast doesn’t mean you can let things be ungoverned and unregulated. Especially when there are more security and integrity issues than ever. But your approach to governing the system and enforcing policies and rules has to change. You certainly can’t just insist that everything is controlled from the center. Moreover, policies can no longer be manually enforced and monitored. Policies and rules have to be automatically applied by the system itself, using embedded intelligence, and the instrumentation throughout the system has to sense potential issues or violations-- it can’t just be a monitoring and management system in central operations or a perimeter-based approach.
In short, in the rapidly transforming worlds of both data and energy, to ensure you have a reliable system, you can no longer design for stasis. A design approach that fundamentally embraces constant change is the only way to deliver reliably in our modern world. And the only way to stay ahead.