Charging Ahead with Data Mesh: A Modern Solution for Chargefox's Infrastructure Analytics

Chargefox is Australia’s largest and fastest growing Electric Vehicle (EV) charging platform. Formed in 2017 to help meet the need for better EV charging infrastructure, innovative technology to support it, and software to make it available to everyone. Chargefox is committed to making charging simple, affordable and fast for everyone. Because simpler charging means more EVs on the road!

Chargefox makes hosting and managing chargers easy for companies, partners with fleet managers, vehicle manufacturers and hire-car providers to keep drivers charged. Chargefox also works with local councils and governments all over Australia to provide EV charging solutions and fleet management services.

Building the platform is central to achieving the organisation’s strategic goal of becoming fully integrated into the infotainment and navigation systems of leading vehicle manufacturers, providing a seamless experience for all EV drivers.

The Challenge

Chargefox is continuing to scale at an accelerated pace to support the roll out of electric vehicle charging infrastructure across Australia. This resulted in the organisation quickly outgrowing the previously suitable data solution and strategy.

As the organisation scaled, the Chargefox team had articulated that their existing data repositories were primarily designed to support the platform’s features, rather than their reporting and analytical needs.

More specifically, Chargefox’s first data strategy implementation created analytics that connected directly to the platform’s database and only allowed the team to analyse the data in those existing structures and forms. This included an integration with QuickSight that was directly connected to the main Postgres Database.

Figure 1. The first data strategy architecture design at Chargefox

The strategy’s design, whilst initially a great MVP, included some limitations:

• QuickSight is an analytical and reporting tool that assumes data sources contain the data Chargefox requires without transformation. To provide meaningful insights, Chargefox sometimes needs to augment and transform data to make it fit for purpose.
  

• The data source QuickSight connects to supports the Chargefox platform’s feature set and frequently changes as new features and enhancements to existing features are built. A lot of care needs to be taken by product teams to ensure compatibility of reporting and dashboards as underlying data sources change to support platform progress.
  

• Chargefox customers often need access to the data and reports generated by QuickSight. This can often result in additional operational and product team effort to securely generate and send these reports to customers.
  

Midnyte City was working with Chargefox when a question of the next stage of development in their data platform was raised. A small co-sourced team from both organisations collaborated to form patterns and capability the Chargefox product teams can extend as they continue to evolve the platform and the business.

The brief was simple. Chargefox needed a new data strategy and capability to have consistent datasets that are:

• Reportable, consistent, accurate, reliable and live
  

• Accessible by those who should have access, using the tools they choose
  

• Centrally accessible
  

• Provide independence for the product teams to develop new and evolving solutions, without affecting reports or external data dependencies
  

• Allow Chargefox to transform and augment information as needed
  

The Solution

The Midnyte City crew were very excited to be working with the Chargefox team addressing secure and scalable data management by building out a Data Mesh.

Data Mesh or Data Lake?
‍The Data Mesh approach was decided upon as it pushes the ownership of data back to the producing team who understands the data in their domain, such as a product team. This also helps to alleviate the need for a team sitting between producers and consumers of data servicing requests.

Traditionally, a Data Lake relies on a team that will sit in between the producers and consumers of the data. The team itself effectively does the work of sourcing and providing the system data into the data lake.

The goal of a Data Mesh is less about the implementation and technology involved in delivering the data, but more about the ownership of the data between producers and consumers.

Domain and Data Ownership and Governance
In order to enable the producers of the data further autonomy and ownership, and to simplify any cross-account and team access to this data, the team defined some high-level patterns of producing and visualising any unstructured data.

The team pushed for each domain’s solutions to be implemented via re-usable Infrastructure-as-Code (IaC) constructs that the wider product teams can bootstrap and then point their data and eventing at. Which would then be published to a central data repository.

This then allows the consumer and centralised catalogue context to be completely decoupled from the producer solution, as well as the producer context being encapsulated and completely decoupled from the product solution. Heavily reducing the operational overhead required by the product teams, as well as increasing the re-usability of the IaC constructs as the team’s solutioning continued to evolve.

At a high-level this looks something akin to:

Figure 2. A high-level design for the Producer, Centralised Catalogue and Consumer Domains used in the Chargefox Data Mesh implementation

Existing Datasets (Postgres)
In order to assist in decoupling and continuing to support the existing Postgres datasets, the team needed to ensure that the RDBMS, served via RDS, was completely decoupled from the visualisation tools. But continues to adhere to the proposed data mesh patterns.

With that, the team utilised AWS Database Migration Service (DMS) to capture snapshots of the data with a configurable cadence as per the needs of the data timeliness requirements. DMS was treated as a “producer” of data and as such assigned to the Producer Domain as part of the defined pattern.

Figure 3. The implementation design for the Producer and Centralised Catalogue domains for the existing Postgres datasets.

The Results

The resulting solution is a modern, scalable, leading-edge Data Mesh infrastructure for a fast growing organisation. The Data Mesh is a modern approach to data governance and ownership, with no central data team required to orchestrate and administer the production and consumption of data. The management of the data remains lean and can continue to be owned by the appropriate product teams.

The Mesh itself generates valuable insights and results for Chargefox, some of which include:

Visibility of Key Operational Data
The data mesh has enabled the visibility of additional, key operational data. As an example; one of the key metrics the team required further visibility of was the availability of EV charger connectors across the entire Chargefox charger fleet at any point in time.

The definition of availability within the context of EV charging can often change depending on a number of factors. The decoupled nature of the Data Mesh design, especially from the product team solutions and datastores allows the Chargefox team to be able to rapidly augment the visualisation and generation of reports of availability as needed.

Generation and Automatic Sending of Operational Reports
Further, the Mesh implementation has also unlocked a simplification of operational tasks and reporting.

One such example is automatically generating and sending charge station incidents reporting to the appropriate charge point operators.

The Mesh solution has greatly reduced the people hours required of the Chargefox operations and product teams to manually generate, orchestrate and send these reports to charge point contacts, as this has now been completely automated via the Data Mesh.