Modernising the Data Platform for AI and Scale
Defining future-state capabilities for a flexible, performant and cost-efficient platform supporting trusted data, analytics and AI services.
ROLE
Technical Business Analyst
DOMAIN
Data Platform · Cloud · AI
Project Overview
I am contributing to a large-scale enterprise data platform transformation focused on creating a more flexible, high-performing, cost-efficient and AI-ready foundation for data, analytics and AI-enabled services.
The programme brings together platform modernisation, cloud-agnostic technologies, streamlined data processing and consistent, AI-enabled tools and services for engineering teams. It aims to reduce vendor dependency and unnecessary compute costs, simplify how data moves through the platform and enable teams to deliver trusted data and business value more quickly.
The transformation also establishes the reusable capabilities, standardised patterns and governance controls needed to support evolving data and AI workloads safely, reliably and at scale.
SCALE
Enterprise Transformation
The Challenge
The existing cloud-based platform had evolved through different technologies, tools and delivery approaches, resulting in duplicated capabilities, inconsistent engineering practices and complex data-processing journeys. Data passed through multiple transformations, increasing compute usage, delivery times and operational costs, while reliance on provider-specific technologies limited flexibility and increased the risk of vendor lock-in.
The platform needed to evolve to support a broader range of processing patterns, workloads and AI-enabled capabilities. This required a more flexible, scalable and observable foundation, supported by consistent engineering tools and services, reusable platform capabilities, and the governance and security controls needed to make trusted data available safely and efficiently.
01 Increase Flexibility and Cost Efficiency
Reduce reliance on provider-specific technologies, improve technology choice and enable more efficient use of storage, compute and platform resources.
Key Objectives
02 Simplify Data Processing
Reduce unnecessary transformations, data movement and duplication to create more efficient processing journeys and make trusted data available more quickly.
03 Improve the Engineering Experience
Provide consistent, AI-enabled tools and reusable platform capabilities that help engineering teams deliver data products faster and more consistently.
04 Enable Trusted Data and AI at Scale
Establish the scalable platform foundations, governance controls and security patterns needed to support evolving data, analytics and AI use cases safely and reliably.
My Contribution
Defining Future-State Platform Capabilities
My role is to translate the programme’s strategic objectives, emerging use cases and technical considerations into clear future-state platform capabilities, features and high-level requirements.
Working across architecture, engineering, security, data governance and AI governance, I bring structure to complex and sometimes ambiguous areas. My focus is on defining what the platform needs to enable while keeping requirements sufficiently solution-agnostic to preserve architectural flexibility and avoid prematurely prescribing the implementation.
My work spans four connected areas:
01 Agentic Access & Governance
For the emerging agentic access model, I translated business, operational and governance needs into a structured set of future-state platform capabilities. These covered agent identity and ownership, authentication and authorisation, delegated authority, access boundaries, human oversight, observability, auditability and accountability.
I brought together architecture, engineering, operations, security, data governance and AI governance stakeholders through workshops and playback sessions. I challenged assumptions, identified gaps and overlaps, clarified ownership boundaries and structured the resulting needs into capabilities, features, user stories, acceptance criteria and behavioural scenarios. This created an implementation-agnostic framework to inform roadmap planning, solution design and future delivery.
02 Data Product Access Discovery
The Data Product work focused on establishing a shared understanding of how enterprise Data Products should be defined, discovered, accessed, consumed and governed. I explored ownership, metadata visibility, governance controls, maturity thresholds, access entitlements, authorisation boundaries and approval responsibilities.
I consolidated agreed definitions, stakeholder perspectives, assumptions, dependencies, risks and unresolved design questions into structured discovery and current-understanding artefacts. These clarified how Data Products should be represented, governed and accessed, providing platform engineering and infrastructure teams with a shared foundation for implementing the tables, metadata, entitlements and supporting controls needed for secure downstream consumption.
03 Platform Performance & Cost
I assessed the platform’s future performance and cost-management needs by establishing the problem statement and documenting the existing operating context, pain points and risks. The analysis covered performance and scalability, compute and storage costs, monitoring and observability, and workload and capacity management.
Through workshops with platform engineering, architecture, infrastructure, cloud operations and FinOps stakeholders, I mapped the findings into high-level capabilities, supporting features, user stories and acceptance criteria. I prioritised these requirements according to the risks and pain points they addressed, creating a solution-agnostic foundation for future platform design and delivery decisions.
04 Streaming
I examined how the future platform could support streaming alongside batch processing without creating divergent data paths, duplicated processing or inconsistent standards. The existing platform primarily supported batch workloads, with streaming handled through bespoke solutions, creating risks around data freshness, representation, governance, and consistent downstream consumption.
Working with platform architecture, data engineering, infrastructure, operations and governance stakeholders, I documented the current state, risks and pain points before mapping them into future-state capabilities, features, user stories and acceptance criteria. These covered unified data flows across batch and streaming, flexible arrival patterns, batch and continuous runtime models, consistent failure and recovery, end-to-end observability, and common governance standards. The resulting requirements gave architects a solution-agnostic basis for assessing implementation options and making architectural decisions about how streaming capabilities should be introduced into the platform.
Business Impact
My work created a clearer and more structured foundation for future platform decisions, enabling complex business, operational and governance needs to be considered before technical solutions were selected.
Better-Informed Architecture Decisions
Provided architects and engineers with prioritised, solution-agnostic capabilities and requirements against which implementation options could be assessed.
Stronger Cross-Functional Alignment
Created a shared understanding across architecture, engineering, infrastructure, operations, product and governance teams, helping stakeholders align around platform needs and responsibilities.
Reduced Design and Delivery Risk
Identified gaps, dependencies, ownership boundaries and unresolved decisions early, reducing the risk of duplicated capabilities, inconsistent controls and divergent platform solutions.
A Foundation for Sustainable Scale
Supported the development of a more flexible, observable and governed platform capable of handling evolving workloads while improving cost control, engineering consistency and readiness for data and AI use cases.