The New Data Governance Experience

In today’s landscape, characterized by the dawn of generative AI, businesses face a dual challenge: embracing the potential of AI while navigating the complexities of cybersecurity, regulatory demands, and their ever-growing data estates. In response to this convergence, Microsoft presents a blended approach for data governance and security, epitomized by Microsoft Purview. Released on the 8^th of April, we are at the dawn of a new era for the company’s Data Governance offering – one where AI is at the centre, surrounded by features asked by the largest customers.

Historically, Data Governance has been a secondary thought in executives’ minds. It has been, as shown in the above image, an “Essential Defense” that puts compliance and security at the heart of activities. In reality, there is a macro shift happening in the area; the realization that Data Governance can be a driving force in the role of democratization of data assets across organizations. With Microsoft Purview Data Governance, there is an opportunity to apply what is called “Responsible Offense” – a practice that brings together enterprise-wide users, with drastically different roles, to the practice of Data Governance. The image above aims to illustrate the participation of all stakeholders in the whole data lifecycle, which means that we begin to introduce a dynamic Data Governance practice, instead of the traditional static approach.

The Dawn of Federated Data Governance

Managing data across an entire organization demands a delicate balance between stringent standards and adaptable frameworks. While ensuring data cleanliness and security requires consistency, catering to the diverse needs of various teams necessitates flexible data access and management strategies – including Data Steward-level definitions. To address these intertwined challenges, Microsoft advocates for a federated governance approach—a methodology that centralizes core data governance principles while empowering teams with self-service capabilities tailored to their specific requirements.

At the core of federated data governance lies the principle of distributed ownership, spreading responsibility and engagement across departments. This decentralized model not only reduces operational bottlenecks but also fosters collaboration and active participation in managing, governing, consuming, and leveraging data assets effectively. BI4ALL and Microsoft, partner up to bring this new Purview experience to life, by addressing not only functional and not functional requirements, but also the Enterprise perspective of data.

In the new Microsoft Purview experience, there are three stakeholder categories that will aggregate value from the experience:

Data Consumers Across the Organization:

• Streamlined Data Discovery: Intuitive tools and interfaces to easily locate and access relevant data assets.
• Enhanced Data Security: Tailored access controls and security measures to safeguard sensitive data.
• Insightful Data Context: Comprehensive data descriptions and metadata to aid in understanding data context and relevance.

Data Owners and Stewards:

• Efficient Data Curation: Tools and workflows for curating and managing high-quality, usable data assets.
• Governance-driven Data Use: Enforcing governance policies to ensure ethical and compliant data utilization practices.
• Data Health Monitoring: Tools for identifying and addressing data quality issues and anomalies proactively.

Data Officers and Leadership Stakeholders:

• Maximized Data Value: Strategies and insights to extract maximum value and insights from organizational data assets.
• Unified Data Management: Standardized governance controls and accountability frameworks across the data landscape for enhanced data integrity and security.

The Rise of Data Products

The latest experience delivered by the Microsoft Purview Data Catalog revolutionizes how businesses interact with data. The intuitive categorization by business domains, which now leverages an AI-powered copilot for seamless searching, will enable data product subscriptions with comprehensive data sets and secure access tools. Before, the platform was mainly designed to be leveraged by IT departments. However, now Microsoft is introducing enhanced tools to efficiently manage expanding data volumes and create more avenues to leverage this data across an organization’s daily operations.

The new Microsoft Purview Data Governance experience is shifting focus into five pillars, all of which enunciate the importance given data products as a central component. A clear alignment with attributes like durability, quality and user friendliness is seen, and aligning with the BI4ALL approach, described later in the article. This is core to any implementation of a federated data governance model, and is better explained and detailed below:

• Data Access

Much like the previous Data Policies, the experience still lies in managing access for certain groups and personas, but now usage purposed can be defined – and these are now merged with DLP-type policies that allow for copies of the data or not. The experience is now interconnected with Data Quality and Glossary terms, as now there is a choice of dynamically assigning glossary terms to data products, and have permissions set for those same terms. These then trickle down to specific assets they are associated with, making it easier to control access.

• Data Curation

The use of Business Domains, which can be seen as different collections in the previous experience, now enable the distribution of assets based on organization-wide known concepts and also allow certain groups of people to publish content to those domains. For example, Finance Analysts only publishing to their Finance Domain. This is then coupled with a hierarchical layer, with Data Products opening the door to grouping various tables and dashboards, for example. This means that analysts and other stakeholders can save time requesting access and empowering Data Owners to create more value from multiple assets – democratizing their own work.

There is also an integrated data health action feature, which authorize different severities to be assigned to actions that need to be taken in order for a data asset to be adherent to rules that have been set by the organization, or by regulatory standards.

• Data Discovery

Although this component stays close to its predecessor, it is now enhanced with AI, much like all other Microsoft Copilots work. Managing self-service requests and explaining what the user is looking for in the catalog  is now a feature of AI-powered Data Governance, with the purpose of identifying the exact needs for the work that requires completion.

• Data Health

The highly anticipated Data Quality feature interconnects various components that feed into the Data Estate insights report, with a combination of dashboards and OKRs (Objectives and Key Results). With DQ, we can now create our own rules, or use out-of-the box rules, to scan our sources and create scores and actions to take with the aim of improving the quality of our assets. We can now drill-down into column-level quality and profile data in order to enable automatic job monitoring based on alerts that notify data owners and stewards of specific actions to be taken.

Data quality dimensions being measured out-of-the-box (completeness, consistency, conformity, accuracy, freshness, and uniqueness) are pivotal. We couple this with OKRs so that there is a bridge between our Data Governance practice and the business goals. This is a way of generating ownership from objectives that are specific and actioned upon by enforcing actions within the portal itself.

• Data Understanding

All components of the Microsoft Purview data catalog combine to provide a larger context to an organization’s data. From glossary terms that have their own access controls to OKRs and Data Products that connect business goals to business data, the intricate details of the catalog are just descriptors that enable all data users to make an informed decision of whether to use of not to use certain assets during their day-to-day at the company. It is important to work with partners like BI4ALL to begin a journey of understanding data management – a journey that starts with the Data Strategy & Governance CoE and runs for the entirety of the company’s data & AI-driven path.

Bridging Business and Data Understanding

Glossary terms serve as a vital link between technical complexities and everyday business language. They empower users to explore and engage with data using familiar vocabulary, enhancing comprehension, and enabling more informed decision-making based on data insights.

Objectives & Key Results are the guiding stars that align business actions with specific outcomes. Whether it’s achieving a sales increase or reducing support cases, OKRs ensure focused efforts towards measurable goals. Complemented by proactive Data Estate Health Actions, organizations not only enhance data governance but also prioritize data improvements effectively, fostering a culture of trust in data-driven initiatives essential for sustained business growth. This effectively leads to a unified transition of Data Management, which is illustrated below with the integration of Microsoft Fabric, Microsoft’s Analytics SaaS platform, and Purview.

It is vital to mention that, with the release of Microsoft Fabric, released in mid-2023, the focus on Single Source of Truth is very much a priority. This will enable a more native approach to a federated data governance model, where lineage will be seamlessly integrated between both platforms, and where holistic data estate realities will reflect in real-time. Through the correct approach, organizations should have a healthy overview of how data is being used internally. However, this will require a guided maturity assessment, based on international best practices; where partners like Bi4All can help, with deep expertise and a custom-built model that adapts to each organization’s needs.

No more silos: leveraging the BI4ALL step-by-step approach to enable the adoption of MS Purview at scale while maximising business benefits.

The new Microsoft Purview offers significant value to any organization prepared to advance its data maturity. Typically, organizations initiate their exploration of new technologies with a Proof-of-Concept (PoC). However, successful PoCs often result in adoption by only a limited number of individuals, primarily data specialists, while the key to maximizing return on investment in Microsoft Purview, lies in fostering sustainable adoption by both technical and business users, in alignment with best practices and the enterprise data strategy, to achieve broader business objectives for the organization.

In response to this need, BI4ALL’s Data Strategy & Governance Center of Excellence has developed the Data Strategy & Governance Scalability Framework (DSGSF), a groundbreaking blueprint designed to drive organizational data transformation. This framework, when effectively implemented, facilitates seamless integration and optimal utilization of technologies like Microsoft Fabric and Microsoft Purview on a significant scale. By aligning with the DSGSF, organizations can ensure the achievement of a democratized Enterprise Data Strategy, establish robust Governance foundations and consolidate their position at the forefront of innovation. In fact, Microsoft Purview, boosted with the latest capabilities, serves as pivotal enabler for data maturity and governance scalability across the organization. However, it is essential to establish strong data strategy and governance foundations before adopting it.

In more detail, the DSGSF, depicted in the image below, delineates the components necessary for progressing through incremental stages of data maturity. These stages correspond to ascending levels of data governance sophistication, spanning from foundational to federated, decentralized, or mesh styles.

The DSGSF is operationalized by the BI4ALL step-by-step approach, which is broken down in 10 manageable steps. This ensures that sponsors stay in control of each step and aligns with consulting data quality assurance processes.

The approach includes:

• One step dedicated to synchronizing efforts between BI4ALL and the Customer.
• Three steps for conducting a Data Maturity Assessment, establishing both qualitative and quantitative foundations for the Customer’s data strategy. Alternatively, these steps can be replaced with a more lightweight audit, focusing solely on qualitative results.
• Two steps for collaboratively designing and implementing the data strategy, backed by a compelling business justification and other ad-hoc deliverables. A lighter data strategy option can be delivered if the Customer prioritizes their focus on Data Governance.
• An architecture workstream tasked with mapping systems, databases, and applications, as well as constructing key conceptual data models and data domains.
• One step for designing the data governance strategy.
• One step to accelerate the adoption of Microsoft Purview.
• A training and knowledge workstream to ensure effective utilization of the implemented solutions.

Follows a more detailed description of each step and workstream.

Step 1: Data Strategy & Governance Stream – Sync

This step consists of 4 sub-steps:

1. Initial Interviews: BI4ALL carries out a small number of interviews to align with the Customer’s current status in relation to data.
2. Review of Documentation: BI4LL reviews available documentation, including data strategy documents, past maturity assessments, existing Data Governance & Quality practices, and any other relevant materials.
3. Scope & Assumptions Review: Following the elements discovered in step 1.1 and 1.2, BI4ALL reviews scope and assumptions with the Customer’s Project Committee or Sponsor, to adapt the step-by-step approach to the Customer’s specific needs.
4. Kick Off Meeting: This is a formal meeting to mark the beginning of the project and initiate collaborative efforts towards successful design.

Step 2,3,4 – Data Maturity Assessment (DMA) or Audit

BI4ALL’s Data Maturity Framework

While several maturity assessment frameworks exist, such as the Data Management Capability Assessment Model from DCAM® and the Enterprise Information Management Maturity Model from Gartner®, BI4ALL has developed a unique data maturity assessment. This assessment combines the strengths of existing methodologies and addresses their inherent gaps.

The assessment is modular and tailored to meet the specific needs and objectives of the Customer.  The DMA acts as a robust diagnostic tool, enabling BI4ALL and its Customers to thoroughly evaluate the current landscape of data management, governance, and infrastructure across various domains. Through meticulous analysis of maturity levels, it facilitates the identification of strengths, weaknesses, and gaps in data practices, technological deployment, and compliance adherence The DMA, acting as a strategic roadmap, assists in defining pragmatic goals and effectively prioritizing initiatives. It ensures that subsequent solutions, like the integration of Microsoft Purview, are tailored, scalable, and sustainable.

This assessment provides Customers with a comprehensive view of data across the enterprise, ensuring alignment between business objectives and strategic data management plans. It offers an unbiased evaluation of critical facets of the Customer’s data initiative through various lenses, encompassing compliance with data protection regulations, formulation of data strategy, implementation of data governance, adoption of technologies—including Generative AI and AI Governance—enhancement of data quality, and more. Furthermore, it sheds light on the organization’s data practices, enabling targeted improvements and uncovering hidden insights that are vital for the development of data-driven services.

Step 2

In this phase, BI4ALL takes the lead in conducting the Data Maturity Assessment (DMA). The format of this assessment may vary, but typically, it is facilitated in small groups led by a Senior Data Governance Specialist. In instances where multiple divisions or data domains are engaged, workshops are organized accordingly, either by divisions or by data domains. Each session will accommodate no more than 7-8 participants to ensure opportunity for comprehension and interaction, allowing everyone to engage fully and address any queries that may arise.

Step 3

In the third phase, BI4ALL engages in one-on-one interviews to validate the findings of the DMA and explore further specific requirements, use cases, and pain points. This step is pivotal in pinpointing priority scenarios for the effective implementation of Microsoft Purview, enhancing the alignment of solutions with the organization’s distinct goals and challenges.

Step 4

In this phase, BI4ALL compiles the DMA’s results and recommendations, which are then presented to the nominated key stakeholders. The DMA’s recommendations are underpinned by the expertise of BI4ALL’s seven Centres of Excellence (see picture 3), providing Customers with the assurance that recommendations stem from seasoned leaders who have instituted best practices and accumulated years of invaluable experience. This collaborative approach ensures that Customers receive recommendations grounded in expertise, tailored to their specific needs, and poised to drive meaningful transformation within their organizations.

Audit

At times, data sponsors may opt against conducting a Data Maturity Assessment (DMA), particularly if they’ve recently completed one. In such cases, BI4ALL offers an alternative solution through a targeted audit. This audit serves to pinpoint data-related requirements, identify use cases, and uncover pain points, all of which serve as inputs for crafting an effective data strategy.

Step 5 & 6– Data Strategy & Governance Design and Development

After presenting the DMA’s results and considering insights from the Architecture stream (discussed in the following paragraph), the Customer gains a clear understanding of the challenges inherent in its unique structure. This knowledge lays the foundation for the upcoming Data Strategy & Governance workshops. These pivotal sessions enable the Customer’s decision-makers and their trusted advisors to collaboratively design a forward-thinking approach to data management.

BI4ALL expertly orchestrates each workshop, fostering discussions, leveraging proprietary frameworks, accelerators, and technology demonstrations, including Microsoft Fabric and Microsoft Purview. These tools trigger discussions leading to solutions that address existing pain points and enable Customers to explore potential opportunities. Subsequently, a prioritization exercise involving senior stakeholders evaluates the significance of identified pain points and use cases gathered from previous engagements.

The project team seeks consensus on the pivotal elements of a visionary Data Strategy and Data Governance Policy. This includes discussions on domain topologies and the operational dynamics of the Data Office in conjunction with other departments. These strategic components form the cornerstone of the revamped Data Organization Model, ensuring alignment of the Customer’s data management practices with its overarching objectives and multi-stakeholder structure.

The amalgamation of insights gathered thus far empowers BI4ALL to tailor the Data Strategy Deliverables to meet the unique needs of the Customer precisely. Notably, a critical deliverable of this phase is the business justification for any proposed data initiatives outlined in the Data Strategy. If it will be a requirement, federated data governance could be fast-tracked by the BI4ALL Data Mesh Framework, preparing the Customer to design, create, implement, and continuously improve data products. In this last case, Microsoft Purview’s new functionalities for data products could be certainly leveraged.

The collaborative co-generation approach described above attests to BI4ALL’s experience and has proven highly effective in securing commitment from key stakeholders towards embracing the new strategy.

Data Architecture Stream

The data architecture stream is paramount to the Customer’s operations, serving as the anchor for their data management, governance, and personal data protection strategy.

To deliver a comprehensive and future-proof solution, BI4ALL recommends implementing four layers of mapping:

• Enterprise Data Systems Mapping: This layer provides a holistic view of the organization’s data ecosystem, enabling a thorough understanding of the interconnections between different data systems.
• Enterprise Data Architecture Mapping: These maps offer a structured blueprint for data management, ensuring alignment with strategic objectives while promoting data consistency and governance.
• Entity Relationships diagrams: Functionals to illustrate data entity relationships, aiding in understanding data flows, dependencies, and hierarchies to enable data lineage.
• Key Conceptual Data Models Mapping: Crucial for visualizing how critical data entities relate to one another, these models facilitate ownership of data governance and enhance data quality.
• Domains Map: This map, constructed from all collected inputs, forms the bedrock for the deployment of Microsoft Purview and Federated/Data Mesh style Data Strategy, ensuring coherence and effectiveness in data management throughout the organization.

Step 7, 8, 9 – Data Governance Foundations

Foundational Data Governance includes enabling the Organization to change the way it operates to stay in control of its own data. By harnessing Data Governance Accelerators, BI4ALL empowers Customers to expedite deliverables while ensuring adherence to global data standards, bolstered by over 19 years of BI4ALL’s industry-leading expertise.

At Step 7, the cornerstone deliverable is the Enterprise Data Governance Policy. This policy serves as a conduit for the Customer’s Data Leader to gain and commit Senior Sponsorship.

In Step 8, the focus shifts to crystallizing the objectives of the Data Governance Initiative in alignment with the Data Governance Policy. A hallmark deliverable during this phase is the Data Governance Roadmap. Additionally, other vital deliverables encompass:

• A Data Governance Operating Model, designed to operationalize the Enterprise Data Governance Policy effectively.
• A Data Governance Framework, serving as a graphical tool in the hands of the Customer’s Data Leader to succinctly convey the core elements of the Data Governance Initiative to the broader organization.
• A tactical plan to start small the initiative and scale it afterwards. For example, from a reporting project on General Ledger, it is possible to craft a plan that will make the Financial Department the example for data governance to be followed by other departments.

In Step 9, all the plans from preceding steps are brought to fruition. This phase may entail not only technological endeavours but also the establishment of business and technical definitions, formulation of data policies, alignment of individuals with data governance roles to activate the Data Governance Operating Model, conducting data lifecycle reviews, and identifying metadata—laying the groundwork for Microsoft Purview endeavours.

Step 10 – Microsoft Purview Kick-Start

BI4ALL approach to kick-start Microsoft Purview aligns Microsoft issued deployment best practices available in this article: Deployment best practices for Microsoft Purview (formerly Azure Purview) | Microsoft Learn. The deployment best practices include the following stages:

1. Objective Identification: Clearly define the objectives and goals of data governance in alignment with Microsoft Purview.
2. Use Case Identification: Identify relevant and practical use cases for leveraging Microsoft Purview within the organization’s data ecosystem.
3. Integration Planning: Develop strategies for seamless integration of Microsoft Purview into existing systems and workflows.
4. Stakeholder Engagement: Foster open communication and encourage questions from diverse stakeholders to address concerns comprehensively and ensure alignment with all stakeholders’ needs.
5. Key Decision-Maker Involvement: Engage key decision-makers and data specialists throughout the deployment process to ensure strategic alignment and informed decision-making.
6. Deployment Planning: Develop detailed plans for the deployment of Microsoft Purview, considering factors such as resource allocation, timelines, and risk management.
7. Post-Deployment Security Measures: Implement robust security measures following deployment to safeguard data integrity and protect against potential threats.
8. Data Lifecycle Considerations: Address considerations related to the entire data lifecycle, including acquisition, storage, processing, and archival, to ensure holistic data governance.

By following BI4ALL’s Data Strategy & Governance approach, which encompasses the first five stages mentioned above, organizations can streamline the deployment process and transition seamlessly towards implementing Microsoft Purview. This approach ensures that the groundwork laid in the Data Strategy and Data Governance Strategy phases effectively informs and guides the deployment of Microsoft Purview, facilitating optimal utilization and alignment with organizational objectives.

Planning the implementation of Purview

Based on the new experience, Microsoft is linking a federated Data Governance model to a technology that enhances its benefits. Coupled with BI4ALL, the task is facilitated by leveraging workshops, assessments and culture trainings that enforce a fit between technology and theoretical practice. In a period of intense work, the implementation of Purview is suggested to run through 5 weeks – as per mentioned here in the documentation:

Stage 1 (Week 1-2)

Data Management

Catalog Setup – In conjunction with BI4ALL’s step 5 in their structured approach, we can align priorities and set domains for usage and data owners. This is where stakeholders define focus groups and scope out accountability towards certain products.

Catalog Curation – Mapping data assets to the correct structure, as defined in the initial steps of BI4ALL’s Governance Framework, providing a seamless experience when looking ahead at Data Consumers’ search for the right assets in the estate. This relates directly to the descriptions and use cases that will have to be aligned with a steering committee.

Publication – Publish the agreed upon data products and business domains, as to make them available for consumption, assigning role access’ in the process.

Operations – Assess current procedures that are in place, identifying areas in which to focus on, and assigning actions to the correct owners. This enables a continuous process of self-evaluation, paving the way to a sustainable governance model.

Stage 2 (Week 2-3)

Data Discovery & Understanding

Discover – Acts as an MVP for the search & discovery of the current products and assets already published. In essence, we are experimenting with the ease-of-use of the catalog, and reviewing with a critical eye. Ideally, this is through an unbiased experimentation of a data consumer browsing through all assets and assessing metadata applied throughout the population of the platform.

Access – Experiment with access requests, attempting to mimic an initial view of a data product by those who will want to use certain assets for further investigation.

Data Management

Access Management – Reviewing requests in the initial MVP, approving and rejecting in order to prove value. The engagement with various IT stakeholder is important to review, and this is complemented by BI4ALL’s workshops and seminars as the implementation timeline flows through.

Catalog Curation – Iterate on data product improvement by reviewing the use of the browse features, making sure that the experience of the data consumer is fed back to data stewards. This will enable improvement opportunities.

Stage 3 (Week 3-4)

Data Management

Data Quality – use one business domain to evaluate data quality dimensions like uniqueness and timeliness, experimenting with out-of-the-box and custom rules to feed the data profiling feature. Running the scans will enable alerts to be set up to evaluate data products and capture feedback on the whole experience. Steps 7 to 10 in BI4ALL’s framework are set to help with setting these up and creating actions that will certify ease-of-use.

Operations – Re-think stage 1, now in the realm of enforcing data quality rules. Assess who should be controlling these tasks and evaluate if these are in-line with the company’s data policies. An assessment with culture is applied, identifying areas for improvement.

Stage 4 (Week 4-5)

Data Estate Health

Reports – The Data Governance Office should now assess and review domain owners and act upon certain controls that need to be enforced. There should be a set of issues and solutions discussed, suggesting new approaches that will lead to addressing certain OKRs.

Health Actions – Although this is a continuous process, these actions should now be set to have owners, defined goals and detailed milestones in order to ensure that certain standards and policies are met.

Data Literacy/Training Stream (recommended but optional)

In conjunction with the outlined steps, BI4ALL can integrate a data literacy/training stream to empower stakeholders at all levels, especially if Customers do not have dedicated workforce for it. This training would equip participants with the fundamental knowledge and skills necessary to understand, interpret, and leverage data effectively.  Training modules could be tailored to the specific needs of the Customer and address topics such as data concepts, data analysis techniques, data visualization tools, and responsible data practices. The BI4ALL data literacy plan, by default, enables stakeholders to maximise their contribution throughout the steps of the aforementioned BI4ALL Data Strategy & Governance 10 Steps Approach. By adopting the full approach, the Customer will benefit of a blended data literacy, delivered as the project develops.

More in detail:

• Step 1 (Data Strategy & Governance Stream: Sync): Introductory data literacy covering basic data concepts (types, sources), data governance principles, and the importance of data-driven decision-making. This equips leadership to understand the value proposition of the data strategy and actively participate in the subsequent steps.
• Step 2,3,4 (Data Maturity Assessment or Audit): More in-depth training modules focused on the topics of the data maturity assessment or the audit. Topics might include data management practices, data quality principles, in depth data governance etc.
• Step 5 & 6 (Data Strategy & Governance Design): Interactive workshops with hands-on demos and exercises in data analysis techniques, data visualization, AI governance tools etc. This allows stakeholders to participate actively in designing the data strategy and governance framework, fostering ownership and buy-in.
• Step 7, 8, & 9 (Data Governance Foundations): Role-specific training covering data governance policies, key processes, data lifecycle management, etc. This equips individuals with the necessary skills to fulfil their designated roles within the data governance framework.
• Step 10 (Microsoft Purview Kick-Start): Training on using Microsoft Purview functionalities, data integration techniques, and data security best practices. This ensures personnel can effectively utilize Purview to support data governance initiatives.