Manage your Data Estate with Data Map In Purview

Collections and sources[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]Let us consider a scenario with more than 1000 resources. Without a proper grouping practise it would be impossible to go through all the resources. Furthermore changing access levels of all 1000 resources which belong to a same category will be an unproductive task that could take hours. To tackle this in purview, collections and sources come quite in handy. Collections and resources can help users to manage their data resources effectively. We can think about it like a hierarchy structure, each collection can include several related sources, we can set different roles for collections so that the key users and groups can access and take response for the resources in collection. All the sources in collections are secured by the credentials. Data map stores the metadata from all the resources with an elastic auto-scale. It can start from the lowest one unit capacity and increase with the size of load. [/et_pb_text][et_pb_image src=”https://hubsters.de/wp-content/uploads/2021/11/purview_datamap1.png” title_text=”purview_datamap1″ _builder_version=”4.8.1″ _module_preset=”default”][/et_pb_image][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Register and scan[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]In purview, we can register data resources in the collection and scan them to get the metadata info. Purview provides many kinds of data sources to register, not only the services from Azure, but also from other platforms. It includes the back-end structured and non-structured database like Hive and SQL servers, as well the front-end data visualization tool Power BI. In the end, users can get the lineage from raw data to visualization. When something goes wrong in the visualization, key users can easily identify the raw sources of the values in the reports and in addition would also get the information about the Data Owners and Data Stewards. The entire process of identifying the cause and the person responsible is significantly shortened. [/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]Purview scanning can discover the technical information about the data sources such as the technical name, data type, and size. It can extract the schema from data resources for structured data. The business information such as classification, glossary, descriptions can be found automatically while scanning or can be added manually to each asset. [/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Credential[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]Each scan needs credentials of the matching resources. This authorization method provides security to customers’ data, since purview does not save any password or access key directly. Instead, users can establish the key vault connection in purview and add secrets as credentials. The four possible ways to authenticate the Purview account are Managed Identity, Service Principal, SQL Authentication, and Account Key or Basic Authentication. When users add the secrets as credentials, they should also choose the matching Authentication method. Credential management provides the interface to manage the key vaults and secrets. [/et_pb_text][et_pb_image src=”https://hubsters.de/wp-content/uploads/2021/11/purview_datamap2.png” title_text=”purview_datamap2″ _builder_version=”4.8.1″ _module_preset=”default”][/et_pb_image][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Scan rules[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px” min_height=”229px” custom_padding=”||0px|||”]After the resource connection is tested, the assets in this resource can be scanned. The users can set the scan for full scope or just for special folders or tables. The options are based on the type of source users want to scan. The scan can be set or regularly scheduled e.g. weekly or monthly. For regular scans, the first-time scan setting will apply for every scan after that as well. Each scan follows a scan rule. This rule defines what information the scan should look for, what classification rule it should use for columns, and so on. There are already some system default scan rules for many kinds of resources such as Azure Data Lake and SQL database. Users can set custom scan rules with special classification rules and pattern rules based on needs. So that the classification and pattern mapping can be generated automatically after the scan. [/et_pb_text][et_pb_image src=”https://hubsters.de/wp-content/uploads/2021/11/purview_datamap3.png” title_text=”purview_datamap3″ _builder_version=”4.8.1″ _module_preset=”default”][/et_pb_image][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px” min_height=”52px”]Within Scan rules there are types like Pattern rules and Classification rules which help categorize your data assets or folders. [/et_pb_text][et_pb_image src=”https://hubsters.de/wp-content/uploads/2021/11/classification_pattern_rules_purview.png” title_text=”classification_pattern_rules_purview” _builder_version=”4.8.1″ _module_preset=”default”][/et_pb_image][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Pattern rules[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px” hover_enabled=”0″ sticky_enabled=”0″]Pattern rules are used for azure file, data lake, blob storage and amazon S3. After choosing the storage account for the pattern rule, the rule will apply for this certain account or container from next time scan. Users can use statistic and dynamic replacer to match the qualified name of assets. So that the rule can map for a group of data and use the display name users add in the rule. When the files shouldn’t be grouped, users can enable ‘do not group as a resource set. [/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Classification[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]Purview classification helps key user to find the special data or certain data type in the data estate. There are many system default classification rules which can be used for scan rules set. Customer can also create the custom classification rules to detect the data type for their own datasets such as primary key or production id. The dictionary lists all the possible objects that the column can include. There is also a minimum match threshold which shows at least how many percentage of column content should match the expression or dictionary. [/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]

Conclusion[/et_pb_text][et_pb_text _builder_version=”4.8.1″ header_2_font_size=”22px”]Data map provides elastic pay as you go model so that the customer don’t need to worry about the scaling and the size limit for the features. It is the fundamental of Purview for data discovery and data governance. With the register and scan feature for all the data sources, you can start you data governance journey with the least efforts. [/et_pb_text][/et_pb_column][/et_pb_row][et_pb_row _builder_version=”4.4.8″][et_pb_column type=”4_4″ _builder_version=”4.4.8″][et_pb_divider color=”#74c2d2″ _builder_version=”4.4.8″][/et_pb_divider][/et_pb_column][/et_pb_row][/et_pb_section][et_pb_section fb_built=”1″ _builder_version=”4.4.8″][et_pb_row column_structure=”1_3,2_3″ _builder_version=”4.4.8″ max_width=”862px” max_width__hover_enabled=”on|desktop”][et_pb_column type=”1_3″ _builder_version=”4.4.8″ custom_padding=”0px||0px|27px|false|false”][et_pb_image src=”https://hubsters.de/wp-content/uploads/2020/12/Qianyu_Chen.jpg” alt=”Linus Trips HUBSTER.S” title_text=”Qianyu_Chen” _builder_version=”4.8.1″ max_width_tablet=”30%” custom_margin=”|||0px|false|false” custom_margin_tablet=”” custom_margin_phone=”||-26px|0px|false|false” custom_margin_last_edited=”on|phone” custom_padding=”|||0px|false|false” module_alignment_tablet=”left”][/et_pb_image][/et_pb_column][et_pb_column type=”2_3″ _builder_version=”4.4.8″][et_pb_blurb title=”Qianyu Chen” _builder_version=”4.8.1″ custom_margin=”36px||||false|false” hover_enabled=”0″ sticky_enabled=”0″]

Qianyu Chen is Solution Architect for Data Analytics und Machine Learning.[/et_pb_blurb][et_pb_blurb use_icon=”on” font_icon=”%%153%%” icon_color=”#303344″ icon_placement=”left” use_icon_font_size=”on” icon_font_size=”20px” _builder_version=”4.8.1″ header_text_align=”left” custom_margin=”||30px||false|false” animation=”off” link_option_url=”mailto:christoph.monzel@hubsters.de” body_text_align_tablet=”center” body_text_align_phone=”center” body_text_align_last_edited=”on|phone” text_orientation_tablet=”left” text_orientation_phone=”left” text_orientation_last_edited=”on|tablet” module_alignment_tablet=”center” module_alignment_phone=”” module_alignment_last_edited=”on|tablet”]

qianyu.chen@hubsters.de[/et_pb_blurb][/et_pb_column][/et_pb_row][/et_pb_section]