Register and maintain datastores

Maintaining datastores in Azure is a vital task for Data Scientists and a key topic in the DP-100 Designing and Implementing a Data Science Solution on Azure Exam. Throughout this discussion, we will describe the best practices to register and maintain datastores and put in some practical examples.

Azure Datastore Overview

Azure datastores represent a storage location, such as Azure Blob Storage, File Share, or Data Lake Store. They could be securely connected to various Azure services, like Azure Machine Learning, Databricks, or Azure Synapse Analytics.

There are two types of datastores that can be created in Azure:

• Azure Blob storage: Blob Storage stores massive amounts of unstructured object data, like text or binary data. This could be used for serving images or documents directly to a browser, storing files for distributed access, streaming video and audio, etc.
• Azure Data Lake Storage: This is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage. It enhances Azure Blob Storage capabilities and is optimized for analytic workloads.

Registering a Datastore

Registering a datastore means connecting your Azure Machine Learning workspace to your storage account. This can be done using the Azure portal, SDK, or CLI.

Here is an example of registering a datastore using Python SDK:

from azureml.core import Datastore
blob_datastore_name='MyBlobDatastore'
account_name=os.getenv("BLOB_ACCOUNTNAME_62", "")
container_name=os.getenv("BLOB_CONTAINER_62", "")
account_key=os.getenv("BLOB_ACCOUNT_KEY_62", "")
blob_datastore = Datastore.register_azure_blob_container(workspace=ws,
datastore_name=blob_datastore_name,
container_name=container_name,
account_name=account_name,
account_key=account_key)


After successfully registering your datastore, you’ll be able to access and manage your data from your workspace.

Maintaining Datastores

Maintenance of datastores is a critical task to ensure that your data is safe, secure, and accessible when needed. Here are some best practices for maintaining your datastores:

• Monitoring: Regularly monitor your data stores. Azure provides built-in services like Azure monitor and Azure Storage monitoring for overseeing your storage accounts.
• Security: Ensure to adhere to security best practices like using managed identities, securing your data at rest and in transit, etc.
• Data Lifecycle Management: Implement a data lifecycle management policy. Azure Storage Lifecycle Management offers a rich, rule-based policy for general-purpose v2 and Blob storage accounts.
• Regular Backup: Regularly backup your data. Azure Backup service provides simple, secure, and cost-effective solutions to backup your data and recover it from the Microsoft Azure cloud.

In conclusion, registering and maintaining datastores is a fundamental aspect of designing and implementing a data science solution on Azure. This has been just a snapshot of what the DP-100 exam might cover on this topic. For deeper understanding, you should explore the Azure documentation and seek practical, hands-on experience.

Remember, the most effective way to prepare for the DP-100 exam is to have real-world experience working with Azure services. Use the Azure free account to explore and play around with the services to gain mastery and confidence. Good luck with your exam preparation!

Practice Test

True or False: Once a datastore is registered in Azure, it cannot be deleted.

• 1) True
• 2) False

Answer: 2) False

Explanation: A datastore registered in Azure can be deleted, but all datasets associated with the datastore must be deleted first.

In Azure Machine Learning Studio, datastores need to be set as the default datastore for each workspace.

• 1) True
• 2) False

Answer: 1) True

Explanation: Setting a datastore as the default is a good practice as it simplifies data access and management tasks.

Multiple Select: Which of the following datastore types are supported in Azure Machine Learning?

• a) Azure Blob Storage
• b) Azure Data Lake Storage
• c) Azure SQL Database
• d) Azure Cosmos DB

Answer: a, b, c

Explanation: Azure Machine Learning service supports Azure Blob Storage, Azure Data Lake Storage, and Azure SQL Database. Azure Cosmos DB is currently not supported as a datastore type.

A datastore can be registered only once in an Azure Machine Learning workspace.

• 1) True
• 2) False

Answer: 2) False

Explanation: A datastore can be registered multiple times in a workspace.

Single Select: What CLI command is used to update an existing datastore?

• a) azure ml datastore update
• b) azure ml update datastore
• c) azure ml upgrade datastore
• d) azure ml datastore upgrade

Answer: a) azure ml datastore update

Explanation: The command “azure ml datastore update” is used to update the details of a registered datastore.

Datastores in Azure Machine Learning are used to store credentials.

• 1) True
• 2) False

Answer: 1) True

Explanation: Datastores are used to securely store connection information to your data source locations and avoid storing credentials in code.

Single Select: Which of the following is not an option to load data into a datastore?

• a) From local files
• b) From cloud data
• c) From streaming data
• d) From web services

Answer: d) From web services

Explanation: In Azure, you can load data into a datastore from local files, cloud data, and streaming data. Loading data from web services is not an option.

True or False: Datastores are version controlled in Azure Machine Learning.

• 1) True
• 2) False

Answer: 2) False

Explanation: Datastores in Azure Machine Learning are not version controlled. The Datasets, derived from datastores, are version controlled.

True or False: You can set permissions for specific users on a datastore.

• 1) True
• 2) False

Answer: 1) True

Explanation: Datastores in Azure Machine Learning allow you to specify permissions for different users.

Single Select: What protocol is used to transfer data between Azure resources and the datastore?

• a) SCP
• b) HTTP
• c) FTP
• d) SFTP

Answer: b) HTTP

Explanation: HTTP (HyperText Transfer Protocol) is used to transfer data between Azure resources and the datastore.

Interview Questions

What is a datastore in Azure Machine Learning?

A datastore in Azure is a place where you can store your data for machine learning workloads. It allows you to manage connection information to Azure storage services without exposing your connection details.

How can you register a datastore in Azure ML?

You can register a datastore in Azure ML by using Python SDK. You can use the

register_azure_blob_container

,

register_azure_file_share

,

register_azure_data_lake

,

register_azure_sql_database

, or

register_azure_data_lake_gen2

method of the

Datastore

class.

What type of data stores can you register with Azure Machine Learning?

You can register several types of data stores, including Azure Blob Storage, Azure Files, Azure Data Lake Storage, Azure SQL Database, Azure PostgreSQL, and Databricks File System.

How do you update the registered datastore's metadata in Azure ML?

To update the registered datastore's metadata, use the update method on the Datastore object. You can update the description and tags but not the datastore type or datastore name.

How would you list all the registered datastores in your Azure ML workspace?

You can use the

Datastore.get_all()

method to list all registered datastores in your Azure ML workspace.

What is the purpose of the default datastore in Azure Machine Learning?

The default datastore is the storage account that is created when you create a workspace in Azure Machine Learning. It is automatically registered as a datastore and is used as the default place to store data unless specified otherwise.

Can you delete a datastore in Azure ML?

No, you can't delete a datastore in Azure ML. However, you can unregister it, which will remove it from the list of datastores available in your workspace.

How can you access data in a datastore?

You can access data in a datastore by creating a data reference or using the path function on the datastore object to create a path to a file or folder.

How do you set a datastore to be the default datastore in Azure ML?

To make a datastore the default, use the

Datastore.set_default()

method.

Can a datastore be registered with multiple workspaces within Azure ML?

Yes, a datastore like an Azure Blob storage or Azure file share can be registered with multiple workspaces.

What is Azure Data Lake Storage and can we register it as a datastore in Azure ML?

Yes, Azure Data Lake Storage is a hyper-scale repository for big data analytics workload which can be registered as a datastore in Azure ML.

Are datastores shared across Workspaces in Azure ML?

No, datastores are not shared across workspaces. Each workspace has its own set of datastores.

What are the two types of data references in Azure ML?

The two types of data references are file streams and path-based.

Is it necessary to register datastores before using them?

Yes, before you can read data from or write data to a datastore in your Azure ML scripts, the datastore must be registered to your workspace.

What three components are linked together when registering a Datastore in Azure ML?

The three components linked together when registering a Datastore are the Azure ML workspace, a named reference to the stored data, and the actual stored data or source of the data.