Implement incremental enrichment

Incremental enrichment is a feature of Azure Cognitive Search which allows you to configure a skillset to run on a schedule basis and only processes modified or new data. This approach is advantageous because it helps to avoid rerunning the whole skillset, which would consume a significant amount of time and resources. Incremental enrichment is particularly helpful when dealing with a large corpus of data that continually receives updates.

Differential Indexing

The baseline for incremental enrichment is differential indexing. This establishes incremental runs limited to updates and additions. It also enables updates or additions to be attached with cognitive skills midway through a pipeline.

Implementation of Incremental Enrichment in Azure AI

In order to implement incremental enrichment in Azure AI, you will need to perform the following steps:

• Set up differential indexing: This is the first step for incremental enrichment. Add a “high watermark” column in your database, which is a timestamp column that can track the changes in each row.
• Add a change tracking policy: Next step is to set a change tracking policy in the data source. You have three options:
• HighWaterMark: This policy uses SQL-integrated change tracking features.
• SQLIntegrated: This tracks changes in the database.
• CDC: Uses SQL Server Change Data Capture (CDC). Applicable to SQL Data sources only.
• Define a schedule: Once indexing and tracking are set, a schedule for the indexer to run needs to be defined. The schedule can be set as per the business requirement like once a week, daily, etc.
• Create an incremental skillset: Next, create a skillset with the desired cognitive skills. This skillset defines information extraction, organization, and transformation stages.
• Create an indexer: This is the final step. An indexer needs to be created that references the data source, skillset and target index.

HighWaterMark Policy

Below is an example of how to set a data source with a HighWaterMark policy.

{
“name” : “mydatabasesource”,
“type” : “azureblob”,
“credentials” : { “connectionString” : “” },
“container” : { “name” : “my-container-name” },
“dataChangeDetectionPolicy” : {
“@odata.type” : “#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy”,
“highWaterMarkColumnName” : “LastModified”
}
}

In the above JSON example, we have an AzureBlob data source “mydatabasesource”. Under “dataChangeDetectionPolicy”, we specify HighWaterMarkChangeDetectionPolicy and the high watermark column is “LastModified”.

SQLIntegrated Change Tracking Policy

Here is an example of how a SQLIntegrated Change Tracking Policy can be implemented.

{
“name” : “myazuresqlsource”,
“type” : “azuresql”,
“credentials” : { “connectionString” : “” },
“container” : { “name” : “my-database-name” },
“dataChangeDetectionPolicy” : {
“@odata.type” : “#Microsoft.Azure.Search.SqlIntegratedChangeTrackingPolicy”
}
}

In the above JSON example, an AzureSql data source “myazuresqlsource” is created. Under “dataChangeDetectionPolicy”, SqlIntegratedChangeTrackingPolicy is specified with no further configuration needed.

It is recommended to choose the policy based on your requirements to track data change: use HighWaterMark-based if the database supports it, or else fallback to SQL Integrated.

Azure AI’s incremental enrichment feature is a powerful tool that can reduce processing time and cost by only reproducing data that has actually changed. By smartly configuring your data sources and skillsets, you can gain more efficiency in cultivating and using your data.

Practice Test

True or False: Incremental enrichment is a process by which new data is processed during a pipeline run to refine the model.

• True
• False

Answer: True

Explanation: Incremental enrichment is a process by which new data is processed during a pipeline run. This new data enriches the model over time, therefore updating and improving the accuracy of the model when making predictions.

In incremental enrichment, data only improves in quality when the model is retrained.

• True
• False

Answer: False

Explanation: Data quality can improve even without retraining the model. As new data is received and processed, it can lead to better insights and predictions.

True or False: Incremental enrichment of knowledge requires periodically reprocessing and linking all the previous data.

• True
• False

Answer: True

Explanation: To keep enriching the knowledge, all previous data needs to be considered for reprocessing and linking in incremental enrichment. This way, the model gets a holistic view of the information it has learned and continues to learn.

Which of the following phase of machine learning life cycle benefits the most from incremental enrichment?

• a) Training
• b) Evaluation
• c) Deployment
• d) None of the above

Answer: a) Training

Explanation: During the training phase of the machine learning life cycle, models learn from data. Therefore, incremental enrichment – by providing additional data over time – often benefits this phase the most.

Which of the following is NOT a step in implementing incremental enrichment?

• a) Loading new data
• b) Reprocessing old data
• c) Disregarding old data
• d) Analyzing new insights

Answer: c) Disregarding old data

Explanation: Incremental enrichment involves adding new data to the previous data, not disregarding it.

True or False: In incremental enrichment, the system is trained from scratch each time new data comes in.

• True
• False

Answer: False

Explanation: In incremental enrichment, the system is not trained from scratch each time but is rather updated with new data. This new processed data enriches the existing model rather than replacing it.

True or False: For implementing incremental enrichment in Azure, knowledge stored in search indexes are used.

• True
• False

Answer: True

Explanation: Azure Cognitive Search uses search indexes to implement incremental enrichment. These indexes store the knowledge from which the model learns and grows.

Azure Cognitive Search’s indexer functionality is crucial for implementing incremental enrichment.

• a) True
• b) False

Answer: a) True

Explanation: Indexers in Azure Cognitive Search pick up the data, run enrichment tasks on it, and load it into an index, which is fundamental for implementing incremental enrichment.

Incremental enrichment requires which of the following two main elements?

• a) A machine learning model and new data
• b) Old data and a data reprocessing system
• c) A machine learning model and old data
• d) New data and a data reprocessing system

Answer: a) A machine learning model and new data

Explanation: Incremental enrichment improves a machine learning model with additional data over time, therefore, requiring both these elements.

True or False: Incremental enrichment is a destructive process that affects the original data.

• True
• False

Answer: False

Explanation: Incremental enrichment is a non-destructive process that keeps the original data intact while adding value to the machine learning model with new processed data.

Interview Questions

What is incremental enrichment in relation to data processing in Azure AI?

Incremental enrichment in Azure AI involves enriching new data incrementally as it becomes available rather then re-processing all data from the beginning.

What role does Azure Cognitive Search play in implementing incremental enrichment?

Azure Cognitive Search provides built-in AI capabilities, such as AI enrichment, which can be used during the indexing process to extract information. Incremental enrichment can be implemented to only process new or updated data.

How can incremental enrichment improve performance in Azure AI?

Incremental enrichment can significantly improve performance by reducing the amount of data that needs to be processed at one time. It allows Azure AI to process only the newly added or updated data instead of reprocessing all data.

What is a skillset in Azure Cognitive Search and how does it relate to incremental enrichment?

A skillset in Azure Cognitive Search is a group of AI processing skills used to enrich and transform data during the indexing process. A skillset is used in incremental enrichment of new or updated data to extract valuable information.

What is Change Tracking in Azure SQL Data Source?

Change Tracking is a lightweight solution provided by Azure SQL Data Source that keeps track of changes made in the rows of a table such as insert, update, and delete operations. It is useful for the incremental enrichment process.

What is the role of indexer in Azure Cognitive Search in implementing incremental enrichment?

The indexer in Azure Cognitive Search is responsible for reading the data from the data source and updating the search index. In the context of incremental enrichment, the indexer only processes on the new or changed data, minimizing workload and improving efficiency.

What are the prerequisites for implementing incremental enrichment in Azure AI?

The prerequisites for implementing incremental enrichment include having an Azure Cognitive Search service, a data source with change tracking capability, and a skillset defined for data enrichment.

Why is incremental enrichment important in Azure AI Solution?

Incremental enrichment is important because it efficiently processes new or changed data without having to reprocess all existing data. This significantly enhances efficiency and saves resources.

How to modify the indexer to support incremental enrichment?

To modify an indexer to support incremental enrichment, you need to specify a “data change detection policy” in the indexer definition, which allows the indexer to identify new or changed data in the data source.

Is it possible to implement incremental enrichment without Azure Cognitive Search in Azure AI solutions?

No, Azure Cognitive Search plays an integral role in implementing incremental enrichment in Azure AI solutions by providing AI processing skills and reading data from the data source and updating the search index. Without Azure Cognitive Search, incremental enrichment feature cannot be utilized.

What is a knowledge store in Azure Cognitive Search?

A knowledge store in Azure Cognitive Search allows projections, or diverse views of enriched documents, to be stored and utilized for machine learning, analytical, or other purposes. A knowledge store is used in incremental enrichment to store and index enriched new or updated data.

What is soft deletion in the context of Azure Cognitive Search?

Soft deletion is a practice in Azure Cognitive Search where data is flagged as deleted but not immediately removed from the system. This strategy is used for managing data in incremental loads, allowing the indexer to remove deleted content from the index while keeping track of changes.

Can incremental enrichment be used with Azure Blob Storage?

Yes, incremental enrichment can be used with Azure Blob Storage. This can be implemented by using a built-in change detection mechanism in Azure Blob Storage called blob versioning.

What are the limitations of implementing incremental enrichment?

The limitations include that the data source used must support change tracking, a hard delete may require a full re-indexing, and complex transformations can cause issues as changes may depend on the overall state of other data.

What type of data sources support incremental enrichment in Azure Cognitive Search?

Azure Cognitive Search supports incremental enrichment from several data sources such as Azure SQL Database, Azure Cosmos DB, Azure Blob Storage, and Azure Table Storage. This is based on the condition that they have change tracking enabled.