Create custom logs in Azure Log Analytics to store custom data

Azure Log Analytics is a powerful tool and part of Azure Monitor that allows you to analyze and check the availability and performance of your applications and services. It collects data from your applications, operating system, and infrastructure to provide near to real-time analysis and comprehensive insights across your environment, logs, and network traffic. However, sometimes the standard logs don’t provide sufficient details for specific scenarios. In such cases, custom logs come into play.

Creating custom logs allows you to store your custom data and supplement the standard logs in Azure Log Analytics. These custom logs provide you with the flexibility to monitor the parameters of your choice.

Creating Custom Logs in Azure Log Analytics

You can create custom logs in Azure Log Analytics using Azure CLI, Azure PowerShell, or Azure REST API.

Note: Before creating custom logs, make sure you have installed the appropriate tool – Azure CLI, or Azure PowerShell.

Using Azure CLI

Here is the step by step guide to create a custom log in Azure Log Analytics using Azure CLI.

1. Log in to your Azure account
   Use the following command to log in to your Azure account.

   az login

2. Create a custom log
   Use the following command to create a custom log.

   az monitor log-analytics workspace create --resource-group MyResourceGroup --workspace-name MyWorkspace

   Replace ‘MyResourceGroup’ and ‘MyWorkspace’ with your resource group and workspace names respectively.
   After running the above command, a custom log will be created under the specified workspace.

Using Azure PowerShell

1. Log in to your Azure account
   Use the following command to log in to your Azure account.

   Connect-AzAccount

2. Create a custom log
   Use the following command to create a custom log.

   New-AzOperationalInsightsWorkspace -ResourceGroupName "MyResourceGroup" -Name "MyWorkspace" -Location "eastus"

   Replace ‘MyResourceGroup’ and ‘MyWorkspace’ with your resource group and workspace names respectively.
   As a result, a custom log will be created under the specified workspace.

Comparing Azure CLI and Azure PowerShell

Here is a comparison of using Azure CLI and Azure PowerShell for creating a custom log in Azure Log Analytics.

Note: These two methods are not the same and cannot be used interchangeably. Each tool should be used in its respective environment.

Conclusion

Creating custom logs enhances the functionality of Azure Log Analytics by providing a method to store and analyze tailored data. This ability to customize the way logs are created and what is stored provides security operations analysts with the flexibility they need to monitor their systems effectively. As you prepare for the SC-200 Microsoft Security Operations Analyst exam, gaining an understanding of how to create and use custom logs will be an essential skillset to master.

Practice Test

True or False: Azure Log Analytics can be used to store custom logs that can be created by users.

• True
• False

Answer: True.

Explanation: Azure Log Analytics is a service in Azure Monitor that allows you to collect and analyze data generated by resources in your cloud and on-premises environments.

In Azure, which service can be used to collect, analyze, and act on telemetry from cloud and on-premises environments?

• A. Azure Logic Apps
• B. Azure Monitor
• C. Azure Data Factory
• D. Azure Pipelines

Answer: B. Azure Monitor

Explanation: Azure Monitor maximizes the availability and performance of applications by delivering a comprehensive solution for collecting, analyzing, and acting on telemetry from resources.

True or False: In Azure Log Analytics, you can only log data from applications running in Azure.

• True
• False

Answer: False

Explanation: Azure Log Analytics is able to collect log data from virtually any cloud and on-premises environments, not just applications running in Azure.

True or False: The data in custom logs in Azure Log Analytics are stored for 90 days.

• True
• False

Answer: False

Explanation: By default, the data in custom logs in Azure Log Analytics are stored for 31 days, but this period can be expanded up to 2 years.

Which type of data can you store in Azure Log Analytics?

• A. Event logs
• B. Performance counters
• C. Azure activity logs
• D. All of the above

Answer: D. All of the above

Explanation: Azure Log Analytics can store and analyze any type of data, including event logs, performance counters, and Azure activity logs.

Can you set up alerts based on custom logs in Azure Log Analytics?

• Yes
• No

Answer: Yes

Explanation: Alerts can be set up based on the data in your custom logs, and these alerts can trigger actions like email notifications or automated scripts.

To what data sources can Azure Log Analytics connect?

• A. Azure resources
• B. AWS resources
• C. On-premises resources
• D. All of the above

Answer: D. All of the above

Explanation: Azure Log Analytics can connect to Azure resources, AWS resources, and on-premises resources, allowing you to collect and analyze data from all of your environments.

True or False: Azure Log Analytics supports querying of data using SQL-like query language.

• True
• False

Answer: True.

Explanation: Azure Log Analytics supports Kusto Query Language (KQL), which is a read-only request to process data and return results.

Can you use Azure Monitor to investigate incidents and issues in Azure Log Analytics custom logs?

• Yes
• No

Answer: Yes

Explanation: Azure Monitor can be used together with Azure Log Analytics to investigate issues, correlate activities, and create comprehensive reports.

Does Azure Log Analytics support PowerShell or REST API commands to pull data from custom logs?

• Yes
• No

Answer: Yes

Explanation: Azure Log Analytics supports both PowerShell and REST API commands, providing developers with greater flexibility in retrieving data from custom logs.

True or False: You can use Log Analytics workspace to manage and analyze the data collected from different Azure services.

• True
• False

Answer: True.

Explanation: Log Analytics workspace provides a consolidated way to store and analyze data from different Azure services.

What is the purpose of creating custom logs in Azure Log Analytics?

• A. To organize and simplify data analysis.
• B. To store custom data that does not fit into other log types.
• C. To increase the cost of Azure Log Analytics.
• D. Both A and B

Answer: D. Both A and B

Explanation: Custom logs in Azure Log Analytics provide a way to store custom data that does not fit into other log types and help organize and simplify data analysis but not to increase costs.

True or False: There is a limit to the amount of data you can store in custom logs.

• True
• False

Answer: False

Explanation: There is no specific limit to the amount of data you can store in custom logs. It is determined by your Azure subscription and any costs associated with data ingestion and retention.

True or False: You need to manually setup data sources every time for Azure Log Analytics to collect and analyze data.

• True
• False

Answer: False

Explanation: Azure Log Analytics automatically pulls data from connected data sources, whether they’re Azure resources, other cloud resources, or on-premises resources.

Can there be multiple workspaces inside one Azure Log Analytics instance?

• Yes
• No

Answer: Yes

Explanation: You can have multiple workspaces within a single instance of Azure Log Analytics, each of which can be independently configured.

Interview Questions

How can you send custom logs to Azure Log Analytics?

You can use the HTTP Data Collector API to send custom logs to Azure Log Analytics. This allows you to create a custom logging solution.

Why might you want to create custom logs in Azure Log Analytics to store custom data?

Custom logs can store data that is not gathered by the built-in Azure Monitor logs. This offers the ability to store custom logs with unique data specific to your application’s needs, making it available for analysis alongside other monitored data.

What PowerShell command is used to create a custom log in Azure Log Analytics?

The command is ‘New-AzOperationalInsightsCustomLogDataSource’, it is used to create a custom log in Azure Log Analytics.

Can you query custom logs in Azure Log Analytics?

Yes, after data has been ingested into Azure Monitor Logs, it’s available for analysis along with the rest of the monitoring data.

Can Azure Log Analytics custom logs store data of any schema?

Yes, Azure Log Analytics custom logs can store data of any schema, but the data must be transformed into a flat record structure that can be represented as a series of columns.

Is there a limit to the amount of data that can be sent in a single post to HTTP Data Collector API?

Yes, according to Microsoft’s documentation, limitations are set that a single post can be a maximum of 30 MB and can contain up to 32,768 records.

How can you view custom logs in the Azure portal?

To view custom logs, go to the Logs section in the Azure portal and run a query against the Log Analytics workspace. Use your custom log name in the search field.

What is the advantage of using HTTP Data Collector API in Azure Log Analytics?

HTTP Data Collector API allows you to send data from any client that can call the REST APIs. It offers high flexibility to work with different sources and custom data.

What is the format for creating new records in Azure Log Analytics?

New records are created in JSON format, which includes fields and their values.

What is the purpose of the LogType parameter in the HTTP Data Collector API?

The LogType parameter defines the record type of the data that is being submitted, essentially naming the custom log.

What does ‘TimeGeneratedField’ specify in the ‘New-AzOperationalInsightsCustomLogDataSource’ command?

‘TimeGeneratedField’ specifies the name of the field in the data input that will be used as the timestamp of the data record.

How does Azure classify the data types of the fields in the custom logs?

Azure classifies the type of data based on the types of data in the first record it processes.

Can you modify the data type of a field in custom logs after it has been set by Azure?

No, once the data type of a field is set, it cannot be changed.

Is there a limit to the number of custom fields I can add to my Azure Log Analytics custom logs?

Custom logs in Azure Log Analytics can include up to 500 fields for a single record type.

What is the general latency for data availability in Azure Log Analytics after sending data via HTTP Data Collector API?

Typically, it takes about 5-10 minutes from the time the data is sent until it is available in Log Analytics for queries.