As part of the security operations work, analysts spend a considerable amount of time investigating security events and incidents. Often, this work requires analysts to correlate and analyze data from various sources. MSTICPy aims to make this investigation process more efficient and intuitive by offering Python tools for data ingestion, analysis, visualization, and enrichment.
Installing MSTICPy
Before we use MSTICPy, we must first install the package in our environment. MSTICPy is compatible with Python 3.6 and later versions. A simple pip install command should suffice.
pip install msticpy
Remember to update regularly the MSTICPy library, as it’s continuously being improved and expanded.
pip install --upgrade msticpy
Configuration
The first step to use MSTICPy is to ensure that it is properly configured. Configuration details are usually maintenance in a msticpyconfig.yaml file, which you can configure to suit your needs. The sections such as AzureSentinel, TIProviders and DataProviders are of special interest.
Here’s an example:
AzureSentinel:
Workspaces:
Default:
TenantId: 12345678-1234-1234-1234-123456789012
WorkspaceId: 12345678-1234-1234-1234-123456789012
TIProviders:
VirusTotal:
Args:
AuthKey: "Your VirusTotal API key"
DataProviders:
LocalData:
Driver: "sqlite"
Args:
Connection_String: "sqlite:///path/to/your/datafile.db"
In your Jupyter notebook enter the following commands to load the configuration.
from msticpy.nbtools import nbinit
nbinit.init_notebook(namespace=globals())
Usage
With MSTICPy installed and configured, it is now time to utilize its modules for investigations and analysis.
Data Import and Processing
MSTICPy allows for connection to data sources like Azure Sentinel to collect data. Let’s see an example of using MSTICPy to query data from Azure Sentinel.
from msticpy.data import QueryProvider
qry_prov = QueryProvider("LogAnalytics")
qry_prov.connect(WorkspaceConfig(workspace="MyWorkspace"))
# construct your query
my_query = """
SecurityAlert
| where TimeGenerated > ago(1d)
"""
# run the query
df = qry_prov.exec_query(my_query)
In the example above, you are establishing a connection to Azure Sentinel workspace and running a Kusto Query Language (KQL) query to fetch security alerts data.
Data Enrichment
MSTICPy can also be used to enrich the data by integrating with multiple Threat Intelligence providers.
from msticpy.sectools import TILookup
ti = TILookup()
details = ti.lookup_ioc(observable="91.189.88.142", providers=["VirusTotal"])
In the above example, we are enriching the data by looking up details for a specific IP address (91.189.88.142), using VirusTotal as a Threat Intelligence provider.
All in all, MSTICPy allows security analysts to shift their focus from data collection and cleaning to the actual investigation and threat hunting. It’s a powerful tool that can be very beneficial, particularly in a professional capacity such as the Security Operations Analyst role represented by the SC-200 exam. For a full understanding of its capabilities and more detailed usage examples, you can refer to the MSTICPy documentation and GitHub repositories.
Practice Test
True/False: MSTICPy is a Python package developed by Microsoft, with a focus on cyber security investigations and hunting in Jupyter notebooks.
- True
- False
Answer: True
Explanation: MSTICPy is indeed a Python package that was developed by Microsoft Threat Intelligence Center (MSTIC) specifically for cyber security investigations and threat hunting.
Multiple Select: Which of the following are core features of MSTICPy?
- A. Data querying
- B. Data engineering
- C. Threat intelligence
- D. Visualization
Answer: All of the above.
Explanation: MSTICPy includes several features which include data querying, data engineering, threat intelligence and visualization.
True/False: The matplotlib library in Python is not necessary while using MSTICPy.
- True
- False
Answer: False
Explanation: Matplotlib is a requirement for MSTICPy operations such as creating graphs.
Single Select: What is the primary mode through which MSTICPy is run?
- A. Python interpreter
- B. Jupyter notebooks
- C. Integrated Development Environments (IDE)
- D. Command line scripts
Answer: B. Jupyter notebooks
Explanation: MSTICPy is designed to be run interactively from Jupyter notebooks.
True/False: MSTICPy supports querying data only from Azure Sentinel.
- True
- False
Answer: False
Explanation: Besides Azure Sentinel, MSTICPy can also query data from other data sources like Log Analytics and Virus Total.
Multiple Select: What are the crucial components you need for successful MSTICPy execution?
- A. Python 6 or later
- B. Authentication to a workspace
- C. A configured msticpyconfig.yaml
- D. Microsoft 365 subscription
Answer: A, B, and C
Explanation: For successful MSTICPy execution, you need Python 6 or later, access to a workspace, and a properly configured msticpyconfig.yaml. Although having a Microsoft 365 subscription might be beneficial, it’s not a necessity for MSTICPy.
True/False: MSTICPy is compatible with Python versions older than
- True
- False
Answer: False
Explanation: MSTICPy requires Python 6 or later versions to function properly.
Multiple Select: What functionalities does MSTICPy data querying offer?
- A. Search for data
- B. Pivot and drill down into data
- C. Join data
- D. Query data from online sources
Answer: All of the above
Explanation: MSTICPy provides operations for data searching, drilling, joining as well as querying data from online sources.
Single Select: The MSTICPy configuration file is named as?
- A. msticpyconfigure.py
- B. msticpyconfig.xml
- C. msticpyconfig.yaml
- D. msticpy.ini
Answer: C. msticpyconfig.yaml
Explanation: The MSTICPy configuration file is named as msticpyconfig.yaml. This file contains settings for various data providers and other services used by MSTICPy.
True/False: MSTICPy does not provide any data enrichment capabilities.
- True
- False
Answer: False
Explanation: MSTICPy provides extensive data enrichment capabilities such as IP geolocation, Threat Intel lookups, and Whois lookups.
Multiple Select: MSTICPy can be installed via which methods?
- A. Pip
- B. Anaconda
- C. Directly from GitHub
- D. PyPI
Answer: All of the above
Explanation: MSTICPy can be installed using pip, anaconda, directly from GitHub and the Python Package Index (PyPI).
Single Select: Out of the listed providers, which is not an out-of-box Threat Intelligence provider supported by MSTICPy?
- A. OTX
- B. VirusTotal
- C. Symantec
- D. XForce
Answer: C. Symantec
Explanation: MSTICPy supports providers like OTX, VirusTotal, and XForce out of the box. Symantec is not an out-of-box supported provider.
True/False: MSTICPy is only designed for threat analysts with proficient programming skills.
- True
- False
Answer: False
Explanation: MSTICPy is designed to simplify many of the routine tasks undertaken during an investigation, making it accessible for threat analysts of varying skill levels, including those with limited programming knowledge.
Multiple Select: MSTICPy provides pre-built analysis techniques including?
- A. Outlier detection
- B. Snap to grid analysis
- C. Time series decomposition
- D. Pivot Analysis
Answer: A, B, C, D
Explanation: MSTICPy provides pre-built analysis for various tasks such as outlier detection, snap to grid analysis, time series decomposition and pivot analysis.
True/False: MSTICPy only works with Windows Operating System.
- True
- False
Answer: False
Explanation: MSTICPy is platform-independent and can be used with any operating system that supports Python 6 or later, which includes Windows, MacOS, and Linux.
Interview Questions
What is MSTICPy used for in network security analysis?
MSTICPy is a tool developed by Microsoft for cybersecurity investigations and hunting in Jupyter Notebooks. It primarily provides data querying, enrichment, analysis and visualization features.
How can you install MSTICPy in Jupyter Notebooks?
You can install MSTICPy using pip by running the command
pip install msticpyin your Jupyter notebook.
Can MSTICPy be used with other notebooks apart from Jupyter?
Yes, MSTICPy can be used with other notebooks that support Python such as Visual Studio Code notebooks.
How to update MSTICPy to the latest version?
You can update MSTICPy to the latest version by running the command
pip install --upgrade msticpyin your Jupyter notebook.
Is it necessary to install the entire MSTICPy package to use a specific functionality?
No, it's not necessary as MSTICPy is designed with a modular structure. You can just install the required sub-packages.
What does the functionality MSTICPy.DataProviders offer?
MSTICPy.DataProviders module provides components to access and query various data sources like Azure Sentinel, Microsoft Defender Advanced Threat Protection, etc.
How can MSTICPy be used to visualize data?
MSTICPy provides several visualization tools such as time series, process trees, and others which help in analyzing and interpreting the security data.
Is there support for geolocation mapping in MSTICPy?
Yes, MSTICPy supports geolocation mapping where it can plot events on a map given IP addresses or host names.
How can you enlist available data providers with MSTICPy?
You can use the command
QueryProvider.list_providers()to enlist all available data providers with MSTICPy.
What role does the MSTICPy settings file play?
The MSTICPy settings file is used for storing configuration details like query provider options, Threat Intel provider options, and other configurations related to data providers and features.
Can you perform threat intelligence lookups using MSTICPy?
Yes, with MSTICPy you can perform threat intelligence lookups using multiple providers like VirusTotal, IBM XForce, and AlienVault.
How can MSTICPy help in threat analysis using data frames?
MSTICPy includes many pandas DataFrame accessor functions. These can calculate and append new features to your data, extract items from complex columns like dictionaries and lists, and parse Windows Event Logs to name a few.
What does the
TILookup
TILookupclass do in MSTICPy?
The
TILookupclass in MSTICPy is used to look up reputation data about different IoC types like IP addresses, URLs, or file hashes using configured Threat Intelligence services.
What is
Browser
Browserin MSTICPy?
The
Browserin MSTICPy is an interactive tool for filtering data in data frames. Filters can be defined to select the data of interest, and the results are immediately updated in the display.
Can we customize MSTICPy for personal use?
Yes, MSTICPy is open-source, hence users can customize it according to their requirements. However, proper coding conventions should be followed for successful customization.
extutorials.com