msticpy Microsoft

msticpy | Microsoft

Microsoft Threat Intelligence Python Security Tools.

msticpy is a library for InfoSec investigation and hunting in Jupyter Notebooks. It includes functionality to:

query log data from multiple sources
enrich the data with Threat Intelligence, geolocations and Azure resource data
extract Indicators of Activity (IoA) from logs and unpack encoded data
perform sophisticated analysis such as anomalous session detection and time series decomposition
visualize data using interactive timelines, process trees and multi-dimensional Morph Charts

It also includes some time-saving notebook tools such as widgets to set query time boundaries, select and display items from lists, and configure the notebook environment.

Msticpy Timeline Control

The package addresses three central needs for security investigators and hunters:

Acquiring and enriching data
Analyzing data
Visualizing data

We welcome feedback, bug reports, suggestions for new features and contributions.

For core install:

pip install msticpy

If you are using MSTICPy with Azure Sentinel you should install with the "azsentinel" extra package:

pip install msticpy[azsentinel]

or for the latest dev build

pip install git+https://github.com/microsoft/msticpy

To upgrade msticpy to the latest public non-beta release, run:

pip install --upgrade msticpy

Note it is good practice to copy your msticpyconfig.yaml and store it on your disk but outside of your msticpy folder, referencing it in an environment variable. This prevents you from losing your configurations every time you update your msticpy installation.

You can also browse through the sample notebooks referenced at the end of this document to see some of the functionality used in context. You can play with some of the package functions in this interactive demo on mybinder.org.

Built-in parameterized queries allow complex queries to be run from a single function call. Add your own queries using a simple YAML schema.

The TILookup class can lookup IoCs across multiple TI providers. built-in providers include AlienVault OTX, IBM XForce, VirusTotal and Azure Sentinel.

The input can be a single IoC observable or a pandas DataFrame containing multiple observables. Depending on the provider, you may require an account and an API key. Some providers also enforce throttling (especially for free tiers), which might affect performing bulk lookups.

The GeoIP lookup classes allow you to match the geo-locations of IP addresses using either:

Plotting Geo IP Location

The AzureData module contains functionality for enriching data regarding Azure host details with additional host details exposed via the Azure API. The AzureSentinel module allows you to query incidents, retrieve detector and hunting queries. AzureBlogStorage lets you read and write data from blob storage.

This subpackage contains several modules helpful for working on security investigations and hunting:

Detect unusual sequences of events in your Office, Active Directory or other log data. You can extract sessions (e.g. activity initiated by the same account) and identify and visualize unusual sequences of activity. For example, detecting an attacker setting a mail forwarding rule on someone's mailbox.

Time series analysis allows you to identify unusual patterns in your log data taking into account normal seasonal variations (e.g. the regular ebb and flow of events over hours of the day, days of the week, etc.). Using both analysis and visualization highlights unusual traffic flows or event activity for any data set.

Time Series anomalies

Msticpy Timeline Control

The process tree functionality has two main components:

Process Tree creation - taking a process creation log from a host and building the parent-child relationships between processes in the data set.
Process Tree visualization - this takes the processed output displays an interactive process tree using Bokeh plots.

There are a set of utility functions to extract individual and partial trees from the processed data set.

Interactive Process Tree

Lets you use MSTICPy functionality in an "entity-centric" way. All functions, queries and lookups that relate to a particular entity type (e.g. Host, IpAddress, Url) are collected together as methods of that entity class. So, if you want to do things with an IP address, just load the IpAddress entity and browse its methods.

Base64 and archive (gz, zip, tar) extractor. It will try to identify any base64 encoded strings and try decode them. If the result looks like one of the supported archive types it will unpack the contents. The results of each decode/unpack are rechecked for further base64 content and up to a specified depth.

Uses regular expressions to look for Indicator of Compromise (IoC) patterns - IP Addresses, URLs, DNS domains, Hashes, file paths. Input can be a single string or a pandas dataframe.

This module is intended to be used to summarize large numbers of events into clusters of different patterns. High volume repeating events can often make it difficult to see unique and interesting items.

Clustering based on command-line variability

This is an unsupervised learning module implemented using SciKit Learn DBScan.

Module to load and decode Linux audit logs. It collapses messages sharing the same message ID into single events, decodes hex-encoded data fields and performs some event-specific formatting and normalization (e.g. for process start events it will re-assemble the process command line arguments into a single string).

Module to support an investigation of a Linux host with only syslog logging enabled. This includes functions for collating host data, clustering logon events and detecting user sessions containing suspicious activity.

A module to support the detection of known malicious command line activity or suspicious patterns of command line activity.

A module to support investigation of domain names and URLs with functions to validate a domain name and screenshot a URL.

Alert browser

Example notebooks:

See the following notebooks for more examples of the use of this package in practice:

msticpy is OS-independent

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Last updated 1 year ago

Was this helpful?