CreateDataOverview — Analysis Tool Box

Introduction

Before modeling or aggregating a new dataset, you need to know what you're actually working with — what types the columns are, how much is missing, how many distinct values each variable takes. CreateDataOverview answers "what does this dataset look like, structurally and in terms of quality, before I build anything on top of it?" by producing a single data-dictionary DataFrame with type, missingness, cardinality, and distribution statistics for every column. Reach for it as the first step of any new analysis, replacing a dozen ad-hoc .info() / .describe() / .isna().sum() calls with one.

Teaching Note

The function is particularly useful for:

Creating automated data dictionaries for documentation
Auditing data completeness and identifying problematic columns
Comparing data types against expected schemas
Rapidly assessing the distribution and boundaries of numeric features
Identifying high-cardinality categorical variables
Visualizing missingness patterns across large datasets
Initial Exploratory Data Analysis (EDA) on new data sources

Parameters

Parameter	Type	Default	Description
`dataframe`required		—	The pandas DataFrame to analyze. Can contain numeric, categorical, and datetime data types.
`plot_missingness`		`False`	If True, generates a horizontal bar chart visualizing the percentage of missing values for each column, sorted by descending missingness.

Returns

A summary DataFrame where each row represents a variable from the original dataset, with columns: Variable, Data Type, Missing Count, Missing Percentage, Non Missing Count, Unique Value Count, Top Value, Frequency of Top Value, Minimum, Maximum, and Range.

Example

python

from analysistoolbox.data_processing import CreateDataOverview
import pandas as pd
import numpy as np

sales_data = pd.DataFrame({
    'TransactionID': range(100),
    'Product': ['A', 'B', 'C', np.nan] * 25,
    'Amount': np.random.uniform(10, 500, 100),
    'Date': pd.date_range('2023-01-01', periods=100)
})
overview = CreateDataOverview(sales_data, plot_missingness=True)