CountMissingDataByGroup — Analysis Tool Box

Introduction

An overall missingness rate can hide a segment where the data is unusable — one region's survey responses might be 90% complete while another's are 90% missing, and the blended average won't tell you that. CountMissingDataByGroup answers "which segments have a data completeness problem I need to know about before I trust an aggregate?" by counting missing values per column within each group and providing a row count for context. Reach for it before pooling groups into a single analysis, or when auditing a newly merged dataset for source-specific gaps.

Teaching Note

The function is particularly useful for:

Identifying data quality issues across different segments or categories
Analyzing survey data to find patterns in non-response
Pre-processing data to determine if certain groups should be excluded due to sparse information
Validating data integrity after merging datasets from multiple sources
Monitoring data pipelines for unexpected drops in data completeness
Reporting on audit and compliance metrics for required fields

Parameters

Parameter	Default	Description
`dataframe`required	—	The pandas DataFrame to analyze.
`list_of_grouping_columns`required	—	A list of column names to group the data by. These variables define the segments for which missing data will be counted.
`list_of_columns_to_analyze`	`None`	A list of column names in which to count missing values. If None, the function counts missing values for all columns in the DataFrame.

Returns

A summary DataFrame indexed by the grouping columns, containing a Row count column (total records per group) followed by columns showing the count of missing values for each analyzed variable.

Example

python

from analysistoolbox.data_processing import CountMissingDataByGroup
import pandas as pd
import numpy as np

# Analyze missing demographic data by region
df = pd.DataFrame({
    'Region': ['East', 'East', 'West', 'West', 'North', 'East'],
    'Age': [25, np.nan, 30, np.nan, 40, 22],
    'Income': [50000, 60000, np.nan, np.nan, 75000, np.nan]
})
missing_summary = CountMissingDataByGroup(df, ['Region'])