Introduction
A categorical column with a long tail of rarely-occurring values — dozens of
industries each representing under 1% of rows — can destabilize a model or clutter a
chart with noise no one can read. CreateRareCategoryColumn answers "which
categories are too rare to treat individually, and how do I collapse them into a
single, honest 'Other' bucket?" by consolidating any category below a frequency
threshold you choose. Reach for it before feeding categorical data into a model or a
visualization, whenever a handful of dominant categories are being overwhelmed by a
long tail of rare ones.
Teaching Note
The function is particularly useful for:
- Reducing high cardinality in categorical features
- Improving the performance and stability of machine learning models
- Simplifying visualizations by grouping long tails of rare categories
- Handling sparse categories that lack statistical significance
- Data cleaning where numerous variations of minor labels exist
- Standardizing categorical data for reporting and dashboards
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
dataframerequired | | — | A pandas DataFrame containing categorical data. |
categorical_column_namerequired | | — | The name of the column containing the categorical variable to analyze. Supports string or object dtypes. |
rare_category_label | | 'Other' | The string label to assign to any category that falls below the threshold. |
rare_category_threshold | | 0.01 | The minimum relative frequency (0 to 1) required for a category to remain distinct. Categories below this value are grouped. |
new_column_suffix | | None | The suffix to append to the original column name for the new binned column. If None, defaults to ' (with {rare_category_label})'. |
Returns
The input DataFrame with a new column added, where infrequent categories have been
replaced by rare_category_label.
Example
python
from analysistoolbox.data_processing import CreateRareCategoryColumn
import pandas as pd
# Group rare industries into "Other" (threshold 15%)
clients = pd.DataFrame({
'company': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J'],
'industry': ['Tech', 'Tech', 'Tech', 'Finance', 'Finance', 'Retail', 'Agri', 'Hosp', 'Edu', 'Gov']
})
clients = CreateRareCategoryColumn(
clients,
'industry',
rare_category_threshold=0.15
)
# 'Agri', 'Hosp', 'Edu', 'Gov' each appear only 10% and will be grouped into 'Other'