ConductEntityMatching — Analysis Tool Box

Introduction

Two datasets rarely spell the same company or person the same way — "Acme Corporation" in one system and "ACME Corp" in another are the same entity, but an exact join will never find that. ConductEntityMatching answers "which records across these two DataFrames actually refer to the same real-world entity, despite naming differences?" using multiple fuzzy string-matching algorithms scored side by side, so you can set a similarity threshold and review candidate matches rather than missing them entirely. Reach for it for deduplication, master data management, or any linkage task where exact keys don't exist or can't be trusted.

Teaching Note

The function is particularly useful for:

Customer data deduplication across multiple systems
Merging company records from different databases
Matching product catalogs from various vendors
Linking person records with name variations
Data integration and master data management
Matching addresses with different formatting
Reconciling financial records across sources

Parameters

Parameter	Default	Description
`dataframe_1`required	—	First pandas DataFrame containing records to match. Must include the primary key column and the columns specified in columns_to_compare.
`dataframe_1_primary_key`required	—	Name of the column in dataframe_1 that serves as the unique identifier for each record. Included in the output for reference.
`dataframe_2`required	—	Second pandas DataFrame containing records to match against dataframe_1. Must include the primary key column and the columns specified in columns_to_compare.
`dataframe_2_primary_key`required	—	Name of the column in dataframe_2 that serves as the unique identifier for each record. Included in the output for reference.
`levenshtein_distance_filter`	`None`	If specified, only pairs with Levenshtein edit distance less than or equal to this value are evaluated with fuzzy matching algorithms, pre-filtering candidates to improve performance.
`match_score_threshold`	`95`	Minimum similarity score (0-100) required for a pair to be included in results. Higher values are more restrictive.
`columns_to_compare`	`None`	List of column names to use for entity matching, concatenated (with spaces) to create the comparison strings. If None, all common columns between the two DataFrames are used.
`match_methods`	`['Partial Token Set Ratio', 'Weighted Ratio']`	List of fuzzy matching algorithms to apply: 'Ratio', 'Partial Ratio', 'Token Sort Ratio', 'Partial Token Sort Ratio', 'Token Set Ratio', 'Partial Token Set Ratio', or 'Weighted Ratio' (a weighted average of multiple methods, recommended).

Returns

A DataFrame containing potential matches: primary key columns from both DataFrames (suffixed _Entity 1/_Entity 2), an Entity 1 and Entity 2 concatenated comparison string, one column per match method with similarity scores (0-100), and the original columns_to_compare values (also suffixed). Returns an empty DataFrame with the correct structure if no matches are found.

Example

python

from analysistoolbox.data_processing import ConductEntityMatching
import pandas as pd

known_entities = pd.DataFrame({
    'entity_id': [1, 2, 3],
    'name': ['Acme Corporation', 'Tech Innovations Inc', 'Global Solutions Ltd'],
})

transactions = pd.DataFrame({
    'transaction_id': ['T001', 'T002', 'T003'],
    'counterparty': ['ACME Corp', 'Tech Innovations Incorporated', 'Global Solutns'],
})

matches = ConductEntityMatching(
    dataframe_1=known_entities,
    dataframe_1_primary_key='entity_id',
    dataframe_2=transactions,
    dataframe_2_primary_key='transaction_id',
    columns_to_compare=['name'],
    match_score_threshold=80,
    match_methods=['Weighted Ratio', 'Token Set Ratio'],
)