Entity Matching for AML and Compliance

Anti-money laundering (AML) and sanctions compliance require matching names across systems that weren't designed to talk to each other. A customer might be recorded as "Mohammed Al-Rashid" in your CRM, "M. Alrashid" on a wire transfer, and "Mohammad Al Rasheed" on a government watchlist. Exact string matching fails completely. You need fuzzy matching — and a way to cluster the results to surface the highest-risk records.

This tutorial uses ConductEntityMatching for fuzzy name matching and CreateKMeansClusters to segment matched records by risk profile.

Prerequisites

Python 3.9+
analysistoolbox installed (pip install analysistoolbox)
A watchlist file and a customer/transaction file (we'll create synthetic ones)

Step 1: Build synthetic data

python

import pandas as pd
import numpy as np

np.random.seed(42)

# Synthetic customer database
customers = pd.DataFrame({
    'customer_id': range(1001, 1021),
    'name': [
        'Mohammed Al-Rashid', 'John Smith', 'Fatima Al-Zahra', 'Robert Johnson',
        'Ibrahim Hassan', 'Maria Garcia', 'Ahmed El-Sayed', 'Jennifer Williams',
        'Yusuf Abdullahi', 'David Brown', 'Aisha Bint Khalid', 'Michael Davis',
        'Omar Shaikh', 'Sarah Miller', 'Tariq Mahmood', 'Emily Wilson',
        'Hassan Al-Farsi', 'Christopher Moore', 'Nadia Petrov', 'James Taylor',
    ],
    'country': ['SA', 'US', 'AE', 'US', 'SO', 'MX', 'EG', 'US', 'NG', 'US',
                'KW', 'US', 'PK', 'US', 'PK', 'US', 'IR', 'US', 'RU', 'US'],
    'transaction_volume_30d': np.random.exponential(scale=50_000, size=20).round(-2),
})

# Synthetic watchlist (OFAC-style, with name variants)
watchlist = pd.DataFrame({
    'watchlist_id': range(1, 9),
    'listed_name': [
        'M. Alrashid', 'Yusuf Abdullahi Mohamed', 'Ahmed Al-Sayed',
        'Tariq Mehmood', 'Hassan Farsi', 'Aisha Khalid', 'Ibrahim Hasan', 'Nadia Petrova',
    ],
    'program': ['OFAC-SDN', 'OFAC-SDN', 'EU-CONSOL', 'UN-1267',
                'OFAC-SDN', 'EU-CONSOL', 'OFAC-SDN', 'EU-CONSOL'],
    'risk_score': [9, 8, 7, 8, 9, 6, 7, 6],
})

print(f"Customers: {len(customers)} records")
print(f"Watchlist: {len(watchlist)} entries")

Step 2: Run entity matching

ConductEntityMatching computes fuzzy string similarity between every customer name and every watchlist name, returning match scores.

python

from analysistoolbox.data_processing import ConductEntityMatching

matches = ConductEntityMatching(
    dataframe_1=customers,
    dataframe_2=watchlist,
    join_column_in_dataframe_1='name',
    join_column_in_dataframe_2='listed_name',
    minimum_similarity_score=0.60,
    print_matches=True,
)

print(f"\nMatches found: {len(matches)}")
print(matches[['name', 'listed_name', 'similarity_score', 'program']].sort_values(
    'similarity_score', ascending=False
))

Teaching Note

The minimum_similarity_score threshold is a policy decision, not a statistical one. A threshold of 0.60 is relatively permissive — you'll get more matches but more false positives. Compliance teams typically start at 0.75–0.85 for automated blocking and use human review for 0.60–0.75. Set this based on your risk tolerance, not on what makes the output look clean.

Step 3: Enrich matches with risk signals

Merge the match results back to the customer data to bring in transaction volume and country risk:

python

# High-risk jurisdiction list (simplified)
high_risk_countries = {'IR', 'KP', 'SY', 'CU', 'SO', 'YE', 'LY', 'SD', 'MM'}

enriched = matches.merge(
    customers[['name', 'customer_id', 'country', 'transaction_volume_30d']],
    on='name',
    how='left'
).merge(
    watchlist[['listed_name', 'risk_score']],
    on='listed_name',
    how='left'
)

enriched['high_risk_country'] = enriched['country'].isin(high_risk_countries).astype(int)
enriched['combined_risk'] = (
    enriched['similarity_score'] * 0.4
    + enriched['risk_score'].fillna(0) / 10 * 0.4
    + enriched['high_risk_country'] * 0.2
)

print(enriched[['name', 'country', 'similarity_score', 'risk_score', 'combined_risk']]
      .sort_values('combined_risk', ascending=False))

Step 4: Cluster matched records by risk profile

With many matches, you need to triage. CreateKMeansClusters groups matched records into risk tiers automatically.

python

from analysistoolbox.descriptive_analytics import CreateKMeansClusters

features = ['similarity_score', 'combined_risk', 'transaction_volume_30d']
cluster_input = enriched[features].fillna(0)

clustered = CreateKMeansClusters(
    dataframe=cluster_input,
    list_of_columns_to_use_for_clustering=features,
    number_of_clusters=3,
    random_seed=42,
    plot_cluster_results=True,
)

enriched['risk_tier'] = clustered['Cluster']

# Review the tiers
print(enriched.groupby('risk_tier')[['similarity_score', 'combined_risk', 'transaction_volume_30d']].mean())

Tip

After clustering, manually inspect the cluster centroids to assign tier labels. The cluster with the highest combined_risk and similarity_score average becomes your "Escalate Immediately" tier. The lowest becomes "Monitor". Re-run with number_of_clusters=4 if your compliance team needs finer granularity.

Step 5: Generate a review queue

python

# Priority queue: highest risk tier first, then by combined_risk
review_queue = (
    enriched
    .sort_values(['risk_tier', 'combined_risk'], ascending=[False, False])
    [['customer_id', 'name', 'listed_name', 'program', 'similarity_score',
      'combined_risk', 'transaction_volume_30d', 'risk_tier']]
)

print("=== PRIORITY REVIEW QUEUE ===")
print(review_queue.to_string(index=False))

# Export for compliance team
review_queue.to_csv('aml_review_queue.csv', index=False)

What makes this approach defensible

Teaching Note

AML compliance programs are regulated — any automated screening tool must be explainable to regulators. This approach is defensible because: (1) the fuzzy match score is a transparent, auditable number; (2) the risk factors (similarity, watchlist risk score, jurisdiction) are explicitly weighted; (3) the clustering is deterministic given a fixed seed. You can show a regulator exactly why Record X was in Tier 1 and Record Y was in Tier 3. Compare this to black-box ML models, which are harder to justify to a bank examiner.

Tuning guidance:

If too many false positives: raise minimum_similarity_score or add a secondary confirming field (date of birth, address)
If too many false negatives: lower the threshold and add human review for the 0.60–0.75 band
At scale (millions of records): pre-filter by country or transaction amount before running entity matching to reduce the candidate pairs

Full code

python

import pandas as pd
import numpy as np
from analysistoolbox.data_processing import ConductEntityMatching
from analysistoolbox.descriptive_analytics import CreateKMeansClusters

np.random.seed(42)

customers = pd.DataFrame({
    'customer_id': range(1001, 1021),
    'name': [
        'Mohammed Al-Rashid', 'John Smith', 'Fatima Al-Zahra', 'Robert Johnson',
        'Ibrahim Hassan', 'Maria Garcia', 'Ahmed El-Sayed', 'Jennifer Williams',
        'Yusuf Abdullahi', 'David Brown', 'Aisha Bint Khalid', 'Michael Davis',
        'Omar Shaikh', 'Sarah Miller', 'Tariq Mahmood', 'Emily Wilson',
        'Hassan Al-Farsi', 'Christopher Moore', 'Nadia Petrov', 'James Taylor',
    ],
    'country': ['SA', 'US', 'AE', 'US', 'SO', 'MX', 'EG', 'US', 'NG', 'US',
                'KW', 'US', 'PK', 'US', 'PK', 'US', 'IR', 'US', 'RU', 'US'],
    'transaction_volume_30d': np.random.exponential(scale=50_000, size=20).round(-2),
})

watchlist = pd.DataFrame({
    'watchlist_id': range(1, 9),
    'listed_name': [
        'M. Alrashid', 'Yusuf Abdullahi Mohamed', 'Ahmed Al-Sayed',
        'Tariq Mehmood', 'Hassan Farsi', 'Aisha Khalid', 'Ibrahim Hasan', 'Nadia Petrova',
    ],
    'program': ['OFAC-SDN', 'OFAC-SDN', 'EU-CONSOL', 'UN-1267',
                'OFAC-SDN', 'EU-CONSOL', 'OFAC-SDN', 'EU-CONSOL'],
    'risk_score': [9, 8, 7, 8, 9, 6, 7, 6],
})

high_risk_countries = {'IR', 'KP', 'SY', 'CU', 'SO', 'YE', 'LY', 'SD', 'MM'}

matches = ConductEntityMatching(
    dataframe_1=customers, dataframe_2=watchlist,
    join_column_in_dataframe_1='name', join_column_in_dataframe_2='listed_name',
    minimum_similarity_score=0.60,
)

enriched = (
    matches
    .merge(customers[['name', 'customer_id', 'country', 'transaction_volume_30d']], on='name', how='left')
    .merge(watchlist[['listed_name', 'risk_score']], on='listed_name', how='left')
)

enriched['high_risk_country'] = enriched['country'].isin(high_risk_countries).astype(int)
enriched['combined_risk'] = (
    enriched['similarity_score'] * 0.4
    + enriched['risk_score'].fillna(0) / 10 * 0.4
    + enriched['high_risk_country'] * 0.2
)

features = ['similarity_score', 'combined_risk', 'transaction_volume_30d']
clustered = CreateKMeansClusters(
    dataframe=enriched[features].fillna(0),
    list_of_columns_to_use_for_clustering=features,
    number_of_clusters=3, random_seed=42,
)
enriched['risk_tier'] = clustered['Cluster']

enriched.sort_values(['risk_tier', 'combined_risk'], ascending=[False, False]).to_csv(
    'aml_review_queue.csv', index=False
)