Introduction
Monthly exports, per-region extracts, and periodic survey dumps tend to pile up as separate CSV or Excel files that need to become one dataset before any real analysis can start. ImportDataFromFolder automates that consolidation: it reads every .csv and .xlsx file in a directory and vertically concatenates them into a single pandas DataFrame, either aligning columns to the first file's schema automatically or raising an error the moment a file's columns don't match — so schema drift gets caught instead of silently corrupting the combined dataset.
Batch data import and combination is essential for:
- Consolidating monthly or periodic reports into annual datasets
- Merging data exports from multiple sources or systems
- Aggregating survey responses collected in separate files
- Combining log files or transaction records across time periods
- Creating master datasets from departmental or regional files
- ETL (Extract, Transform, Load) workflows and data pipelines
- Simplifying analysis of distributed data collections
- Automating recurring data consolidation tasks
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
folder_pathrequired | | — | Absolute or relative path to the directory containing CSV and Excel files to import. All .csv and .xlsx files in this folder will be processed and combined. |
force_column_names_to_match | | True | Whether to automatically rename columns in subsequent files to match the first file's column names. If True, forces alignment by renaming columns (useful for files with minor naming variations). If False, raises ValueError when column names differ across files. Defaults to True. |
Returns
A pandas DataFrame containing all rows from all CSV and Excel files in the folder, vertically concatenated. Column names match the first file processed (if force_column_names_to_match is True) or are required to be consistent across all files (if False).
Example
from analysistoolbox.file_management import ImportDataFromFolder
# Combine monthly sales reports into one annual dataset
annual_sales = ImportDataFromFolder('data/monthly_reports')
# Imports all CSV and Excel files, forcing column alignment