Introduction
Staring at an unfamiliar dataset, the hardest part is often just knowing where to look first — which columns are worth plotting, what relationships might matter, which questions are even askable. GenerateEDAWithLIDA uses Microsoft's LIDA framework to have a large language model do that first pass for you: it builds a semantic summary of the dataset, proposes a set of analytical questions with rationale, and can generate (and optionally execute) the visualization code needed to answer each one. It turns the blank-page problem of exploratory data analysis into a reviewable list of starting points.
AI-powered EDA with LIDA is essential for:
- Rapid initial data exploration and hypothesis generation
- Automated insight discovery in unfamiliar datasets
- Generating visualization ideas for complex multivariate data
- Reducing time-to-insight for data analysts and scientists
- Creating reproducible EDA workflows with generated code
- Educational purposes to learn effective EDA techniques
- Augmenting human analysis with AI-suggested perspectives
- Standardizing EDA practices across teams and projects
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
dataframerequired | | — | A pandas DataFrame to analyze. LIDA will examine its structure, data types, distributions, and relationships to generate relevant EDA goals. |
llm_api_keyrequired | | — | API key for the LLM provider (e.g., OpenAI API key). Required for authentication with the language model service. |
llm_provider | | 'openai' | Name of the LLM provider to use. Supported options include 'openai', 'azure-openai', 'palm', 'cohere', and others supported by LIDA. Defaults to 'openai'. |
llm_model | | 'gpt-3.5-turbo' | Specific model name to use from the provider. For OpenAI: 'gpt-3.5-turbo', 'gpt-4', 'gpt-4-turbo'. More capable models generate better insights but cost more. Defaults to 'gpt-3.5-turbo'. |
visualization_library | | 'seaborn' | Python visualization library for generated code. Options: 'seaborn', 'matplotlib', 'plotly', 'altair', 'ggplot'. Defaults to 'seaborn'. |
goal_temperature | | 0.5 | Temperature parameter (0-1) for goal generation. Higher values (0.7-1.0) produce more creative/diverse goals; lower values (0.1-0.3) produce more focused/conservative goals. Defaults to 0.50. |
code_generation_temperature | | 0.05 | Temperature parameter (0-1) for code generation. Lower values (0.0-0.2) produce more deterministic, reliable code; higher values may introduce variability. Defaults to 0.05. |
data_summary_method | | 'llm' | Method for data summarization. 'llm' uses the language model for semantic understanding; 'default' uses statistical methods. LLM method provides richer insights. Defaults to 'llm'. |
number_of_samples_to_show_in_summary | | 5 | Number of sample rows to include in the data summary sent to the LLM. More samples provide better context but increase token usage. Defaults to 5. |
return_data_fields_summary | | True | Whether to return a DataFrame containing the LLM-generated field descriptions and properties. Useful for understanding how LIDA interpreted the data. Defaults to True. |
number_of_goals_to_generate | | 5 | Number of EDA goals/questions to generate. Each goal includes a question, rationale, and recommended visualization. More goals provide broader coverage. Defaults to 5. |
plot_recommended_visualization | | False | Whether to execute the generated visualization code and display the plots. Requires the specified visualization library to be installed. Defaults to False. |
show_code_for_recommended_visualization | | False | Whether to print the generated Python code for each visualization. Useful for learning and customization. Defaults to False. |
Returns
A pandas DataFrame or None. If return_data_fields_summary is True, returns a DataFrame with columns like description (LLM-generated semantic description of each field), dtype, samples, and additional properties such as min/max/mean for numeric fields. If False, returns None. The function prints EDA goals and visualizations to stdout.
Example
from analysistoolbox.descriptive_analytics import GenerateEDAWithLIDA
import pandas as pd
# Basic EDA goal generation with OpenAI
sales_df = pd.DataFrame({
'date': pd.date_range('2024-01-01', periods=100),
'product': ['A', 'B', 'C'] * 33 + ['A'],
'revenue': [100 + i * 10 for i in range(100)],
'units_sold': [10 + i % 20 for i in range(100)]
})
field_summary = GenerateEDAWithLIDA(
sales_df,
llm_api_key='sk-...',
llm_model='gpt-4',
number_of_goals_to_generate=3
)
# Generates 3 analytical questions with rationale