A visitor typed
A Python script that reads a folder of CSV files, combines them into one file, removes duplicate rows, and saves the result as combined_output.csvMore visitor examples →Report this example
import pandas as pd
import glob
import os
# --- CONFIGURATION: change these to match your setup ---
INPUT_FOLDER = "." # folder containing the CSV files
OUTPUT_FILE = "combined_output.csv"
FILE_PATTERN = "*.csv" # only files matching this pattern
# -------------------------------------------------------
# Find all CSV files in the folder
csv_files = glob.glob(os.path.join(INPUT_FOLDER, FILE_PATTERN))
if not csv_files:
print("No CSV files found.")
exit()
# Read and combine all CSV files
dataframes = []
for file in csv_files:
df = pd.read_csv(file)
dataframes.append(df)
combined = pd.concat(dataframes, ignore_index=True)
# Remove duplicate rows (keep first occurrence)
combined.drop_duplicates(inplace=True)
# Save the result
combined.to_csv(OUTPUT_FILE, index=False)
print(f"Combined {len(csv_files)} files -> {OUTPUT_FILE} ({len(combined)} rows after deduplication)")This script finds all CSV files in the given folder, reads them into a single table using pandas, removes any duplicate rows, and writes the result to combined_output.csv. To run it, first install pandas (pip install pandas), then place the script in or above your CSV folder, adjust INPUT_FOLDER if needed, and run python script_name.py.