Spam Email Detector

Spam Email Detector

An end-to-end machine learning solution designed to accurately classify emails as spam or legitimate (ham) using Support Vector Machines.

Project Overview

This project focuses on building a high-performance spam detection system from the ground up. Unlike simple tutorials, this system handles the entire lifecycle of a machine learning project: from aggregating multiple raw datasets (totaling over 94,000 emails) into a MySQL database to deploying a Flask API for real-time predictions.

Technologies Used

• Language: Python
• ML Libraries: Scikit-learn, Joblib, Pandas
• Backend API: Flask
• Database: MySQL
• Frontend: React (Client Interface)
• Tools: Regex, PyCharm, DataGrip

Key Features

1. Massive Dataset Integration: Merged and balanced four distinct datasets to create a corpus of 94,000 emails.
2. Custom Feature Engineering: Extracted specific metadata features such as symbol ratios, link counts, and keyword presence (e.g., "Win", "Buy Now").
3. Database-Driven Pipeline: Utilized MySQL tables for structured storage of raw data, cleaned features, and model specifications instead of flat files.
4. Real-time API: A Flask backend that serves the trained model, allowing for instant classification of user input.

Project Details

The core of this project is its rigorous data pipeline. It implements a custom preprocessing engine that sanitizes raw text, removing HTML tags and noise, before feeding it into a Support Vector Machine (SVM) model tuned with various kernels (Linear, RBF, Poly).

Challenges

• Data Quality: Cleaning varied formats of raw text data from multiple sources to ensure consistent feature extraction.
• Class Imbalance: Ensuring the model doesn't become biased towards the more frequent "Ham" emails by carefully balancing the training sets.
• Model Selection: Determining the optimal kernel and hyperparameters for the SVM to maximize precision and recall.

Solutions

• Unified SQL Storage: Created a central finalDatasetTables in MySQL to manage the merged data, ensuring integrity and easy splitting for training/validation.
• Regex Preprocessing: Developed a robust set of regular expressions to identify and quantify suspicious patterns like excessive capitalization or currency symbols.
• Iterative Tuning: Implemented an automated testing loop to evaluate different model configurations and store performance metrics in the database.

Conclusion

The Spam Email Detector showcases the application of classical machine learning algorithms to solve real-world cybersecurity problems, emphasizing the importance of clean data and structured engineering pipelines.