If you spend enough time in a group chat, you start to recognize the distinct digital “voice” of your friends. One person might type “kya” while another strictly uses “kia”. One sends a wall of text, while another fires off five consecutive single-word messages.
But can a machine learn these idiosyncratic signatures?
For my latest project, I set out to answer exactly that. I built a Forensic Authorship Attribution System by ingesting, cleaning, and analyzing over 100,000 raw WhatsApp messages from my university group chats. The goal was to engineer a system capable of identifying individuals based entirely on their highly informal, code-mixed (Roman Urdu + English) texting habits.
Here is a breakdown of how the system was engineered, the machine learning techniques used to solve code-mixing, and the hard data on how the models performed.
By The Numbers: The Dataset
Real chat data is messy. I started with 19 group chat exports spanning over three years. Before any modeling could happen, the data required aggressive regex parsing to strip invisible Unicode control characters, mask personally identifiable information (PII) like phone numbers and URLs, and filter out system messages.
- Raw Corpus: 109,651 messages across 74 authors.
- Cleaned Dataset: 89,372 structural text messages.
- Active Senders Profiled: 36 (filtered to senders with > 100 messages).
- Average Message Length: Just 5.93 words.
Caption: Summary table of the 4 filtered senders displaying balanced message counts, average word counts, and average character counts.
1. The “Length Leakage” Trap: Why NLP Models Cheat
When I initially fed the chat data into a model, the accuracy was suspiciously high. Digging into the data, I discovered a classic machine learning pitfall: Length Leakage.
In instant messaging, people have radically different verbosity habits. The models weren’t actually learning linguistic style; they were lazily predicting authors based on message length.
The Fix: I engineered an iterative Gaussian Bell-Curve Balancing Algorithm. I forced every candidate author into an identical synthetic word-count distribution (a mean of 10 words, standard deviation of 6). By stripping the model of the ability to use message length as a crutch, I forced it to learn the actual syntax and vocabulary.
2. Cracking Roman Urdu & The 4-Author Benchmark
Standard NLP models are built for formal English. My dataset was a chaotic blend of English and Roman Urdu, which has zero spelling standards. The word for “good” might appear as “acha”, “achha”, or “achaa”. Standard Bag-of-Words (BOW) models treat these as completely separate tokens.
The Fix: I evaluated multiple vectorization pipelines, specifically testing Subword Character N-Grams (3-to-5 grams) against traditional word-level TF-IDF. By breaking text down into character sequences, the model could mathematically connect that “k-y-a-a” and “k-i-a-a” share similar sub-roots. I also engineered a 12-dimensional handcrafted stylometric vector tracking punctuation ratios, capitalization discipline, and emoji usage.
Here is how the architectures performed when tasked with identifying the top 4 most prolific authors (tested on a perfectly balanced 12,000-message cohort):
| Model Architecture | Feature Representation | Test Accuracy | Macro F1 |
| Word TF-IDF + NB | Word (1, 2)-grams, TF-IDF | 79.75% | 0.7979 |
| 5-Model Ensemble | Soft Probability Average | 79.67% | 0.7971 |
| Word BOW + NB | Word (1, 2)-grams, Count | 78.83% | 0.7888 |
| Char TF-IDF + LogReg | Char (3, 5)-grams, TF-IDF | 77.75% | 0.7777 |
Caption: Horizontal bar chart comparing all 4-author models. Word TF-IDF with Naive Bayes achieved the highest single-model accuracy, while the soft-voting ensemble closely matched it with superior stability.
Caption: Normalized confusion matrix demonstrating the clean diagonal separation achieved by the Word TF-IDF + Naive Bayes model.
3. Stress-Testing Scalability: 2 to 23 Authors
How does an NLP model behave when you add one suspect after another? To find out, I ran a progressive scaling experiment, starting with 23 authors and iteratively pruning the candidate pool down to 2, tracking exact degradation curves.
| Candidate Pool | Ensemble Accuracy | Word TF-IDF + NB Accuracy | Random Guess Baseline |
| 2 Senders | 94.13% | 93.72% | 50.0% |
| 4 Senders | 89.35% | 86.25% | 25.0% |
| 8 Senders | 77.88% | 68.11% | 12.5% |
| 23 Senders | 64.81% | 50.43% | 4.3% |
Caption: Line plot tracking all 6 model accuracies across 22 iterations, showing logarithmic accuracy decay as author complexity scales from 2 up to 23 senders.
This experiment revealed the Ensemble Advantage. At massive sender counts (15+), raw Word BOW models performed well. But in the critical 4-to-8 author range, the Soft-Voting Ensemble consistently provided the highest stability by balancing the strengths of word-level frequencies and character-level subwords.
Caption: Differential plot analyzing the margin between the Ensemble and the best individual model, highlighting where the soft-voting architecture provides the most value.
From Notebooks to a Production Dashboard
To make this tangible, I wrapped the entire predictive pipeline into a live, interactive Streamlit Web Dashboard.
The application utilizes lazy deserialization to load the pre-trained .pkl models efficiently into memory. You can type a phrase in Roman Urdu, and the app generates a real-time probability distribution across the suspects. I also built-in forensic UX guardrails—if a message is too short (under 4 words) or the confidence score drops below 50%, the dashboard automatically flags the prediction as analytically unsafe.
The Takeaway:
Applying machine learning to the messy, code-switched reality of South Asian instant messaging requires moving beyond standard textbook NLP pipelines. By combining rigorous data balancing and sub-word phonetic tracking, we can accurately capture the digital fingerprints hidden in our daily chats.