Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection

In plain words

Phishing is when an attacker pretends to be a trusted party to get private information out of someone. This paper builds a classifier that reads an email's sender, subject and text and labels it as spam or not. It is trained on six public collections of spam emails that were merged into one set of about 82,000 messages.

The text of each email is turned into numbers. The numbers give more weight to words that are unusual across the collection and less to words that appear everywhere. The authors compare several standard classifiers on these numbers and find that a linear support vector machine does best. For individual predictions, they use LIME, a method that shows which words pushed one email towards "spam".

The winning model was placed in a small web page: a user pastes an email and gets a verdict. The reported accuracy is very high, but it comes from one split of one merged dataset, and the paper leaves out many details that someone rebuilding it would need.

TL;DR

• Classic text-classification pipeline for email: six public spam datasets merged into 82,486 emails (42,891 spam, 39,595 ham), TF-IDF or Word2Vec features, and an 80/20 split p. 6 p. 7 p. 8.
• Best author-reported model: linear-kernel SVC on TF-IDF, accuracy 0.991, precision 0.99, recall 0.99, F1 0.99 on the 16,498-email test split tab. 2 p. 8 p. 12.
• The same classifiers on Word2Vec features reach F1 0.71 to 0.83 in the paper's comparison tab. 2.
• LIME explains single predictions, and a Flask app serves the chosen model p. 10 p. 11.
• No official code was found, and the processed dataset is "available upon reasonable request" p. 17. Treat the numbers as unverified until someone rebuilds the pipeline.

Problem

Phishing causes financial losses and security breaches, and the authors frame it as impersonation of trusted entities to obtain sensitive information p. 1. They argue that much prior work relies on proprietary datasets or small public ones, and that high-scoring models are rarely deployed as usable applications p. 1 p. 5.

Their literature table lists accuracies between 95.5 and 99.00 for earlier email-classification systems, each on a different dataset tab. 1 tab. 3. The paper does not test any of them on a common benchmark, so the table does not rank methods.

The working assumption is that the email's sender, subject and body are available at prediction time p. 13. Receiver address and URL presence are tested as features and are dropped from the final model p. 16.

Method

The pipeline combines six public spam datasets, turns each email into a text field, cleans the text, vectorises it with TF-IDF (and, for comparison, Word2Vec), trains three classifiers, and keeps the best one p. 6 fig. 3. Each step below is taken from the paper's text and figure 3 p. 6.

1. Data collection. Enron and Ling are merged into one frame with subject, body and label, called mdf_1. CEAS, Nazario, Nigerian Fraud and SpamAssassin are merged into a second frame, mdf_2, with sender, receiver, subject, body, date and label p. 6.
2. Text combination. In mdf_1, subject and body are joined into a column named text_combined. In mdf_2, sender, date, subject and body are joined into text_combined; receiver is not in the list p. 7.
3. Cleaning. Punctuation and stop words are removed and the text is tokenized. The paper's text gives this order: tokenization first, then removal. Figure 3 shows removal before the merge and tokenization after it. The order is ambiguous p. 6 p. 7 fig. 3.
4. Integration. The two frames are stacked on text_combined. The result has 42,891 spam and 39,595 ham emails, 82,486 in total. The text rounds this to "approximately 82,500" p. 7 p. 8.
5. Vectorization. TF-IDF weights each word in a document by its frequency there times its rarity across the corpus eq. 1 p. 7. Word2Vec is also used to turn words into vectors, as an alternative feature set p. 8.
6. Split. The data is split 80% training and 20% test, giving 65,988 training and 16,498 test emails p. 8.
7. Classifiers. Three models are trained with random state 42 p. 9:
   • linear-kernel support vector classifier (SVC), which the authors report as their chosen model p. 8;
   • multinomial Naive Bayes, which handles word counts p. 9;
   • random forest of 100 decision trees, each on a random subset of features at each split, with a majority vote p. 9.
8. Metrics. Accuracy, precision, recall and F1 are reported on the test split eq. 2 eq. 3 eq. 4 eq. 5 p. 9 p. 10.
9. Explanation. LIME fits a simple local model around one email and reports the words with the largest weights p. 10 p. 11.
10. Deployment. The chosen model is exported and loaded into a Flask web app. The user pastes sender, subject and body into a form, and the app returns spam or safe p. 11 p. 13.

The ablation tests whether each input field is needed p. 16. Removing the receiver field makes no significant difference, and a binary URL column adds little, so it is excluded p. 16. Merging all textual fields into one column raises F1 from 0.71 to 0.82; in tab. 2, those are the two Word2Vec SVC rows p. 16.

TF-IDF weight for word w in document d, within corpus D eq. 1 p. 7:

\text{TF-IDF}(w,d) = \text{TF}(w,d) \times \text{IDF}(w,D) eq. 1

TF(w,d) is the count of w in d divided by the number of words in d. IDF(w,D) is the logarithm of |D| divided by the number of documents that contain w p. 8. The logarithm's base and any smoothing are not stated.

Metrics, with TP, TN, FP and FN as the usual confusion counts p. 9 p. 10:

\text{Accuracy} = \frac{TP + TN}{\text{Total emails}} eq. 2

\text{Precision} = \frac{TP}{TP + FP} eq. 3

\text{Recall} = \frac{TP}{TP + FN} eq. 4

F1 = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} eq. 5

The core training path, in the paper's order fig. 3 p. 6 p. 7 p. 8 p. 9:

# Training pipeline for the reported best model  [fig. 3] [p. 6-9]
# Input:  six labelled spam datasets, one row per email
# Output: fitted TF-IDF weights and a linear SVC

def train_best_model(rows):
    # Build one text field per email, with the fields used by each group  [p. 7]
    texts = [combine_fields(row) for row in rows]
    # Label 1 is spam and 0 is ham  [p. 11]
    labels = [row.label for row in rows]
    # Remove punctuation and stop words, then tokenize (order ambiguous)  [p. 7]
    tokens = [
        tokenize(remove_stop_words(strip_punctuation(text)))
        for text in texts
    ]
    # Split 80% train and 20% test; stratification is not stated  [p. 8]
    train_idx, test_idx = split_80_20(len(tokens))
    # Weight each term by TF-IDF  [eq. 1] [p. 7]; fit scope is not stated
    weights = fit_tfidf(tokens[train_idx])
    train_x = apply_tfidf(weights, tokens[train_idx])
    # Linear-kernel SVC; random state is 42 for all computations  [p. 9]
    model = fit_linear_svc(train_x, labels[train_idx], random_state=42)
    return weights, model

Hyperparameters

Results

Author-reported. Setup: the six-dataset merged corpus, 80/20 split, one test set of 16,498 emails p. 8 p. 11. Table 2 compares the authors' models and feature sets on that corpus and on a second, unexplained subset tab. 2.

Model results tab. 2 p. 11 p. 12. Labels: 1 = spam, 0 = ham p. 11. Values are as printed, to two decimals in the precision, recall and F1 columns.

The text states the best model is SVM with TF-IDF, at 99.1% accuracy and F1 of 99 p. 12.

Comparison with literature tab. 3 p. 12 p. 13. These are the paper's reported numbers on the datasets and splits of each cited work. They are not a like-for-like comparison with the proposed row.

Figures 5 to 7 show the deployed page and LIME output for one generated phishing-like email, with prediction probabilities of 0.08 (not spam) and 0.92 (spam) fig. 5 fig. 6 fig. 7 p. 14 p. 15. The paper gives this as an illustration, not as a measured result.

Paper vs code

No official code.

Searched, in order: the paper and its arXiv metadata have no code link; the live site the paper names, https://phishingdetection.onrender.com/ p. 13, returned HTTP 503 on two attempts, so the project-page step is incomplete. A GitHub repository search for "phishing email detection lime", "phishing email interpretable robust web" and "Al-Subaiey phishing" found no repository owned by an author. Third-party repositories were not used.

The paper says the processed dataset is "available upon reasonable request" from the corresponding author p. 17. No discrepancy between paper and code can be checked, because no code was found.

Pitfalls

• Phishing or spam labels. The title and abstract say phishing, but every dataset is a spam collection (Enron, Ling, CEAS, Nazario, Nigerian Fraud, SpamAssassin) and the labels are spam and ham p. 6 p. 11. No phishing-specific label or dataset is described. A model trained on spam labels may not behave as a phishing detector on new phishing emails. The paper does not evaluate this.
• Selection on the test split. Table 2 lists eleven configurations, and all are scored on the same 16,498-email test split tab. 2 p. 8. The headline 0.991 is the best of the rows on the merged dataset. The paper does not say whether a validation split or repeated runs were used, or whether choices were made after seeing test scores.
• One split, one seed. Results come from one 80/20 split with random state 42 p. 8 p. 9. No variance, confidence interval or repeated split is reported. Differences of 0.01 between rows of Table 2 cannot be ranked from these numbers.
• Unexplained subset. Some Table 2 rows (for example svm_994, the highest accuracy in the table at 0.994) use a different dataset of 28,457 spam and 21,403 ham tab. 2. The methods section does not describe this subset. The row is not comparable with the proposed one.
• The "-url" rows. Rows marked "-url" appear in Table 2 without any explanation of what the suffix changes tab. 2. The discussion says URL presence was tested and dropped p. 16, which may be what the suffix means. That is an inference.
• Field list conflicts. Section 2.2 merges sender, date, subject and body for mdf_2 p. 7. The ablation says the receiver and URL features were tested and dropped p. 16. Which fields the final model used is therefore clear only from the deployment description p. 13.
• Preprocessing order. The text gives tokenization before punctuation and stop-word removal p. 7. Figure 3 shows removal first fig. 3. The order changes the token set, so an implementation must pick one and check its effect.
• No deduplication check. The paper does not state whether duplicate or near-duplicate emails were removed, either within a dataset or across the six sources p. 6 p. 7. Duplicates across the train and test split would inflate the scores. The paper does not check this.
• Word2Vec is under-specified. The Word2Vec results (F1 0.71 to 0.83) are far below TF-IDF's tab. 2. The paper does not say how word vectors become a document vector, which can be a large part of the gap. Treat the comparison as the authors' setup, not a verdict on Word2Vec.
• TF-IDF scope. It is not stated whether the TF-IDF vocabulary and IDF were fitted on the training rows only. Fitting on all rows would leak test information p. 8.
• Rounding. Precision, recall and F1 are printed to two decimals, and the text gives "99" for F1 p. 12. The exact values are not stated.
• LIME is qualitative. LIME is shown on one generated email p. 15. The paper does not measure whether the explanations are faithful or stable. The text says highlighted words are coloured red, but figure 7 shows orange p. 15 fig. 7.
• Deployment is untested. The paper says the app is "real time" p. 11 p. 13. It gives no latency, throughput or error rate. The hosted site returned HTTP 503 when this card was compiled, so it could not be checked.
• Dataset availability. The processed dataset is available "upon reasonable request" p. 17. Without it, the exact split and the random state 42 cannot be reproduced.
• Section numbering. The text refers to "Section 1" to "Section 6", but the printed headings are numbered differently (the methodology subsections are printed as 1 to 7, and "Model Deployment" is printed as 7) p. 6 p. 11. This card cites printed headings by page.
• Author claims not checked. "largest available public dataset" p. 1 is the authors' statement. The paper gives no comparison with other public collections.

Lineage

• Builds on: TF-IDF (the paper cites Encyclopedia of Machine Learning, ref. 37) and Word2Vec (Mikolov et al., ref. 38) p. 7 p. 8. LIME is used, and the paper cites Garreau's analysis of it (ref. 40) p. 10. None of these have arXiv ids in this wiki, so none are linked.
• Compared against: the twelve systems in tab. 3 and the in-paper variants in tab. 2. None have cards in this wiki.
• Superseded by: not known.