multimodal-phishing-detection-system: Multimodal Phishing Detection System: When a Fake Site Has to Lie Twice

A phishing detector that reads the URL, looks at the page, and deliberately trusts the messier clue more when the two disagree.

6 to 7 min read • View on GitHub • More from AnkitMukherjee044

A phishing page appears as two competing identities. One side looks polished and familiar, while the other side is a messy URL ribbon wrapped like a forged stamp. Both are funneled into a single scale that tips toward the address bar. The image explains the repo's core idea: phishing is not one deception, but two signals that have to be reconciled. — The repo treats phishing as a dispute between appearance and identity. The final call comes from weighing both signals, not trusting either one in isolation.

Key Takeaways

The repo's central move is not machine learning for its own sake, but a security judgment: it treats URL signals as more trustworthy than page appearance when the two conflict.
Late fusion is the product here, because the system resolves disagreement between two weak witnesses instead of pretending one modality can carry phishing detection alone.
MobileNetV2 and focal loss give the visual branch a way to survive the hard cases, but the project still behaves like a compact research prototype rather than a production platform.
The strongest value of the repo is educational clarity, because it turns multimodal phishing detection into a pipeline you can inspect, reason about, and extend.

The deception is split across two surfaces

Phishing works because attackers can win on two fronts at once. They can make a page look legitimate, and they can make the URL look just plausible enough to pass a quick glance.

This repo starts from a useful insult to intuition: the eye is easier to fool than the address bar. So instead of asking one model to solve everything, it asks two models to vote, then gives the URL vote more weight.

Why the repo trusts the URL more than the screenshot

The key design choice is late fusion with a 0.6 URL / 0.4 image split. That is a small number with a big editorial opinion baked into it.

The system is easiest to understand as two independent witnesses. The URL branch and the screenshot branch both speak, then a weighted fusion block decides how much to trust each one.

Approach	Input signal	Main strength	Main weakness
URL-only detectors	Lexical features from the address bar	Fast and cheap	Can miss visually convincing impersonations
Screenshot-only detectors	Rendered page appearance	Good at brand mimicry	Easier to spoof with cosmetic changes
Multimodal late-fusion detectors	URL plus screenshot	Covers two attack surfaces	More moving parts and more failure modes

The weighting matters because phishing is asymmetric. Attackers can restyle a page quickly, but they cannot always erase the structural weirdness of a suspicious domain. The repo turns that intuition into a score.

What the app actually does

The app is a Streamlit wrapper around two trained models. A user enters a URL, the app normalizes it, the local URL model scores it, and a screenshot service renders the site for the visual model.

def clean_url(url):
    url = url.replace('https://', '')
    url = url.replace('http://', '')
    url = url.replace('www.', '')
    return url.strip('/')

url_score = url_model.predict_proba([clean_url(user_url)])[0][1]
img_score = 0.5  # fallback when screenshot capture fails
final_score = (0.6 * url_score) + (0.4 * img_score)

That fallback is a quiet but important piece of engineering. If the screenshot API fails, the app does not collapse. It degrades gracefully, which means the URL signal can still carry the decision.

The visual branch is built to survive the hard cases

The screenshot model is based on MobileNetV2, which is a practical choice for a visual classifier that needs to stay relatively light. The interesting part is not the backbone alone, but the loss function.

The notebook uses focal loss. That tells you the author cares about hard examples, not just easy wins. In phishing, the hardest cases are often the most dangerous ones: pages that look almost right.

A close-up of two instruments aimed at the same suspicious site. One magnifies the URL characters, while the other captures the rendered page. Their outputs converge at a weighted balance marked 60 and 40, with the URL side slightly heavier. The image explains why the repo uses late fusion instead of equal voting. — The model does not average two opinions mechanically. It resolves conflict by giving the URL branch more authority when the signals disagree.

The project is a prototype, and that is part of its value

This repo reads like a research prototype that was pushed into a usable app, not a product polished by a platform team. That is not a flaw. It is the reason the architecture is easy to inspect.

Prototype trait	What it signals	Tradeoff
Notebook lineage	The models were explored in notebooks first	Training artifacts are easy to follow but not production-hardened
Local pickle artifact	The URL model is packaged for quick loading	Serialization is convenient but brittle across environments
Remote screenshot dependency	The app can inspect live pages	The system now depends on an external service
Graceful fallback	The app stays usable when screenshots fail	Confidence drops, but the pipeline keeps moving

That is the real shape of the repo: a compact path from research to demo, with enough engineering discipline to keep the demo from breaking the moment one dependency misbehaves.

How this differs from single-modality phishing detectors

Compared with URL-only systems, this repo sees more of the attacker's playbook. Compared with screenshot-only systems, it avoids over-trusting surface design. The multimodal version is slower and messier, but it is also harder to fool in one shot.

That matters because adversaries adapt. If you only watch the URL, attackers tune the URL. If you only watch the pixels, attackers tune the pixels. Fusion raises the cost of deception by making the attacker satisfy two different constraints at once.

What a production version would need next

A production build would need sharper error handling, calibration on real traffic, cleaner packaging, and no hardcoded external placeholders. It would also need stronger observability around when the screenshot branch fails and how much that changes the final confidence.

But as a learning artifact, the repo already does something valuable. It shows that phishing detection can be framed as an argument, not a single prediction.