Architecture Dependent Generalization in Antimicrobial Peptide Classifiers: A Comparative Study Using Protein Language Model Embeddings

Abstract

Antibiotic resistance is a growing global health threat, largely driven by the misuse and overuse of existing therapeutics. Antimicrobial peptides (AMPs) offer a promising alternative to traditional antibiotics, making reliable AMP identification tools a research priority. Computational classifiers built upon pretrained protein language model embeddings are common practice for this prediction task. While datasets for training AMP classifiers provide similar positive samples derived from extant AMPs, differences in procedures for preparing negative peptide samples can dramatically affect model performance, limiting these classifiers’ use as general predictors. How this vulnerability plays out across different model architectures, however, has received less attention. Our comparative study addresses this gap by training four model types on ESM-C and ProtBERT embeddings of amino acid sequences: logistic regression, feedforward neural network, convolutional neural network, and a dual-embedding fusion architecture. The models were then evaluated blindly on an independent external dataset to probe the impact of negative sample preparation on model performance. Every architecture achieved strong internal accuracy, ranging from 92% to 95%. External accuracy ranged from 67% to 76%, with an AUC between 0.82 and 0.85. The dual-embedding fusion model had a significantly higher external accuracy (75.6%) and MCC (0.525) compared to other models, while its external AUC (0.826) was statistically indistinguishable from a substantially smaller feedforward neural network. This work highlights that model selection should depend on the deployment context. In this study, results suggest dual-embedding fusion of ESM-C and ProtBERT embeddings for threshold-based classification, and simpler architectures for ranking-based screening, where added complexity yielded no significant advantage.