Exact model names to reproduce your error rate

[guanqun-yang]

Thanks for the great set of templates for NLI model testing! I noticed you provided error rates for five models in data/checklist_master.tsv (shown below)

BERT	DistilBERT	RoBERTa-large	DeBERTa	RoBERTa-SNLI-MNLI-FEVER-ANLI

I am trying to reproduce some of the errors for a smaller subset of your dataset. However, I am not sure what exactly these models correspond to. Are they models you find-tuned from scratch or are they the checkpoints already fine-tuned and hosted on HuggingFace model hub?

If you fine-tuned you models from scratch, would it be possible to share your fine-tuning scripts so I could start in the exactly same setting (mostly hyperparameters) as you did.

Thank you in advance!

[ishan00]

Hello, Thanks for your interest in our work.

Two of the models (BERT and DistilBERT) we trained overselves on MNLI and you can check their model cards for accuracy on huggingface.co

Here are the exact model names
DistilBERT -> https://huggingface.co/ishan/distilbert-base-uncased-mnli
BERT -> https://huggingface.co/ishan/bert-base-uncased-mnli
RoBERTa -> https://huggingface.co/roberta-large-mnli
DeBERTa -> https://huggingface.co/microsoft/deberta-large-mnli
RoBERTa (SNLI, MNLI, FEVER, ANLI) -> https://huggingface.co/ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli

You can load and run models like this

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Loading DeBERTa
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-large-mnli")
model = AutoModelForSequenceClassification.from_pretrained("microsoft/deberta-large-mnli")