Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

"Do I need to send each tensor to a device, really?"
Not really, no. It turns out, the output of the tokenizer isn’t just a dictionary, but
also an instance of BatchEncoding, and we can easily call its to() method to send
the tensors to the same device as the model:
print(type(tokens))
tokens.to(loaded_model.device)
Output
<class 'transformers.tokenization_utils_base.BatchEncoding'>
{'input_ids': tensor([[ 101, 2091, 1996, 3756, 5318, 10442,
4920, 102]], device='cuda:0'),
'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1]], device
='cuda:0')}
That was easy, right?
Let’s call the model using these inputs!
Even though the model is loaded in evaluation mode by default, it is always a good
idea to explicitly set the model to evaluation mode using the PyTorch model’s
eval() method during evaluation or test phases:
loaded_model.eval()
logits = loaded_model(input_ids=tokens['input_ids'],
attention_mask=tokens['attention_mask'])
logits
Output
SequenceClassifierOutput(loss=None, logits=tensor([[ 2.7745, -
2.5539]], device='cuda:0', grad_fn=<AddmmBackward>), hidden_states
=None, attentions=None)
The largest logit corresponds to the predicted class as usual:
1000 | Chapter 11: Down the Yellow Brick Rabbit Hole