Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

Model Configuration
1 torch.manual_seed(42)
2 # Layers
3 enclayer = EncoderLayer(n_heads=3, d_model=6,
4 ff_units=10, dropout=0.1)
5 declayer = DecoderLayer(n_heads=3, d_model=6,
6 ff_units=10, dropout=0.1)
7 # Encoder and Decoder
8 enctransf = EncoderTransf(enclayer, n_layers=2)
9 dectransf = DecoderTransf(declayer, n_layers=2)
10 # Transformer
11 model_transf = EncoderDecoderTransf(
12 enctransf, dectransf, input_len=2, target_len=2, n_features=2
13 )
14 loss = nn.MSELoss()
15 optimizer = torch.optim.Adam(model_transf.parameters(), lr=0.01)
The original Transformer model was initialized using Glorot / Xavier uniform
distribution, so we’re sticking with it:
Weight Initialization
1 for p in model_transf.parameters():
2 if p.dim() > 1:
3 nn.init.xavier_uniform_(p)
Next, we use the StepByStep class to train the model as usual:
Model Training
1 sbs_seq_transf = StepByStep(model_transf, loss, optimizer)
2 sbs_seq_transf.set_loaders(train_loader, test_loader)
3 sbs_seq_transf.train(50)
fig = sbs_seq_transf.plot_losses()
836 | Chapter 10: Transform and Roll Out