Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

5. Encoder "Layer"
The encoder "layer" implements a list of two "sub-layers" (7), which are going to be
called with their corresponding operations:
Encoder "Layer"
1 class EncoderLayer(nn.Module):
2 def __init__(self, n_heads, d_model, ff_units, dropout=0.1):
3 super().__init__()
4 self.n_heads = n_heads
5 self.d_model = d_model
6 self.ff_units = ff_units
7 self.self_attn_heads = \
8 MultiHeadedAttention(n_heads, d_model, dropout)
9 self.ffn = nn.Sequential(
10 nn.Linear(d_model, ff_units),
11 nn.ReLU(),
12 nn.Dropout(dropout),
13 nn.Linear(ff_units, d_model),
14 )
15 self.sublayers = nn.ModuleList(
16 [SubLayerWrapper(d_model, dropout) for _ in range(2)]
17 )
18
19 def forward(self, query, mask=None):
20 # SubLayer 0 - Self-Attention
21 att = self.sublayers[0](query, 1
22 sublayer=self.self_attn_heads,
23 is_self_attn=True,
24 mask=mask)
25 # SubLayer 1 - FFN
26 out = self.sublayers[1](att, sublayer=self.ffn) 1
27 return out
1 Calls "Sub-Layer" wrapper (7) twice (self-attention and feed-forward network)
"Wait, I don’t remember this SubLayerWrapper module…"
Good catch! It is indeed brand new! We’re defining it shortly (it’s number seven,
hang in there!).
Putting It All Together | 869