Daniel Voigt Godoy - Deep Learning with PyTorch Step-by-Step A Beginner’s Guide-leanpub

forward() method of the encoder and decoder "layers," here is a before-after
comparison of the first "sub-layer" (self-attention) of the encoder "layer":
# Before
def forward(self, query, mask=None):
# query and mask go in
norm_query = self.norm1(query)
self.self_attn_heads.init_keys(norm_query)
# the sub-layer is the self-attention
states = self.self_attn_heads(norm_query, mask)
att = query + self.drop1(states)
# att comes out
...
# After
def forward(self, query, mask=None):
# query and mask go in
# the sub-layer is the self-attention
# norm, drop, and residual are inside the wrapper
att = self.sublayers[0](query,
sublayer=self.self_attn_heads,
is_self_attn=True,
mask=mask)
# att comes out
...
872 | Chapter 10: Transform and Roll Out