Designing a Machine Translation System for Canadian Weather ...

More documents

Recommendations

Info

minima in the parameter space for the same tuning corpus. Although the tuning scores converge for a given configuration, the test scores are different between MERT tuning results. This phenomenon is illustrated in Figure 7, showing 5 different tuning runs, for the configuration French-to-English, integral, msd. While tuning does converge nicely to reach an average of BLEU = 74.7%, the test results (Figure 7’s legend) vary more significantly, with a mean BLEU score of 76.7% (standard deviation of 0.4%). Therefore, the standard deviation (in the form of an error on the measure) factors in when comparing configurations. BLEU score (%) Figure 7 – MERT tuning results, for 5 different runs, for the configuration French-to-English, integral, msd. The progression of the tune score is shown with each iteration. While the tuning converges to an average of BLEU = 74.7%, the BLEU test results shown in parentheses vary more significantly, average BLEU = 76.7%, stdev = 0.4%. The number of MERT iterations needed varies between 6 and 17. Designing a Weather Warning Translation System 18 75.0 74.5 74.6 74.7 74.7 74.6 74.8 74.0 run1 (Test: 77.1) 73.5 run2 (Test: 76.3) run3 (Test: 77.1) 73.0 run4 (Test: 77.0) 72.5 run5 (Test: 77.3) 72.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 MERT tuning iteration Fig. 5. We MERT decided to tuning take this results, instability for into account five different by running runs, each MERT for optimization the configuration 5 times in French– to-English, order integral, to obtain msd. a mean The and progression standard deviation of on the the tune end BLEU score test isresults. shown The with comparison each iteration. While thebetween tuningconfigurations convergescan tothen an average be made on ofthe BLEU basis of = the 74.7%, BLEU score thebut BLEU also according test results to shown in parentheses vary more RALI Designing significantly, A Machine Translation average System BLEU For Canadian = 76.9%, Weather stdev Warnings = 0.4%. 17 The number of MERT iterations needed varies between 6 and 17 here. MERT tuning relies on a pseudo-random component and often finds different local minima in the parameter space for the same tuning corpus. Although the tuning scores converge for a given configuration, the test scores are different between MERT tuning results. This phenomenon is illustrated in Fig. 5, showing five different tuning runs, for the configuration French-to-English, integral, msd. While tuning does converge nicely to reach an average of BLEU = 74.7%, the test results (Fig. 5’s legend) vary more significantly, with a mean BLEU score of 76.9% (standard deviation of 0.4%). Therefore, the standard deviation (in the form of an error on the measure) must be factored in when comparing configurations. To compensate for optimizer instability when comparing experimental results, we took to heart some of the suggestions made by (Clark et al. 2011) and ran each MERT optimization five times in order to obtain a mean and standard deviation on the final BLEU test results. The comparison between configurations can then be made on the basis of the BLEU score but also according to potential error represented by the standard deviation. The test results measured with BLEU are shown in Fig. 6 when English is the source language and in Fig. 7 for French. For the English-to-French translation direction (Fig. 6), the BLEU score hovers around 78.5 when not using the maxdelta4 filtering, and 75 when using it. Clearly, the latter is too heavy-handed a technique and should be discarded. That being said, a BLEU score of 78.5 is very high for an MT task. Our results are comparable to those reported by (Langlais et al. 2005), for a similar task, where they obtained a BLEU score of 77% when translating severe storm warnings from English to French. On a more diverse warning corpus, however, this figure dropped to 58% in their study. When French is the source language (Fig. 7), the BLEU score is slightly lower, which is a bit surprising given the fact that the target language, i.e. English, is morphologically poorer. The BLEU score now peaks at 77.2% when discarding the clearly unusable maxdelta4 configurations. This time, even when taking the error
BLEU Score BLEU (%) Score (%) BLEU Score BLEU (%) Score (%) potential error represented by the standard deviation. The test results measured with BLEU are shown in Figure 8 when English is the source language and in Figure 9 for French. potential error represented by the standard deviation. The test results measured with BLEU are For shown the in English-to-French Figure 8 when English translation is the source direction language (Figure and 8), in the Figure BLEU 9 for score French. hovers around 78.5 when not using the maxdelta4 filtering, and 75 when using it. Clearly, the latter is too heavyhanded For the a English-to-French technique and should translation be discarded. direction That (Figure being 8), said, the a BLEU BLEU score score hovers of 78.5 around is very high 78.5 for when an not MT using task. the Our maxdelta4 results are filtering, comparable and to 75 those when reported using it. Clearly, by (Langlais the latter et al., is 2005), too heavyhanded for a similar task, a technique where they and obtained should be a discarded. BLEU score That of 77% being when said, translating a BLEU score severe of 78.5 storm is very warnings high from for an English MT task. to French. Our results On a are more comparable diverse warning to those corpus, reported however, by (Langlais this figure et dropped al., 2005), to for 58% a in similar their task, study. where they obtained a BLEU score of 77% when translating severe storm warnings from English to French. On a more diverse warning corpus, however, this figure dropped to 58% in their 80study. 79 80 78 79 77 78 76 77 75 76 74 75 73 74 73 Designing a Weather Warning Translation System 19 78.4 78.5 78.5 78.2 78.4 78.4 Figure 8 – Average BLEU integral results (n = 5) on the test maxdelta5 corpus for 9 different configurations maxdelta4of the Moses Fig. 6. pipeline, Average when translating BLEU results from English. (n The = 5) error onbars the correspond test corpus to the standard for 9 different deviation. configurations of the Figure Moses 8 – pipeline, Average BLEU when results translating (n = 5) on the from test corpus English. for 9 different The error configurations bars correspond of the Moses to the pipeline, when translating from English. The error bars correspond to the standard deviation. standard deviation. 80 79 80 78 79 77 78 76 77 75 76 74 75 73 74 73 78.4 78.5 78.5 78.2 78.4 78.4 Figure 9 – Average BLEU integral results (n = 5) on the test maxdelta5 corpus for 9 different configurations maxdelta4of the Moses pipeline, when translating from French. The error bars correspond to the standard deviation. Figure 9 – Average BLEU results (n = 5) on the test corpus for 9 different configurations of the Moses Fig. 7. pipeline, Average when translating BLEU results from French. (n The = 5) error onbars the correspond test corpus to the standard for 9 different deviation. configurations of the Moses pipeline, when translating from French. The error bars correspond to the RALI Designing A Machine Translation System For Canadian Weather Warnings 18 standard deviation. RALI Designing A Machine Translation System For Canadian Weather Warnings 18 74.8 75.3 74.9 dist msd none dist msd none dist msd none integral maxdelta5 maxdelta4 dist msd none dist msd none dist msd none 76.6 76.6 76.9 76.9 76.7 76.8 76.7 76.8 77.2 77.2 76.5 75.2 75.7 75.2 dist msd none dist msd none dist msd none integral maxdelta5 maxdelta4 dist msd none dist msd none dist msd none into account, all configurations are not equal. The msd-integral and msd-maxdelta5 settings improve the BLEU score by an average of 0.2%. As a result, when translating from French, we will favor the msd-maxdelta5 configuration. This means that we will indeed be using a lexicalized reordering model. We prefer using the maxdelta5 configuration, because the phrase model is slightly smaller as a result of the filtering phase. Moreover, it has the added merit of removing some dubious phrase associations that crept into our preliminary translations. When translating from English, since all configurations excluding maxdelta4 perform equally well, we opt for the same configuration as the reverse translation direction, i.e. the msd-maxdelta5. Having the same configuration for both translation directions simplifies Watt’s implementation without harming its performance. Table 5 shows the complete tuning and test results for this configuration. Out of the five configurations the tuning process created using our methodology, we picked the one with the best BLEU score to produce the test results shown in Table 5. The word error rate (WER) and sentence error rate (SER) are also reported. A good 76.5 74.8 75.2 75.3 75.7 74.9 75.2
Page 1 and 2: Natural Language Engineering 1 (1):
Page 3 and 4: Designing a Weather Warning Transla
Page 5 and 6: MTCN01 CWWG 221130 HEADER=WW AREA=7
Page 17: Designing a Weather Warning Transla

Designing a Machine Translation System for Canadian Weather ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?