Here is an interesting report comparing the different models.
I’ve been asked to share my thoughts on the report so I would like to see if anyone in the community has anything relevant to contribute. Please refrain from posting anything off topic.
2 Likes
While reading all this information which (I’m grateful that you shared this with all of) us here at Discussion.
I found very odd research was done only q/u:
We focused on English-to-Spanish and English-to-German translations, conducting two sub-studies: General domain translation (both EN>ES and EN>DE), and Domain-specific translation (Legal and Healthcare for the EN>DE pair).
I felt that in this report something was missing, and that we the reader needed more added information. I felt they were vague and only touched the surface, and in my personal opinion we need to see more of examples like this one. Instead of graphs that don’t say anything to me.
{target_lang}. The output should only contain the translated text.
For specific domains (Legal and Healthcare):
You are a professional medical translator. Translate the text provided by the user from {source_lang} to {target_lang}. The output should only contain the translated text.
You are a professional legal translator. Translate the text provided by the user from {source_lang} to {target_lang}. The output should only contain the translated text.
Las vistas magníficas se pueden disfrutar desde los senderos y el Rim Rock Drive, que se wind along the plateau. (Note: “”se wind”” is the correct translation of “”winds”” in this context, as it refers to the drive winding along the plateau, rather than the wind blowing.)
So in conclusion; we just see a statistical graph that we can’t tell how each one of these Machine translation greatly performed/accomplished while translating material. The above example was very vague.
It could be I was expecting something different from the intended purpose of this report so I apologize if that’s the case.
It so happens that recently I received a report from my Healthcare/Insurance Company, and the letter starts (in Spanish) with ‘‘AI has been used to provide for you this report if you have further questions please call ■■■-■■■-XXXX.’’
Most of the Spanish information was confusing I felt a ‘‘drunk person’’ wrote this (they jumped from one subject to the other leaving me with halfway information), and I just tossed it to the side. Days later I received the English version and realized that the poor Spanish reader only would never know what the information was about.
It turns out that someone had ‘‘stolen’’ my personal information from my Health Insurance and I was going to be provided with free Privacy And Identity Protection Services for 2 years (but I never understood this from the AI translated to Spanish letter).
Getting back to the subject these report did let me know why I couldn’t understand the Spanish translation a bit more
General conclusion
- Generative AI models generally match the translation quality of top Machine Translation (MT) models, based on their semantic similarity to human translations. However, the performance drops significantly in specialized fields like Legal or Healthcare.
So I got an answer
Generative AI models tend to add unnecessary explanations and reasoning to their output. (that provoked my frustration bc I read so much but never understood what was the letter about).
From the above report I don’t agree with this
- However, we recommend sticking with MT models for now due to LLM’s slow speed. Generative AI should only be considered for translating large volumes of text, cross-dialect translation (e.g., between Spanish variants), and in-context translation (e.g., UI strings).
They really need to do more indepth research as to why is so hard for people to understand Machine Translation in other Language like Spanish for example.
They themselves in the report admitted:
‘‘However, the performance drops significantly in specialized fields like Legal or Healthcare.’’
I hope they make an extensive report on added NEW and improved performance in Machine Translation and Generative AI translation.
1 Like
Thank you for sharing your thoughts.
1 Like
Your very welcome.
I wanted to ask you since as hard as I tried couldn’t determine if the graph was to show speed of translation from the different models mentioned in there or the accuracy in translation from English to Spanish/English to German or vice versa.
Las vistas magníficas se pueden disfrutar desde los senderos y el Rim Rock Drive, que se wind along the plateau. (Note: “”se wind”” is the correct translation of “”winds”” in this context, as it refers to the drive winding along the plateau, rather than the wind blowing.)
As you can see in the text translated to Spanish it’s a real bad translation, but they did added.
Evaluation results
We conducted all translations from December 1st to 21st, 2023. Some systems we tested were in limited availability, which may have impacted their performance and accuracy.
If you don’t mind me asking, and can clear this doubt for me; What information you got from this report? I felt everything was left inconclusive. What are your thoughts on that? Thanks in advance.
I think the gist of the report is that when you compare the different models of MT and Generative AI for translation:
- Gen AIs are significantly slower
- MTs lead in translation quality
- Gen AIs seem to be cheaper
Although the lower cost of Gen AIs is appealing, speed and quality of translation need to be factored in.
Thus, the recommendation is to stick with the MTs for now.
1 Like
@choitrio
Thank you so much!
MTs lead in translation quality, but are more costly to use, but it seems they just concentrated on Business. If you ever come around more reports like this please share them with us.
Have a wonderful day.
1 Like
OMG N T FX and others are using AI translation in movies/dramas, and it has become a living nightmare!!!
Viki does it, too, but they still hire people to edit it.
2 Likes
If they haven’t fully conquered the translations in English, and have so many issues that get overlooked, and they leave all that mess for us to see, I doubt any in depth research will be done regarding the Spanish language or any other language.
Youtube is using AI for speedy translation in Chinese dramas, but about 50% of these subtitles are pure garbage. I always burst out laughing bc they can really be hilariously funny. It’s really frustrating because some of these conversations halfway done leave us wondering what happened in that scene; so I end up feeling I never got closure to know what the FL or ML saying in that specific scene. 
Thank you for sharing your feedback with us.