TX Spell .NET for Windows Forms

Language Detection

Topic

Which language is this sentence written in? TX Spell .NET answers that question for text that doesn't say: a document converted from plain text, pasted from a web page, or saved by an application that stores no language. Language detection splits such a text into language scopes, sections that each have one language, so that every part is checked, hyphenated and given suggestions in the right language. It works in TX Text Control without code and on any string in code.

How Language Detection Works

The spell checker recognizes more than 30 languages and marks where one language ends and the next begins. A few properties of the method matter in practice:

  • A sentence is enough. One sentence of four or more words is usually enough to detect its language. Shorter pieces may stay without a language.
  • No dictionaries are needed. Detection uses its own data, the file languageRecognition.resources, not the dictionaries of the detected languages.
  • Scopes can be nested. A German phrase in brackets inside an English sentence becomes a scope of its own, and bidirectional text that mixes, for example, Arabic and English is split as well.
  • Only the languages you allow are detected. The spell checker looks for the languages in its list of detectable languages; others are counted as one of those or left without a language.

Documents from TX Text Control and from Microsoft Word usually carry the language of their text already. Detection is for the rest.

Detect Languages in TX Text Control

A TextControl with a connected spell checker can detect languages by itself. Set its IsLanguageDetectionEnabled property to true, in the Properties window or in code:

// Detect a language for all text without one
textControl1.IsLanguageDetectionEnabled = true;

Every piece of text with a neutral language then gets the language that was detected for it. The detection needs a spell checker connected through the SpellChecker property, as Create Your First Windows Forms Application shows in its last steps.

The status bar of TX Text Control shows the language of the text at the input position, so the user sees which language a word was given:

TX Text Control Words with the Proofing tab open and the sentence "This is an US English language region. This is a Spanisch region.": the status bar shows English (United States) in the first sentence and Spanish (United States) in the second.

Its ShowLanguage property turns this field on and off. The Language group of the Proofing tab holds the commands to set and detect languages.

Detect Language Scopes in Code

DetectLanguageScopes detects the language scopes of a string and fills LanguageScopes with them. The TXSpellChecker on a form and the TXSpell class for code without a user interface have the same members. This console application uses TXSpell from the package TXTextControl.TXSpell.Core.SDK, as in Checking Text in Code:

using System;
using TXTextControl.Proofing;

TXSpell spell = new TXSpell();

// Initialize the new instance before using it
spell.Create();

string text = "This is a test. Dieses ist ein Test. Esto es una prueba.";

spell.DetectLanguageScopes(text);

foreach (LanguageScope scope in spell.LanguageScopes)
{
    Console.WriteLine($"{scope.Language?.Name} Start {scope.Start} Length {scope.Length}: \"{text.Substring(scope.Start, scope.Length)}\"");
}

The console shows three scopes:

en Start 0 Length 16: "This is a test. "
de Start 16 Length 21: "Dieses ist ein Test. "
es Start 37 Length 19: "Esto es una prueba."

The code works as follows:

  • Each LanguageScope has a Language, a neutral CultureInfo such as en or de, and a Start and Length in the text, counting from 0.
  • Language is null for a scope whose language couldn't be detected, for example a text that is too short, so the code reads it with ?..
  • The space after a sentence belongs to the scope of that sentence.

With a TXSpellChecker from the toolbox, write txSpellChecker1.DetectLanguageScopes(text) and read txSpellChecker1.LanguageScopes instead.

The LanguageScopes property holds a LanguageScopeCollection. Its GetItem method returns the scope at a text position, or, with a language as the second argument, the scope of that language at or after the position, and the first scope of that language when none follows.

The sample that came with language detection colors each scope of a text in a different color, here English, French, German and Spanish:

A text with English sentences in red, a French sentence in green, a German sentence in blue and a Spanish sentence in orange, each colored by its detected language scope.

Choose the Detectable Languages

DetectableLanguageScopes is the list of languages the spell checker looks for. By default, it holds seven of the most widely spoken languages: Arabic (ar), German (de), English (en), Spanish (es), French (fr), Portuguese (pt) and Russian (ru). In the designer, the list appears in the Properties window of txSpellChecker1:

The Properties window of txSpellChecker1 with the DetectableLanguageScopes array expanded: ar, de, en, es, fr, pt and ru.

A language that isn't in the list isn't detected. Italian text, for example, is detected as Spanish or not at all with the default list. Set the list to the languages your documents contain:

using System.Globalization;

txSpellChecker1.DetectableLanguageScopes = new CultureInfo[]
{
    new CultureInfo("en"),
    new CultureInfo("de"),
    new CultureInfo("it"),
    new CultureInfo("nl")
};

The list takes neutral cultures only: new CultureInfo("de") works, while a specific culture such as new CultureInfo("de-DE") throws an ArgumentException. 34 languages can be detected: the column "Detection" of Supported Languages gives the code of each, and Galician (gl), Lithuanian (lt) and Ukrainian (uk) can be detected as well. Occitan is the one language written with its region, oc-FR.

Use the Detected Languages

Detection tells you which languages a text contains, so it is a way to decide which dictionaries to load: add the dictionary of each detected language, and the text is checked in all of them. Loading and Managing Dictionaries shows how to add a dictionary, and Multilingual Documents and Language Scopes how the spell checker picks the dictionary for each piece of text.

Where the Language Recognition Data Is

Detection reads languageRecognition.resources from the folder in LanguageRecognitionDirectoryPath, by default the folder LanguageRecognition that the packages copy next to the application; Installing Dictionaries and Language Data describes the folder. A folder set in the property must exist; otherwise the property throws an ArgumentException. Without the file, DetectLanguageScopes fails with an exception, and Dictionaries Not Found lists the messages.

Next Steps