Language Classifer

Hypothesis 1: Certain char-n-grams are more frequent in a language than most other char-n-grams. Hypothesis 1 Validation: Created char-n-gram (only trigrams) to see if there Zipf’s is followed by char-n-grams Hypothesis 1 is validated. There are certain n-grams (tri-grams) that are more frequent than most other char-n-grams.

Approach 2 (Character n-gram) approach might still work better if more n-grams and more data are taken. Approach 2: Using Distributed Character-Level Word Representation

Skip-Gram Model: For the CBOW model, the model takes input ‘n’ before-and-after words and predicts the middle word. For the Skip-Gram model, the model takes input a word and predicts ‘n’ before-and-after words of that word. In the latest work called FastText [3, 4], the CBOW and Skip-Gram models have extended to incorporate character-level information to better understand the text. The major contribution of the work was to modified a word vector to be a sum of the vector of its character n-grams.

The results of word embedding-based classification are very impressed with the small data like tweets, chats, and short messages.

Conclusions: The stop-word approach is strong and doesn’t require any training but fails when involves short texts. The Char-n-gram approach showed some improvement in short-text datasets but the results weren’t impressive. Word embedding based on the Char-gram can identify the short -texts of a particular language successfully.

Hence, for long texts, the stop words approach might be the simplest fit except for short texts, a pre-trained model supported char-n-grams might be used for better results.