Fog factor readability & Language stats
The 'fog factor' is a writer's yardstick to measure the acceptability of written work. It measures tautology, padded syllables, sentence length, multi-syllable words and other factors as a guide to meaningful writing. Briefly, here's how to calculate the fog factor or fog index: A. Count the total number of words B. Count the total number of sentences C. Count the number of 3-syllable or more words D. Divide A by B to get average sentence length E. Divide C by A to get the percentage of big (3-syllable) words F. Add D and E G. Multiple F by 0.4 The result shows the fog factor or fog index. The ideal range is between 7 and 9. A fog factor above 1 1 indicates the writing is too heavy for most people to accept The 'fog factor' is a writer's yardstick to measure the acceptability of written work. It measures tautology, padded syllables, sentence length, multi-syllable words and other factors as a guide to meaningful writing. Briefly, here's how to calculate the fog factor or fog index: A. Count the total number of words B. Count the total number of sentences C. Count the number of 3-syllable or more words D. Divide A by B to get average sentence length E. Divide C by A to get the percentage of big (3-syllable) words F. Add D and E G. Multiple F by 0.4 The result shows the fog factor or fog index. The ideal range is between 7 and 9. A fog factor above 1 1 indicates the writing is too heavy for most people to accept
https://en.wikipedia.org/wiki/Gunning_fog_index
https://able2know.org/topic/114565-1
Spanish document: 25%-30% longer than the English source. Finnish document: about 30% shorter than the English source. Russian document: about 30% shorter than the English source (same for Bosnian, Croatian and Serbian) From German into Finnish the character count decreases by 10% and the word count by 40%. From German into Russian: about one-third more. From German into English: about one-third shorter. From Georgian into English: about 45-50% more. From English into Estonian: about 30% fewer words. French is 15-25% longer than English.
japanese is 34.72 characters per word for that page.
Hebrew has something like 30% less words than english when translated (and a much shorter word length. The longest hebrew word possible is 11 letters
The data from blogamundo.net tells us that average word lengths in English, French, Spanish and German are approximately 5.10, 5.13, 5.22 and 6.26. 5-letters is the unit of calculating typing speed (the word in WPM is calculated in 5-letter units). Adding another character for space, it seems that 80 characters-per-line will fit about 10 words per line, sounds good?
Well, data from askoxford.com shows that just the top 10 most frequently appearing lemmas (words such as the, of, and, to, that, have) account for 25% of the sources in the Oxford English Corpus. Increase that size to the top 100 most frequently appearing lemmas (words such as from, because, go, me, our, well, way) will cover 50% of the sources used by the Oxford dictionary. In other words, the low average word length is heavily influenced by short words frequently used in the English language.
The last data from w3.org and tug.org show that the ratio and average of word length between languages. English has the shortest average length, and other Latin-based languages are at least a few characters longer. These sites also show that the average word length can vary greatly.
Not related, but this shows the number of words in various Latin-based languages:
1.http://arxiv.org/ftp/cs/papers/0006/0006032.pdf
is most useful to count base words or lemmas rather than individual inflectional word-forms; for example, climbs, climbing, and climbed are counted as examples of the lemma climb. Just ten different lemmas (the, be, to, of, and, a, in, that, have, and I) account for a remarkable 25% of all the one billion words used in the Oxford English Corpus. If you were to read through the corpus, one word in four would be an example of one of these ten lemmas. Similarly, the 100 most common lemmas account for 50% of the corpus, and the 1,000 most common lemmas account for 75%. But to account for 90% of the corpus you would need a vocabulary of 7,000 lemmas, and to get to 95% the figure would be around 50,000 lemmas.
The remaining 5% of the corpus consists of a very large number of lemmas which occur rarely: words like evidentialist or microhouse, which may occur only once every several million words. Like all natural languages, English consists of a small number of very common words, a larger number of intermediate ones, and then an indefinitely long 'tail' of rare terms.
Vocabulary size (no. lemmas) % of content in OEC Example lemmas 10 25% the, of, and, to, that, have 100 50% from, because, go, me, our, well, way 1000 75% girl, win, decide, huge, difficult, series 7000 90% tackle, peak, crude, purely, dude, modest 50,000 95% saboteur, autocracy, calyx, conformist >1,000,000 99% laggardly, endobenthic, pomological
The long tail means that accounting for 99% of the Oxford English Corpus requires over a million lemmas.This would include some words which may occur only once or twice in the whole corpus: highly technical terms like chrondrogenesis or dicarboxylate, and one-off coinages like bootlickingly or unsurfworthy.
If we decide that around 90-95% of the corpus gives a reasonable idea of an average vocabulary, we are left with a figure somewhere in the range of 7,000-50,000 lemmas: say, 25,000.What does a vocabulary of this size represent? It represents the set of most significant words in English: those which occur reasonably frequently and which account for all but a small part of everything we may encounter in speech or writing. It includes all the words that we actively use in general everyday life.
It is interesting to note that most reasonably sized dictionaries contain significantly more than 25,000 lemmas.The 11th edition of the Concise Oxford English Dictionary lists more than 75,000 single-word lemmas, which means that the majority of its entries must belong to the long tail of extremely rare words.This makes good sense: such terms occur infrequently, but when they do occur they are likely to be crucial to what is being said, and the reader might well want to look them up.The idea of a quantifiable vocabulary should be seen in this light: the words we ignore for the purposes of the exercise may be very rare, but in context they may be very important.
What is the commonest word? Based on the evidence of the billion-word Oxford English Corpus, the 100 commonest English words found in writing around the world are as follows:
1 the 2 be 3 to 4 of 5 and 6 a 7 in 8 that 9 have 10 I 11 it 12 for 13 not 14 on 15 with 16 he 17 as 18 you 19 do 20 at 21 this 22 but 23 his 24 by 25 from 26 they 27 we 28 say 29 her 30 she 31 or 32 an 33 will 34 my 35 one 36 all 37 would 38 there 39 their 40 what 41 so 42 up 43 out 44 if 45 about 46 who 47 get 48 which 49 go 50 me 51 when 52 make 53 can 54 like 55 time 56 no 57 just 58 him 59 know 60 take 61 people 62 into 63 year 64 your 65 good 66 some 67 could 68 them 69 see 70 other 71 than 72 then 73 now 74 look 75 only 76 come 77 its 78 over 79 think 80 also 81 back 82 after 83 use 84 two 85 how 86 our 87 work 88 first 89 well 90 way 91 even 92 new 93 want 94 because 95 any 96 these 97 give 98 day 99 most 100 us
Many of the most frequently used words are short 'function words' whose main purpose is to join other, longer words rather than determine the meaning of a sentence. We are often more interested in the frequency of 'content words': we explore this below, showing the ranking according to the main word classes:
Nouns Verbs Adjectives 1 time 2 person 3 year 4 way 5 day 6 thing 7 man 8 world 9 life 10 hand 11 part 12 child 13 eye 14 woman 15 place 16 work 17 week 18 case 19 point 20 government 21 company 22 number 23 group 24 problem 25 fact 1 be 2 have 3 do 4 say 5 get 6 make 7 go 8 know 9 take 10 see 11 come 12 think 13 look 14 want 15 give 16 use 17 find 18 tell 19 ask 20 work 21 seem 22 feel 23 try 24 leave 25 call 1 good 2 new 3 first 4 last 5 long 6 great 7 little 8 own 9 other 10 old 11 right 12 big 13 high 14 different 15 small 16 large 17 next 18 early 19 young 20 important 21 few 22 public 23 bad 24 same 25 able
Nouns The commonest nouns are time, person, and year, followed by way and day (month is 40th). Notice that many of these words are very common because they have more than one meaning: way and part, for example, are listed in the Concise OED as having 18 and 16 different meanings respectively. They often also form part of common phrases: some of the frequency of time, for example, comes from adverbial phrases like on time, in time, last time, next time, this time, etc.
Verbs As one would expect, the commonest verbs express basic concepts. Strikingly, the 25 most frequent verbs are all one-syllable words; the first two-syllable verbs are become (26th) and include (27th). Of these 25, 20 are Old English words, and three more, get, seem, and want, entered English from Old Norse in the early medieval period. Only try and use came from Old French. It seems that English prefers terse, ancient words to describe actions or occurrences.
Adjectives Again, most of the top adjectives are one-syllable words, and 17 out of 25 derive from Old English: only different, large, and important are from Latin. In terms of the words' meanings, great is higher in the ranking than big, probably because of its informal sense 'very good'. Little is surprisingly high at 7, as compared with small at 15. Bad is unexpectedly low at 23: is this because we have such a large choice of synonyms available for expressing 'bad things'?
Language Translation Ratio Korean ?? 0.8 English views 1 Chinese ??? 1.2 Portuguese visualizações 2.6 French consultations 2.6 German -mal angesehen 2.8 Italian visualizzazioni 3
- Because of the width of the glyphs involved, each Chinese and Korean character is counted as two English characters in width.The 300% expansion from English to Italian is not at all surprising for a small string such as this. The following are average expected expansion rates for text translated from English into European languages, as published by IBM in their National Language Design Guide Volume 1, in 1994.
No. of characters in English source Average expansion Up to 10 200-300% 11 - 20 180 - 200% 21 - 30 160 - 180% 31 - 50 140 - 160% 51 - 70 130 - 140% Over 70 150%
There are some online websites that allow to score texts in such a way, e.g. http://www.readability-score.com/. However, when you need to use these scores in your own code and applications, it's preferable to have your own implementation or third party library that you can call upon to get things done.
Unfortunately, C#.net suffers from a limited open-source repertoire of libraries and code-snippets (relative to e.g. Python or JAVA). This is why I've written up a quick implementation of some readability indices in C#. Automated Readability Index Gunning-Fog Index Flesh-Kincaid Index
Automated Readability Index is probably the easiest one to compute, as it is the only one of the three measures that relies on character counts as a measure of word-complexity, rather than syllables count. This also means that it can be applied to different languages, and not just to English.
public static double CalculateAutomatedReadabilityIndex(string inputstring) { int charcount = BasicNLP.RemoveWhitespace(inputstring).Length; //space characters need to be ignored in character count. int wordcount = BasicNLP.Tokenise(inputstring).Length; int sentencecount = BasicNLP.SegmentSentences(inputstring).Length;
double indexval = 4.71 * ((double)charcount / wordcount) + 0.5 * ((double)wordcount / sentencecount) - 21.43; return indexval; }
The fourth line of code inside the function is the actual formula for the index computation. As you can imagine characercount, wordount and sentencecount are relatively straight-forward (for sentence segmentation I've simply checked for few common sentence punctuation symbols). Also to note is that in my code I've set-up a separate static class called BasicNLP that contains these utility functions, in order to keep the code organised.
The formula to compute the Gunning-Fog index (5th line in the code below) is very simple again, except that instead of character count, count of complex words (i.e. words consisting of three or more syllables) is used.
public static double CalculateGunningFogIndex(string inputstring) { int sentencecount = BasicNLP.SegmentSentences(inputstring).Length; string[] tokens = BasicNLP.Tokenise(inputstring); int complexwords = BasicNLP.CountComplexWords(tokens); int wordcount = tokens.Length;
double indexval = 0.4 * (((double)wordcount / sentencecount) + 100 * ((double)complexwords / wordcount)); return indexval; }
Finally Flesh-Kincaid uses the total count of syllables rather than the count of complex words (i.e. words that have three or more syllables), but is otherwise a rather similar formula.
public static double CalculateFleshKincaidIndex(string inputstring) { int sentencecount = BasicNLP.SegmentSentences(inputstring).Length; string[] tokens = BasicNLP.Tokenise(inputstring); int syllablescount = BasicNLP.SyllableCount(tokens); int wordcount = tokens.Length;
double indexval = 0.39 * (((double)wordcount / sentencecount) + 11.8 * ((double)syllablescount / wordcount) - 15.59); return indexval; }
Computing the syllable-count isn't too difficult either, especially as I found a function on the web that readily achieves this, and roughly does the syllable counting job.
public static int SyllableCount(string word) { word = word.ToLower().Trim(); int count = System.Text.RegularExpressions.Regex.Matches(word, "[aeiouy]+").Count; if ((word.EndsWith("e") || (word.EndsWith("es") || word.EndsWith("ed"))) && !word.EndsWith("le")) count–; return count; }
A simple console application in C#.net is available for download (MIT open source license, unless otherwise stated, .net v. 4.5), please use with care and at your own risk! Some of the measures can also vary slightly from other tools, which mostly has to do with how the syllables are counted and slight differences in the indices measures. The way I use it at the moment is that I calculate an average of all three, to get a more stable measure.
Download c#.net code http://www-staff.lboro.ac.uk/~comds2/files/Readability_Measures.zip
private static int SyllableCount(string word) { word = word.ToLower().Trim(); int count = System.Text.RegularExpressions.Regex.Matches(word, "[aeiouy]+").Count; if ((word.EndsWith("e") || (word.EndsWith("es") || word.EndsWith("ed"))) && !word.EndsWith("le")) count–; return count; }
private static int SyllableCount(string word) { word = word.ToLower().Trim(); bool lastWasVowel; var vowels = new []{'a','e','i','o','u','y'}; int count;
//a string is an IEnumerable<char>; convenient. foreach(var c in word) { if(vowels.Contains(c)) { if(!lastWasVowel) count++; lastWasVowel = true; } else lastWasVowel = false; }
if ((word.EndsWith("e") || (word.EndsWith("es") || word.EndsWith("ed"))) && !word.EndsWith("le")) count–;
return count; }
FUSSBODENSCHLEIFMASCHINENVERLEIH FUSSBODEN SCHLEIF MASCHINEN VERLEIH
Judenstraffolzugsanstanlt
Dampfschifffartkapitainsmurze
