Character count of language X in mixed text file?

Question

I have mixed-language text files, and would like to count the simple total number of printable characters of one of the languages. It helps that the languages inhabit different unicode ranges.

My specific use-case involves Hebrew, Polytonic Greek, and English -- but I imagine a solution to this problem could be generalized for other contexts, too.

I would like to count to the Hebrew characters only -- that's Unicode [\u0590-\u05ff]. Here's a brief sample input file (which, by my manual count, contains 62 Hebrew characters):

[ Ps117 ]‬ h1: ‫  הללו את יהוה כל גוים שבחוהו כל האמים ‬ r1: Praise the LORD, all nations! Extol him, all peoples! g1: Αλληλουια. Αἰνεῖτε τὸν κύριον, πάντα τὰ ἔθνη, ἐπαινέσατε αὐτόν, πάντες οἱ λαοί, b1: Alleluia. Praise the Lord all you nations: praise him all you peoples. h2: ‫  כי גבר עלינו חסדו ואמת יהוה לעולם הללו יה ‬ r2: For great is his steadfast love toward us; and the faithfulness of the LORD endures for ever. Praise the LORD! g2: ὅτι ἐκραταιώθη τὸ ἔλεος αὐτοῦ ἐφ' ἡμᾶς, καὶ ἡ ἀλήθεια τοῦ κυρίου μένει εἰς τὸν αἰῶνα. b2: For his mercy has been abundant toward us: and the truth of the Lord endures for ever. 

I'm on Ubuntu 16.04.2 LTS, if that helps. I imagine perl would be a likely option here, or some shell script ... but I don't know these things, which is why I'm asking!

---

_{For the curious, the lines in my input are: h= Hebrew; r= Revised Standard Version; g = Greek Septuagint; b = Brenton translation of Septuagint; in each case followed by a verse number.}

Answer 1

There is potentially an issue with determining the length of Unicode strings. See this page from Twitter's developer docs for more details on Normalization

The character count will depend on the locale you have configured. You can run locale to verify that you have a UTF-8 locale configured. Once this is done, the code from @stephen-rauch should work.

Depending on which regex library you use, you might also be able to use named scripts like \p{Hebrew} and \P{Greek} Here is an example of using \P{Hebrew} to remove all non-Hebrew characters: Link

Edited: Initial results were due to mis-configured locale

Answer 2

These seem to come close for me (tested on Ubuntu 16.04)

$ perl -0777 -MEncode -ne 'print decode("UTF-8",$_) =~ tr/\x{0590}-\x{05ff}//,"\n"' input 62 $ perl -0777 -MEncode -ne 'print decode("UTF-8",$_) =~ tr/\p{Hebrew}//,"\n"' input 63 

I'm not sure what the "right" answer should be.

Answer 3

Using python you can do something like this:

Code:

# coding: utf-8 import re import codecs #find_hebrew = re.compile(ur'[\u0590-\u05ff]+') # python 2 find_hebrew = re.compile(r'[\u0590-\u05ff]+') # python 3 count = 0 with codecs.open('text_file', 'rU', encoding='utf-8') as f: for line in f.readlines(): for n in find_hebrew.findall(line): count += len(n) print(count) 

Result:

62