Tokenizer Phase

After that, each sentence/phrase will be segmented into separate words (tokens). So, the morphological analyzer takes place to analyze and make derivations of the current word and remove affixes from it, and therefore, find root, stem, prefix, suffix, and infix (if it is existing). The affixes is processed to find out additional parameters or indictors to support the word type (verb, noun, character, etc.), gender, tense for verb, number “singular or plural” (Al-Barahamtoshy & Al-Barhamtoshy, 2017) (Al-Barhamtoshy, et al., 2014). Details of the preprocessing phase and the details of tokenizer module are shown in Fig 2 (Al-Barhamtoshy, et al., 2007). Arabic text content as inputting will be translated into stream of signs.

Figure 2: Language Model and Part of Speech Tagging (POS)

Ontology Phase

The ontology term is used to describe conceptualization of linguistic terms for representing entities in the domain. Other researchers are using ontology as resources of knowledge to measure the semantic similarity between analyzed tokens.

We are now moving from corpus data (unstructured data that includes programming contents) to deep structure or internal representation (semi structured data). If we know the course author name, we also know that author lives in country; author write books that are published on certain dates and written in a particular language, etc. There is a whole series of information to be drawn from this story scenario- this is the goal of the ontology. In addition, phonology helps us understand missing data. If the NLP analyst has realized the author and title, what has not been recognized? It seems that the book is published in programming domain. So let’s look for history – it’s there. Moreover, it seems that the language is also involved – we can find it too.

This section presents additional formal linguistic information about word meaning (Arabic ontology). This formal representation describes the concepts of the Arabic programming terms within the course description. The formal representation includes terms concepts and their semantic relationships (see Fig. 3).

Figure 3: Word Ontology Parsing

The detailed design of the complete proposed solution is illustrated in Fig 4. We are now moving from corpus data (unstructured data that includes an introduction for Java programming contents) to deep structure or internal representation (semi structured data).

Figure 4: Arabic text 2 (ArSL) Sign  Translator

The ontology provides semantic information about the analyzed words. Thus, the system performs “word sense disambiguation” (WSD) using the analyzed part of speech (POS) for deciding best notation. Therefore, an ontology-based in the information technology domain is created. It is used to enhance the semantic accuracy in the Arabic-signs translation. The syntactic analysis is done first (Al-Barahamtoshy & Al-Barhamtoshy, 2017) (Al-Barhamtoshy, et al., 2014) using Arabic grammatical rules (Al-Barhamtoshy, et al., 2007)].

Ontology Domain

The deep structure of the analyzed Arabic text is used to generate ontology programing domain. The proposed ArSL translator is mainly intended to provide translation process of the “primary programming content” to stream of Arabic signs at KAU university using both the two methodologies (video and Avatar). Therefore, SQL database can be used to generate the output stream using RDF, OWL and XML (visual ontology design). Fig 5 illustrates relation between the analyzed text content (internal/deep features’ structure) and the visual ontology design module.

Sign Language Dictionary and the Signer

Many of bilingual dictionaries have been designed to support several sign languages over the world (Bouzid & Jmni, 2017). Each of these dictionaries tries to search and find the equivalent signs.

In this paper, we will use two methodologies of dictionary building. The first methodology is the video and written word utterance equivalent. The second is the sign written word with Avatar transcription. The ArSL translator try to find the video/avatar equivalent with the analyzed Arabic text. The only thing we need is large data. However, depending on how we plan to use our model, we need to be more or less satisfied about the quality of the dictionary and lexicon we use. When in doubt, the general rule is the more data we have, is the better.

Moreover, depending on the corpus size (text content), training can take several hours or even days, but fortunately we can store the analyzed data and extracted features on a storage disk. This way we do not have to do the analyzed tasks of model training every time we need to use it.

Figure 5: Anayzed text and Visual ontology Design

The solution takes the analyzed deep structure of the analyzed text and converts this structure into RDF using OWL and XML. Therefore, if ontology (repository) signer is ready with semantic concepts, the signer is ready to translate.

Corpora Creation

Many courses related to programming can be used to create dataset (dictionary and lexicon) of the proposed solution with the Arabic Sign Language (ArSL). The generated video terms with our corpus includes 149 samples. Table 1 and table 2 illustrate some examples that are selected from the created corpus. The following equations describes the tree domain of corpus regular expression that, includes our proposed corpora.

Table 1: Arabic Computational Terms with English Equavalent Data Examples

The equations in 7, 8, and 9 illustrate the different regular expression (in BNF grammars) to formulate the ArSL derivation grammars.

Table 2: ArSL Definitions Examples

The ArSL language is a derivative of spoken-written Arabic language, and therefore it is not a language by itself. This in addition to reordering of the signs or the position of signs. Such position or reordering of the signs’ criteria will be taken into consideration, table 3 illustrates the word evaluation form, according to position or reordering of 10 Arabic signs.

Table 3: Dataset Subset of Control Statements Evaluation Form

Testing and Evaluation

To test and evaluate the proposed “Arabic Text to Arabic Signs” translation system, a human effort is used with respect to the equivalent translated signs. There are evaluation metrics that used to measure the quality between different machine translation solutions, but they are not used in sign translation. Therefore, an evaluation sheet has been prepared, includes the 10 technological translated definitions in the proposed dataset of the ArSL. Table 4 illustrates the proposed evaluation form.

Word Error Rate (WER) is used to evaluate machine translation and automatic speech recognitions (Al-Barhamtoshy, et al., 2014). The WER criterion based on the following:

WER = ( I + D + S ) / N .………. (10)

Where N represents total words in the dataset, I represents the number of words/signs that are inserted, D is the number of words/signs that are deleted, and S is the number of substituted words/signs in the translation process.

The WER is implemented based on the Levenshtein distance for words matching between original content (Java course) and the signed content (signed by ArSL).

Table 4: Meaning Dataset Subset of ArSL Evaluation Form

Red color means that these words (characters) are not signed.

Therefore, the accuracy is computed by:

Accuracy = (1-WER) x 100 ……………………… (11)

The ArSL prototype system is implemented in Python language. The following figures (Fig 6 (a):(f)) and (Fig 7 (a): (f)) are the outputs of the two proposed models (video and Avatar).

Figure 6: Evaluated samples of output results for the Video approach

Next, we compare between the two models’ video and avatar, using the predefined corpora. The deaf human expert evaluates each corpus’ word by word using the created dataset. Table 4 describes the evaluation processes of the proposed dataset, considered word error rate (WER), sensitivity, and specificity.

Table 5: Dataset Subset of ArSL Evaluation Form

P = True positive value,    N = True negative value.

P’= False positive value,   N’= False negative value.

We analyzed the proposed corpora: Arabic text content corpus, video corpus and avatar corpus. We collected the errors, and then classified them into the two using approaches video and avatar. Table 4 illustrates the details of the evaluated work achieved by human experts. Therefore, the table demonstrates the WER, precision, and recall of the proposed translation using the two approaches.  We found no significant difference between the two approaches video and avatar for the Arabic sign translation. Also, no statistically differences between the two approaches for precision and recall. The overall evaluation analysis of the output result demonstrates the effectiveness of our two approaches using the proposed translation solution of the Arabic text content to the Arabic sign language.

The accuracy performed through two experiments. Experiment # 1 using evaluator procedure with human experts using WER to measure the errors in the generated Arabic sign results. We selected the original course text content compared to the generated (translated) signs output using ArSL. Fig. 5(a) illustrates the WER for the two approaches. Experiment #2 describes the precision and recall for the used corpora using the two approaches video and avatar.

Figure 7: Evaluated samples of output results for the avatar approach

Fig 8 displays the comparison between the two approaches for the WER and the precision and recall for the evaluation form.

Figure 8: Video and Avatar evaluation for the WER, precision and recall

Experimental Testing

In real test, the ArSL corpus includes 150 of video signs stream in the domain of Java programming.  It consists from 55 video for computational terms, 50 videos signs for reserved terms, and 45 terms for control and conditional statements. The ArSL signer test shows the number of correct and non-correct for deaf people at the committee student test (Table 6).

Table 6: Number of Correct and Non-Correct During Signer Pronounciation Test

The total average value of WER for the sign understanding is derived with the proposed approach on the perception of the conducted human expert. The experimental results show that the WER for signing of (1) computational-terms is 7.27%, (2) for Java reserved words (terms) is 4.00%, and (3) for control and conditional statement is 11.11%. The overall WER for the proposed work is 7.46 %.

Performance Testing

In order to evaluate the speed of the proposed system, performance testing is done. Table 7 illustrates the performance testing results, during playing every term.

Table 7: Performance Testing Results

Conclusion

We have presented in this paper, a proposed machine translation model to translate from Arabic text content to Arabic sign language (ArSL). The real application domain includes an introduction to Java programming in the secondary school for deaf and hearing-impaired students. The solution build two corpus/corpora for Arabic sign language, one for avatar and second for real video in scope domain of programming.

This is the first corpora in Arabic sign language that covered the programming domain. Two prepared corpora approaches are implemented, one for video model and the second for Avatar model. The two models are used in translation for deaf and hearing-impaired children. For this purpose, 100 video and 100 avatars were tested and evaluated.

An expert human assessed manually, the two corpora of the proposed work. Then, this evaluator measured improvement according to the information technology and programming domain.

Future work will include well-known bilingual dictionaries/lexicons with mapping process to RDF and OWL for content translation. In addition, BLEU will be involved as metric to evaluate the meaning in the output result of the proposed system.

Acknowledgement

The authors would like to pay special thanks to Ms. Khadija Kidwai for her valuable assistance during the research process.

References

Abuzinadah, N., Malibari, A. & Krause, P., 2017. Towards Empowering Hearing Impaired Students’ Skills in Computing and Technology. International Journal of Advanced Computer Science and Applications, 8(1), pp. 107-118.

Al-Barahamtoshy, O. & Al-Barhamtoshy, H., 2017. Arabic Text-to-Sign (ArTTS)Model from Automatic SR System. Procedia Computer Science, Volume 117, pp. 304-311.

Al-Barhamtoshy, H., Abdou, S. & Jambi, K., 2014. Pronunciation Evaluation Model for Non-Native English Speakers.. Life Science Journal, 11(9), pp. 216-226.

Al-Barhamtoshy, H. et al., 2014. Speak Correct: Phonetic Editor Approach. Life Sience, 11(9), pp. 626-640.

Al-Barhamtoshy, H., Saleh, M. & Al-Kheribi, R., 2007. Infra Structure for Machine Translation. Cairo, The Seventh Conference for Language Engineering.

Al-Barhamtoshy, H., Thabit, K. & Ba-Aziz, B., 2007. Arabic Morphology Template Grammar. Cairo, The Seventh Conference on Language Engineering, Ain Shams University..

Bouzid, Y. & Jmni, M., 2017. ICT-based Applications to Support the Learning of Written Signed Language. s.l., 6th International Conference on Information and Communication Technology and Accessibility (ICTA).

ElMaazouzi, Z., El Mohajir, B., Al Achhab, M. & Souri, A., 2016. Development of a Bilingual Corpus of Arabic and Arabic Sign Language based on a Signed Content. s.l., 4th IEEE International Colloquium on Information Science and Technology (CiSt).

Escudeiro, P. et al., 2017. Digital Assisted Communication. s.l., Proceedings of the 13th International Conference on Web Information Systems and Technologies .

Harris, R., 2018. Transforming My Teaching Through Action Research. Journal of American Sign Languages and Literatures. Deaf Eyes on Interpreting ed. Washington, DC: Gallaudet Press.

Kong, W. & Ranganath, S., 2008. Signing Exact English(SEE) Modeling and Recognition. Pattern Recognition, 41(5), pp. 1638-1652.

Konstantinidis, D., Dimitropoulos, K. & Daras, P., 2018. A Deep Learning Approache for Analyzing Video and Skeletal Features in Sign Language Recognition.. s.l., IEEE International Conference on Imaging Systems and Techniques.

Konstantinidis, D., Dimitropoulos, K. & Daras, P., 2018. Sign Language Recognition on Hand and Body Skeletal Data, s.l.: European Union.

Lim, K. M., Tan, W. C. & Tan, S. C., 2016. A Feature Covariance Matrix with Serial Particle Filter for Isolated Sign Language Recognition. Expert Systems and Applications, Volume 54, pp. 208-218.

Morrissey, S. & Way, A., 2005. An example-based approach to translating sign language. Structure, pp. 109-116.

Napier, J. et al., 2018. Using Video Technology to Engage Deaf Sign Language Users in Survey Research: An Example from the Insign Project.. The International Journal for Translation & Interpreting Research, 10(2), pp. 101-122.

Nikam, A. & Ambekar, A. G., 2016. Bilingual Sign Recognition using Image Based Hand Gesture Technique for Hearing and Speech Impaired People. s.l., Second International Conference on Computing, Communication, Control & Automation.

Patel , U. & Ambekar, A. G., 2017. Moment Based Sign Language Recognition for Indean Language. s.l., Third International Conference on Computing, Communication, Control and Automation (ICCUBEA).

Quer, J. & Steinbach, M., 2019. Handling Sign Language Data: The Impact of Modality. Front. Psychol. 10:483. doi: 10.3389/fpsyg.2019.00483 ed. Barcelona, Spain: Autonomous University of Barcelona, Spain.

Ronchetti, F. et al., 2016. LSA64: A Dataset of Argentinian Sign language. s.l., Congreso Argentino de Ciencias de la Computacion (CACIC).

Simoy, K., Osenova, A. & Popov, A., 2016. Towards semantic-based hybrid machine translation between Bulgarian and Engish. In: Proceedings of the 2016 Conference of the North American Chapter of the Association fo Computational Linguistics: Human Language Technologies. s.l.:s.n.

Ulisses, J. et al., 2018. ACE Assisted Communication for Education: Architecture to Support Blind & Deaf Communication. Canary Islands, IEEE Global Engineering Education Conference (EDUCON).