23 Entity-Event Ontology Construction by Conceptualization …
351
As it is always hard to evaluate ontology learning and relating tasks, we used a
slightly different evaluation approach. Given that system supports structured queries
and can be used for exploratory search, we formulated 15 queries with interpretations
like “Which countries suffered from earthquakes and floods in 2015–2019?”, “What
are Vladimir Putin’s international visits in summer 2019?”, “What exploit kits are
made in 2018?”, “What world leaders did Donald Trump meet in years 2018–2019?”.
We used human assessors to obtain true answers for each query.
Comparing true answers with the system’s ones, we found that precision is 0.91
and recall is 0.68. It should be noted that precision is substantially high, while recall
is unremarkable. One of the reasons is that when some event is described using a set
of rules, they are usually relevant and rigorous, but they are always incomplete: it is
difficult to capture all ways of saying something. This problem can probably be solved
by taking advantage of new methods like neural machine translation mentioned in
[5]. Although it is unlikely that they are capable of achieving such high precision
nowadays, they can possibly be combined with the existing ones to improve recall
without significant loss of precision.
The first release of a system was tailored to the needs of civil aviation and was
used in the following tasks:
• Safety occurrence reports’ analysis in Aviation Safety Network.
• Structuring and navigating through technical specifications and local regulations.
• Social media interaction and brand monitoring for airlines.
• Monitoring and analysis of international civil aviation news and new regulations.
• Early risk identification (natural disasters, political changes, etc.).
• Exploratory enterprise search among contracts, documents, supporting documentation, work regulations, instructions, reports, etc.
Another result is that extracted information can be used as informative features in
machine learning tasks for the target domain. For example, we used system to predict
protests in Moscow, Russia during 2011–2020 based on social network data with
an overall accuracy of 72.4%. Also, it was successfully used to enhance document
classification and clustering algorithms by providing high-informative extra features.
Moreover, it can be viewed as an automatic text corpus processing method that
allows using of classic statistical and data analysis methods by extracting domainspecific information from text. As extracted knowledge is highly structured and easily
operated, it can be used by such methods without any further reference to the source
texts.
References
1. Maedche, A., Staab, S.: Ontology learning for the semantic web. IEEE Intell. Syst. 16(2), 72–79
(2001)
2. Buitelaar, P., Cimiano, P., Magnini, B.: Ontology learning from text: methods, evaluation and
applications. Frontiers Artif. Intell. Appl. 123 (2005)
351
As it is always hard to evaluate ontology learning and relating tasks, we used a
slightly different evaluation approach. Given that system supports structured queries
and can be used for exploratory search, we formulated 15 queries with interpretations
like “Which countries suffered from earthquakes and floods in 2015–2019?”, “What
are Vladimir Putin’s international visits in summer 2019?”, “What exploit kits are
made in 2018?”, “What world leaders did Donald Trump meet in years 2018–2019?”.
We used human assessors to obtain true answers for each query.
Comparing true answers with the system’s ones, we found that precision is 0.91
and recall is 0.68. It should be noted that precision is substantially high, while recall
is unremarkable. One of the reasons is that when some event is described using a set
of rules, they are usually relevant and rigorous, but they are always incomplete: it is
difficult to capture all ways of saying something. This problem can probably be solved
by taking advantage of new methods like neural machine translation mentioned in
[5]. Although it is unlikely that they are capable of achieving such high precision
nowadays, they can possibly be combined with the existing ones to improve recall
without significant loss of precision.
The first release of a system was tailored to the needs of civil aviation and was
used in the following tasks:
• Safety occurrence reports’ analysis in Aviation Safety Network.
• Structuring and navigating through technical specifications and local regulations.
• Social media interaction and brand monitoring for airlines.
• Monitoring and analysis of international civil aviation news and new regulations.
• Early risk identification (natural disasters, political changes, etc.).
• Exploratory enterprise search among contracts, documents, supporting documentation, work regulations, instructions, reports, etc.
Another result is that extracted information can be used as informative features in
machine learning tasks for the target domain. For example, we used system to predict
protests in Moscow, Russia during 2011–2020 based on social network data with
an overall accuracy of 72.4%. Also, it was successfully used to enhance document
classification and clustering algorithms by providing high-informative extra features.
Moreover, it can be viewed as an automatic text corpus processing method that
allows using of classic statistical and data analysis methods by extracting domainspecific information from text. As extracted knowledge is highly structured and easily
operated, it can be used by such methods without any further reference to the source
texts.
References
1. Maedche, A., Staab, S.: Ontology learning for the semantic web. IEEE Intell. Syst. 16(2), 72–79
(2001)
2. Buitelaar, P., Cimiano, P., Magnini, B.: Ontology learning from text: methods, evaluation and
applications. Frontiers Artif. Intell. Appl. 123 (2005)