Dataa science

Kannada is a phonetic language. In Kannada language, the morphological forms of terms (especially of nouns and verbs) are formed by adding different morphological suffixes to their pure forms. Hence, when queried for morphological forms, search engines based on exact matching fail to identify other semantically similar and morphologically different terms, and thus reduce the quality of the search results. We observe that even though the morphological forms of a term look different, they can be grouped together based on their common prefixes. In this work we propose fuzzy matching based indexing and retrieval algorithms. We propose an indexing mechanism inspired from prefix trees. We also derive our inspirations from the fact that the Unicode encodes the Kannada terms very similar to the way terms are generated using Kannada grammar. We also discuss a query term truncation and decayed score based retrieval algorithm for better retrieval of the documents for the given query. The indexing and retrieval systems still are based on the tf-idf based indexing and retrieval. However, the novelty of the work lies in the way the algorithms bring together the similar terms. This solution can be scaled to work for other South Indian languages with no or little modification as their Unicode encoding and morphological behaviors are similar to Kannada. This work is done as a part of Kanaja project, an encyclopedic knowledge portal in Kannada conceptualised by Karnataka Jnana Ayoga. The authors would like to thank Aditya Ramana Rachakonda for his inputs.

Kannada is a phonetic language. In Kannada language, the morphological forms of terms (especially of nouns and verbs) are formed by adding different morphological suffixes to their pure forms. Hence, when queried for morphological forms, search engines based on exact matching fail to identify other semantically similar and morphologically different terms, and thus reduce the quality of the search results. We observe that even though the morphological forms of a term look different, they can be grouped together based on their common prefixes. In this work we propose fuzzy matching based indexing and retrieval algorithms. We propose an indexing mechanism inspired from prefix trees. We also derive our inspirations from the fact that the Unicode encodes the Kannada terms very similar to the way terms are generated using Kannada grammar. We also discuss a query term truncation and decayed score based retrieval algorithm for better retrieval of the documents for the given query. The indexing and retrieval systems still are based on the tf-idf based indexing and retrieval. However, the novelty of the work lies in the way the algorithms bring together the similar terms. This solution can be scaled to work for other South Indian languages with no or little modification as their Unicode encoding and morphological behaviors are similar to Kannada. This work is done as a part of Kanaja project, an encyclopedic knowledge portal in Kannada conceptualised by Karnataka Jnana Ayoga. The authors would like to thank Aditya Ramana Rachakonda for his inputs.

Expanding a topic into its constituent terms used by a population is an important problem in social space analytics. A topic is defined as a concept representing the semantic "aboutness" of a set of other concepts. What a topic is really "about" may differ across different populations, and discovering this provides sound insights about the population itself. In this paper, we propose an approach for topic expansion, inspired by models from cognitive science-specifically, episodic and semantic memory. We propose an episodic hypothesis that asserts a relationship between a topic and other concepts that are collectively about the topic. The canonical form of this algorithm, while shown to produce good results, does not run in interactive response time. We then propose several simplifications over the canonical form to provide interactive response times without reducing too much on the quality of the results. The proposed simplifications also help in creating topical clusters on repositories where there is not enough representation of different terms for the topic.

Mining latent semantics from text corpora is an important problem with several applications. Models to explain how semantic associations get embedded into natural language, are of immense interest. In this work, we take a cognitive modeling approach to this problem. The text corpus is viewed as a "cognitive space," and each document is treated as a cognitive "episode" whose objective is to convey some meaning within some context. Communication of semantics is proposed to take place in a 3-layer model. The topmost layer is called the analytic layer that maintains the semantic worldview of the communicator. Communication happens in the context of an episode, where semantic associations from the analytic layer are combined with the episodic objectives to create conceptual sentences. This is performed at the second layer called the episodic layer. The last layer, called the linguistic layer provides language specific vocabularies and inflections that convert conceptual sequences from the episodic layer into communication units like speech utterances, gestures or written text. Mining latent semantics becomes the problem of inferring semantic layer associations by observing linguistic layer recordings like text. This is addressed by proposing one or more episodic hypotheses on patterns of co-occurrences of terms in the linguistic layer. An episodic hypothesis builds on observational patterns and generalizes observations to semantic associations. A suite of semantic associations are described in the paper and are reliably mined from text using their corresponding episodic hypotheses.

We report a tool called Akshaya, which implements a framework to mine four types of "general knowledge semantics" (analytical semantics) from unstructured text. The semantics being mined are - semantic siblings, topical anchors, topic expansion and topical markers. The framework provides options to embed more such general knowledge semantic mining algorithms into it. We use a term co-occurrence graph representation of unstructured text corpora to mine these semantics relations between terms. The semantic mining algorithms use different graph algorithms like random walk, graph clustering and so on to mine semantic relations. The tool can currently read plain text documents and generate a term co-occurrence graph and perform semantic association mining on it.

This paper addresses the problem of "deep matching"-or matching different classes of entities based on latent underlying semantics, rather than just their visible attributes. An example of this is the "automatic task assignment" problem where several tasks have to be assigned to people with varied skill-sets and experiences. Datasets showing types of entities (tasks and people) along with their involvement of other concepts, are used as the basis for deep matching. This paper describes a work in progress, of a deep matching application called SortingHat. We analyze issue tracking data of a large corporation containing task descriptions and assignments to people that were computed manually. We identify several entities and concepts from the dataset and build a co-occurrence graph as the basic data structure for computing deep matches. We then propose a set of query primitives that can establish several forms of semantic matching across different classes of entities.

We present our tool UCliDSS, an Unsupervised Clinical Decision Support System for textual data. UCliDSS is aimed at building a retrieval engine to retrieve documents from a bio-medical data-set to provide decision support for medical professionals. In this work, we use a term co-occurrence graph (TCG) based approach augmented with Solr to build the document retrieval engine (UCliDSS). The TCG is built from the documents of a given bio-medical data-set and is used for query expansion using a variant of random walk algorithm. The expanded query is given to Solr as input to retrieve relevant documents.

We report a tool called Akshaya, which implements a framework to mine four types of "general knowledge semantics" (analytical semantics) from unstructured text. The semantics being mined are - semantic siblings, topical anchors, topic expansion and topical markers. The framework provides options to embed more such general knowledge semantic mining algorithms into it. We use a term co-occurrence graph representation of unstructured text corpora to mine these semantics relations between terms. The semantic mining algorithms use different graph algorithms like random walk, graph clustering and so on to mine semantic relations. The tool can currently read plain text documents and generate a term co-occurrence graph and perform semantic association mining on it.

We report a framework called SortingHat to perform semantic matching between labeled concepts in a partially labeled corpora such as workflow data. We create a labeled term co-occurrence graph as the representative data-structure of the given corpus. The semantic matching between concepts is performed using a variant of random walk algorithm on the term co-occurrence graph. The SortingHat system takes a set of concepts as input and generates a semantically matching set of concepts for them. In this experiment, we use data from bug tracking system of a large enterprise to demonstrate the results.

We present our tool UCliDSS, an Unsupervised Clinical Decision Support System for textual data. UCliDSS is aimed at building a retrieval engine to retrieve documents from a bio-medical data-set to provide decision support for medical professionals. In this work, we use a term co-occurrence graph (TCG) based approach augmented with Solr to build the document retrieval engine (UCliDSS). The T CG is built from the documents of a given bio-medical data-set and is used for query expansion using a variant of random walk algorithm. The expanded query is given to Solr as input to retrieve relevant documents.