Fast medical concept normalization for biomedical literature based on stack and index optimized self-attention

Abstract

Medical concept normalization aims to construct a semantic mapping between mentions and concepts and to uniformly represent mentions that belong to the same concept. In the large-scale biological literature database, a fast concept normalization method is essential to process a large number of requests and literature. To this end, we propose a hierarchical concept normalization method, named FastMCN, with much lower computational cost and a variant of transformer encoder, named stack and index optimized self-attention (SISA), to improve the efficiency and performance. In training, FastMCN uses SISA as a word encoder to encode word representations from character sequences and uses a mention encoder which summarizes the word representations to represent a mention. In inference, FastMCN indexes and summarizes word representations to represent a query mention and output the concept of the mention which with the maximum cosine similarity. To further improve the performance, SISA was pre-trained using the continuous bag-of-words architecture with 18.6 million PubMed abstracts. All experiments were evaluated on two publicly available datasets: NCBI disease and BC5CDR disease. The results showed that SISA was three times faster than the transform encoder for encoding word representations and had better performance. Benefiting from SISA, FastMCN was efficient in both training and inference, i.e. it achieved the peak performance of most of the baseline methods within 30 s and was 3000–5600 times faster than the state-of-the-art method in inference.

Appendix A: Proof of Eq. 6

Given $E_{char} \in R^{C \times D_{emb}}$, $W_{Q} \in R^{D_{model} \times R_{emb}}$, i is an integer, and $i \in [0, C - 1]$. Let

$$\begin{aligned} a = \left( E_{char}[i,] \right) W_{Q}^{T}, \end{aligned}$$

a denotes the ith row of $E_{char}$ dot product by $W_{Q}^{T}$. Similarly, let

$$\begin{aligned} b = \left( E_{char} W_{Q}^{T}\right) [i,], \end{aligned}$$

b denotes the ith row of $E_{char} W_{Q}^{T}$. $E_{char} W_{Q}^{T}$ denotes the dot products of each row in $E_{char}$ and each column in $W_{Q}^{T}$, so b is equal to the ith row of $E_{char}$ dot product by $W_{Q}^{T}$. Thus,

$$\begin{aligned} \left( E_{char}[i,] \right) W_{Q}^{T} = \left( E_{char} W_{Q}^{T}\right) [i,]. \end{aligned}$$