Improving the markup of multimodal data: fewer assessors, more layers

Hello! We - scientists at ITMO's Machine Learning laboratory and the Core ML team on VKontakte - are doing joint research. One of the important tasks of VK is the automatic classification of posts: it is necessary not only to generate thematic feeds, but also to identify unwanted content. Assessors are involved for such processing of records. At the same time, the cost of their work can be significantly reduced using such a machine learning paradigm as active learning.

It is about its application for the classification of multimodal data that will be discussed in this article. We will tell you about the general principles and methods of active learning, the peculiarities of their application to the task, as well as the insights obtained during the research.

Introduction

— machine learning, . , , , .

, (, Amazon Mechanical Turk, .) . — reCAPTCHA, , , , — Google Street View. — .

. , Voyage — , . , , . , .

Amazon DALC (Deep Active Learning from targeted Crowds). , . Monte Carlo Dropout ( ). — noisy annotation. , « , », .

Amazon . : / . , , . , : , . .

— ! , . pool-based sampling.

. 1. pool-based

. , , ( ). : , .

, — . (. — query). , . ( , ) .

, , .

, — . ( ). ≈250 . . () 50 — — :

1. , (. embedding), ;
2. .

, (. . 2).

. 2 —

ML — . , .

. , . , , , . , , early stopping. , .

. residual , highway , (. encoder). , (. fusion): , .

— , . -.

, — , . , .

. , (. 3):

. 3.

. , . , , . , ( + ) — .

, . 3, :

. 4.

, , . , ó , , .

, : ? :

1. ;
2. ;
3. .

. : maximum likelihood , - . :

$$

$$L = \frac{1}{\sigma_1 ^ 2}L_1 + \frac{1}{\sigma_2 ^ 2}L_2 + \frac{1}{\sigma_3 ^ 2}L_3 + \log{\sigma_1} + \log{\sigma_2} + \log{\sigma_3}$$

$$$L_1, L_2, L_3$ — ( -), $$$\sigma_1, \sigma_2, \sigma_3$ — , .

Pool-based sampling

— , . pool-based sampling :

1. - .
2. .
3. , , .
4. .
5. ( ).
6. 3–5 (, ).

, 3–6 — .

, , :

1. , . , : . , , , . . , 2 000.

   
   
   

2. . , . ( ). , , . , . 20 .
   
   . , . — , . 100 200.

   
   
   

, , , .

№1: batch size

baseline , ( ) (. 5).

. 5. baseline- .

random state. .

. «» , , .

, (. batch size). 512 — - (50). , batch size . . :

1. upsample, ;
2. , .

batch size: (1).

$$

$$current\_batch\_size =b + \Big \lfloor\frac{n \mod b}{\lfloor\frac{n}{b}\rfloor}\Big\rfloor [1]$$

$$$b$ — batch size, $$$n$ — .

“” (. 6).

. 6. batch size (passive ) (passive + flexible )

: c . , , batch size . .

.

Uncertainty

— uncertainty sampling. , , .

:

1. (. Least confident sampling)

, :

$$

$$x^{*}_{LC} = \underset{x}{\arg\max} \ 1 - P_{\theta }(\hat{y}|x) [2]$$

$$$\hat{y} = \underset{y}{\arg\max}\ P_{\theta}(y|x)$ — , $$$y$ — , $$$x$ — , $$$x^{*}_{LC}$ — , .

2. (. Margin sampling)

, , , :

$$

$$x^{*}_{M} = \underset{x}{\arg\min} \ P_{\theta }(\hat{y}_{1}|x) - P_{\theta }(\hat{y}_{2}|x)[3]$$

$$$\hat{y}_1$ — $$$x$, $$$\hat{y}_2$ — .

3. (. Entropy sampling)

:

$$

$$x^{*}_{H} = \underset{x}{\arg\max} -\sum \ P_{\theta }(y_{i}|x)\log{P_{\theta }(y_{i}|x)}[4]$$

$$$y_{i}$ — $$$i$- $$$x$ .

, , . , entropy sampling .

(. 7).

. 7. uncertainty sampling ( — , — , — )

, least confident entropy sampling , . margin sampling .

, , : MNIST. , , entropy sampling , . , .

. $$$O(p\log{q})$, $$$p$ — , $$$q$ — . , .

BALD

, , — BALD sampling (Bayesian Active Learning by Disagreement). .

, query-by-committee (QBC). — . uncertainty sampling. , . QBC Monte Carlo Dropout, .

, , — . dropout . dropout , ( ). , dropout- (. 8). Monte Carlo Dropout (MC Dropout) . , . ( dropout) Mutual Information (MI). MI , , — , . .

. 8. MC Dropout BALD

, QBC MC Dropout uncertainty sampling. , (. 9).

. 9. uncertainty sampling ( QBC ) ( — , — , — )

BALD. , Mutual Information :

$$

$$a_{BALD}=\mathbb{H}(y_1,...,y_n)-\mathbb{E}[\mathbb{H}(y_1,...,y_n|\omega)] [5]$$

$$

$$\mathbb{E}[\mathbb{H}(y_1,...,y_n|w)]=\frac{1}{k}\sum_{i=1}^{n}\sum_{j=1}^{k}\mathbb{H}(y_i|w_j) [6]$$

$$$n$ — , $$$k$ — .

(5) , — . , , . BALD . 10.

. 10. BALD



, , .

query-by-committee BALD , . , uncertainty sampling. , — $$$O(kp\log(q))$, $$$p$ — , $$$q$ — , $$$k$ — , .

BALD tf.keras, . PyTorch, dropout , batch normalization , .

№2: batch normalization

batch normalization. batch normalization — , . , , , , batch normalization. , . , . BALD. (. 11).

. 11. batch normalization BALD

, , .

batch normalization, . , .

Learning loss

. , . , .

, . — . , . learning loss, . , (. 12).

. 12. Learning loss



learning loss . .

. , . «» learning loss: , , . ideal learning loss (. 13).

. 13. ideal learning loss



, learning loss.

, . , , - , . :

1. (2000 ), ;
2. 10000 ( );
3. ;
4. ;
5. 100 ;
6. , , 1;
7. .

, , . , ( margin sampling).

1.

, margin sampling — , , , . c .

: ?

, , (. 14).

. 14. ideal learning loss ideal learning loss

, MNIST :

2. MNIST

ideal learning loss , (. 15).

. 15. MNIST ideal learning loss. — ideal learning loss, —



, , , , . .

learning loss , uncertainty sampling: $$$O(p\log{q})$, $$$p$ — , $$$q$ — . , , . , .

, . . , margin sampling — . 16.



. 16. ( ) , margin sampling

: ( — margin sampling), — , , . ≈25 . . 25% — .

, . , , .

, , . , :

• batch size;
• , , — , batch normalization.