TabNet. Few details

Deep high performance neural network for tabular data TabNet   

Introduction 

Deep neural networks (GNNs) have become one of the most attractive tools for creating artificial intelligence systems (SRI), for example, speech recognition, natural communication, computer vision [2-3], etc. In particular, due to the automatic selection of GNS important, defining features, connections from data. Neural network architectures (neocognitronic, convolutional, deep trust, etc.), models and algorithms for learning GNS (autoencoders, Boltzmann machines, controlled recurrent, etc.) are developing. GNSs are difficult to train, mainly due to vanishing gradient problems.  

The article discusses the new canonical architecture of GNS for tabular data (TabNet), designed to display a "decision tree". The goal is to inherit the advantages of hierarchical methods (interpretability, sparse feature selection) and GNS-based methods (step-by-step and end-to-end learning). Specifically, TabNet addresses two key needs - high performance and interpretability. High performance is often not enough - GNS must interpret, replace tree-like methods. 

TabNet is a neural network of fully connected layers with a sequential attention mechanism, which: 

• uses a sparse selection of objects by instances, obtained from a training dataset; 

  
  
  
  
  

• creates a sequential multistage architecture in which each decision-making step can contribute to the part of the decision that is based on the selected functions;

  
  
  
  
  

• improves learning ability through non-linear transformations of selected functions;

  
  
  
  
  

• simulates an ensemble, involving more accurate measurements and more improvement steps.  

  
  
  
  
  

Each layer of a given architecture (Fig. 1) is a solution step that contains a block with fully connected layers for transforming characteristics - a Feature Transformer and an attention mechanism for determining the importance of the input original characteristics. 

1. Converter of functions 

1.1. Batch normalization 

    -   . . , (,   ), , . (covariate shift).  

. , — . ( ) , . , , , .  

. , — , . ,   , (   ,   – )   . .    - (batch normalization), 2015  [4]. 

 - .  

1. d: x = (x₁, . . . , x_d). k-  x ( ): 

2. . , . ,  , (   

[−1, 1] ).  

, :  

γ, β .  

3. , ,  -,  

4. . 

 -:  

• , , ;  

  
  
  
  
  

• , ;  

  
  
  
  
  

• ,      ;  

  
  
  
  
  

•    . 

  
  
  
  
  

1.2. GLU

 [5]   Gated Linear Unit,       , , LSTM-.  

GLU  

, , , .     H = [h₀ ,..., h_N]   w₀, ... ,w_N,  P (w_i |h_i).   f  H   h_i = f(h_{i - 1} , w_{i - 1})  , i ( ,  ). 

 f   H = f * w , , , , , . . , ,  [5] , , . 

. 2 . ,  D ^{|V| x e},  |V| - ( ),  e - .  w₀, … , w_N,   E = [Dw₀, … , Dw_N].  h₀ , …h_L  

 m, n – , ,  k - , X ∈ R ^N×m -  h_l ( , ), 

, σ -  ⊗  . 

 ,  h_i . , . , k-1, , - , ,  k - . 

 X * W + b,   σ(X * V + c). LSTM,  X * W + b  , . (GLU). E  H = h_L◦. . .◦h₀ (E). 

(GLU) ,      . 

3.3  LSTM 

LSTM (long short-term memory,  –  ) — ,   . LSTM    , ,  [5]. 

LSTM . — , ! 

.   , ,  tanh. 

LSTM  

LSTM . 

 

LSTM , . , « ».  h  x  0 1  C. 1  « »,  0 — « ». 

. , . , . , . 

   

, . .   , « », ,  .  tanh   - C, . . 

, . 

 

 C. , . 

 f, ,   .  i*C. , , . 

,   . 

 , , . .  , , .  tanh ( [-1, 1]) . 

, , , . ,   , ( ) . 

   TabNet 

 

3.4.  Split:  

 Feature Transformer  , .  ,      ,  Attentive Transformer  ,   .     (backpropagation) , «» ,    ( ). , .   ,  Attentive Transformer  . , "" , , . 

SPLIT 

:  (. . 1) . 

,  , ( ), , . 

. 3       . FC BN (GLU) ,   . √0.5 , , . . BN, , , BN BV  mB.      , , BN. , , . 3,  

: 

.  softmax  ( argmax  ). 

4.   

.   (), ( )  Softmax, ,   , :  ,  -  ,   —  . 

   

, ,  h_t, t=1 …m,  d  ,  . 

C  d     d_i−1 . 

 s —  h_i  « ». 

,   s  softmax.  e=softmax(s) 

softmax : 

:

 cc ,  h_i   e_i. 

   .  , , , ,    , .  Softmax,  Sparsemax. ,  , - , Softmax  ,  . «»  «» ,    - .  

5.   SPARSEMAX 

, z z, . : 

τ(z) S(z), p. softmax  , , ,  softmax .  

, .  softmax   ,  sparsemax  : 

, : 

,  sparsemax   ,  , : 

|S(z)| - S(z). 

, , , ,  Sparsemax.  

,   

6.   

,   , ,   , -  . . , ,      .  ( ),      () , , , . 

:   

.   ,  , , ,   . : M[i] · f. (. . 1) , , a[i − 1]: 

 Sparsemax [6] , . 

,    

h[i] - , . 4., FC, BN, P[i] - , , : 

γ - : γ = 1, γ, . P[0] , 

- . ( ),  P[0]  , .  : 

ϵ -    .    λ,     , . 

 

   , ,   . ,    ,  , -   .    ,  [5]  , . 

TabNet - . TabNet . , () ,   .  

, , , .

1. .. //  . : . 2017. .6, №3. .28–59. DOI: 10.14529/cmse170303 

   
   
   
   
   

2. LeCun Y., Bengio Y., Hinton G. Deep Learning // Nature. 2015. Vol.521. Pp.436–444. DOI: 10.1038/nature14539. 

   
   
   
   
   

3. Rav`ı D., Wong Ch., Deligianni F., et al. Deep Learning for Health Informatics // IEEE Journal of Biomedical and Health Informatics. 2017. Vol.21, No.1. PP.4–21. DOI: 10.1109/JBHI.2016.2636665. 

   
   
   
   
   

4. Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal // Proceedings of The 32nd International Conference on Machine Learning (2015), pp.448-456. 

   
   
   
   
   

5. Sercan O. Arik, Tomas Pfister. TabNet: Attentive Interpretable Tabular Learning // ICLR 2020 Conference Blind Submission 25 Sept 2019 (modified: 24 Dec 2019). URL:https://drive.google.com/file/d/1oLQRgKygAEVRRmqCZTPwno7gyTq22wbb/view?usp=sharing 

   
   
   
   
   

6. Andre F. T. Martins and Ram´on Fern´andez Astudillo. 2016. From Softmax´ to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. arXiv:1602.02068.