Catching bugs on the client: how we wrote our system to collect client errors

Badoo . Badoo Bumble, - ( ), (Android iOS), UI.  Gelat. Error Tracking . 

, , , , .

:

• , , , . 

• , ;

• , , .

— . , , , , .

— . Badoo : 

— -: - , ; 

— : ( , App Store Google Play). 

. - , , .

. , , QA-. 

Badoo : HockeyApp - JS-.

HockeyApp

HockeyApp , 2014 Microsoft , App Center. App Center : , , - Android- DexGuard mapping-, . ; , , - . 

Microsoft — 16 2019 , HockeyApp App Center. , DexGuard App Center 2019 , HockeyApp.

HockeyApp. Microsoft HockeyApp, , . .

JS-

JS-, , .

:

• MySQL ( 10—20 );

• ;

• MySQL, Sphinx .

2017 -. — . - , - , - , .

, , . , 2014 , , . 

, .

• .

• . (, ) . , .

• . , . .

• — .

• « ?», « ?».

• - ( , , . .).

• .

• : , , , . .

• Jira. Jira, .

• Self-hosted. , . .

, , . 

SaaS-, : — . Bugsnag, TrackJS, Raygun, Rollbar Airbrake. , : , , HockeyApp. — .

Open-source-

open-source- . , .

Sentry , . ( 2018 ) :

• ;

• PostgreSQL, ;

• , Python, — PHP Go;

• (, Jira).

2018 Sentry, issue- , , — ClickHouse ( ). , . , , , . 

Gelato (General Error Logs And The Others), .

, , , , , , . : , , , .

.

, .

, .

, , , , . Jira .

:

, .

(1), (2) (3).

, .

, , — . , , , , - ().

: 

, :

, , . , , :

• A/B- — , -;

• ;

• ( );

• email- .

, . :

1. .

2. .

3. .

:

.

, , , — API .

API:

• ;

• , «» ( « », «» API);

• .

?

, EPS (errors per second), , , .

- , . . , , , EPS ( iOS Facebook SDK 10 2020 ). , , .

, , — . : , - — , . 

, ( ), , .

?

1. , , — , Redis RabbitMQ.

2. Apache Kafka, , (, - ). Kafka () Sentry.

3. LSD (Live Streaming Daemon). , . Badoo , .

: « ?» ( ) « ?» ( ).

: Elasticsearch ClickHouse.

Elasticsearch

:

• ;

• , ;

• ;

• UPDATE ( , , , );

• DELETE ( , , );

• API, .

, , Elasticsearch :

• , ( SQL, (X-Pack) Open Distro Amazon);

• JVM, « », — PHP Go (, , ; 6.8 7.5, (, ));

• ; , , (, deflate- LZ4, CPU, ).

ClickHouse

:

• ; 

• , ;

• MySQL- , , Elasticsearch;

• , Elasticsearch.

2018 ClickHouse :

• DELETE ( , ; ClickHouse , ( — ) );

• UPDATE : ClickHouse , — ( GitHub — 2018 , );

• ( RegExp, (full scan), ).

, ( ()). , , Elasticsearch.

, Elasticsearch ClickHouse, . . , Elasticsearch, , — .

, .

:

• -;

• .

( ) — . , warm- ( hot-warm-cold-).

JS-, - , , ( , , , ). , , attributes . 

Sentry . Sentry , tags , , extra — .

, , — . , , , , - . — .

.

- Android-

, Android- , :

• (code shrinking — );

• , (optimization — );

• , , , - (obfuscation — ).

.

:

• ProGuard ( ); 

• DexGuard ProGuard ( );

• R8 Google.

, - :

o.imc: Error loading resources: Security check required
  at o.mef.b(:77)
  at o.mef.e(:23)
  at o.mef$a.d(:61)
  at o.mef$a.invoke(:23)
  at o.jij$c.a(:42)
  at o.jij$c.apply(Unknown Source:0)
  at o.wgv$c.a_(:81)
  at o.whb$e.a_(:64)
  at o.wgs$b$a.a_(:111)
  at o.wgy$b.run(:81)
  at o.vxu$e.run(:109)
  at android.os.Handler.handleCallback(Handler.java:790)
  at android.os.Handler.dispatchMessage(Handler.java:99)
  at android.os.Looper.loop(Looper.java:164)
  at android.app.ActivityThread.main(ActivityThread.java:6626)
  at java.lang.reflect.Method.invoke(Native Method)
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:811)

, . -, . , mapping.txt, . :

AllGoalsDialogFragment -> o.a:
    java.util.LinkedHashMap goals -> c
    kotlin.jvm.functions.Function1 onGoalSelected -> e
    java.lang.String selectedId -> d
    AllGoalsDialogFragment$Companion Companion -> a
    54:73:android.view.View onCreateView(android.view.LayoutInflater,android.view.ViewGroup,android.os.Bundle) -> onCreateView
    76:76:int getTheme() -> getTheme
    79:85:android.app.Dialog onCreateDialog(android.os.Bundle) -> onCreateDialog
    93:97:void onDestroyView() -> onDestroyView

, , mapping- — -.

(, , ), , , ProGuard ( DexGuard ) ReTrace, .

Android- Kotlin, : 

• - ;

• mapping- Ceph ( TeamCity);

• -.

- iOS-

- iOS- , - ( - ). : - , .

Thread 0:
0   libsystem_kernel.dylib              0x00000001bf3468b8 0x1bf321000 + 153784
1   libobjc.A.dylib                     0x00000001bf289de0 0x1bf270000 + 105952
2   Badoo                               0x0000000105c9c6f4 0x1047ec000 + 21694196
3   Badoo                               0x000000010657660c 0x1047ec000 + 30975500
4   Badoo                               0x0000000106524e04 0x1047ec000 + 30641668
5   Badoo                               0x000000010652b0f8 0x1047ec000 + 30667000
6   Badoo                               0x0000000105dce27c 0x1047ec000 + 22946428
7   Badoo                               0x0000000105dce3b4 0x1047ec000 + 22946740
8   Badoo                               0x0000000104d41340 0x1047ec000 + 5591872

. -, - (dSYM), , , .

?

• , , , macOS. , , Linux.

• Symbolicator Sentry, ( , ). - , as is : , Rust . 

• Symbolic Sentry, , Rust, C-ABI — FFI.

Golang , Symbolic cgo.

, , — . , .

, , , , - . (fingerprint), . -. , , ( ).

:

• JS- message, type origin;

• Android - - ( );

• iOS - (, crashed -).

, .

, Sentry, . 

, , . 

, :

• : , ;

• ;

• , ;

• iOS - — , Symbolicator;

• : ; 

• .