We develop the world's most convenient * interface for viewing logs

If you have ever used web interfaces to view logs, then you probably noticed how cumbersome, as a rule, these interfaces are and (often) not very convenient and responsive. You can get used to some, some are completely awful, but it seems to me that the reason for all the problems is that we are approaching the problem of viewing logs incorrectly: we are trying to create a web interface where the CLI (command line interface) works better. I am personally very comfortable working with tail, grep, awk and others, and therefore for me the ideal interface for working with logs would be something similar to tail and grep, but which at the same time could be used to read logs that came from many servers ... That is, of course, read them from ClickHouse!

* according to the personal opinion of the habrapuser youROCK

Meet logscli

I have not come up with a name for my interface, and, to be honest, it rather exists in the form of a prototype, but if you want to see the source right away, then welcome: https://github.com/YuriyNasretdinov/logscli (350 lines of selected Go code) ...

Capabilities

My goal was to make an interface that looks familiar to those who are used to tail / grep, that is, to support the following things:

1. View all logs, without filtering.
2. Leave lines containing a fixed substring (flag -Fy grep).
3. Leave the lines matching the regular expression ( -Ey flag grep).
4. By default, scans are in reverse chronological order, since the most recent logs are usually of interest first.
5. ( -A, -B -C grep, N , , ).
6. , ( tail -f | grep).
7. less, head, tail — ; , ; SIGPIPE , , tail, grep UNIX-.

, - ClickHouse. , lsd kittenhouse, .

. , . , — , ClickHouse ( ~1 ).

, :

CREATE TABLE logs(
    category LowCardinality(String), --   ()
    time DateTime, --  
    millis UInt16, --  (   ,  ..):  ,   ,       
    ..., --   ,   ,  ,   
    message String --  
) ENGINE=MergeTree()
ORDER BY (category, time, millis)

, - , , Amazon 2015 . , , , .

CREATE TABLE amazon(
   review_date Date,
   time DateTime DEFAULT toDateTime(toUInt32(review_date) * 86400 + rand() % 86400),
   millis UInt16 DEFAULT rand() % 1000,
   marketplace LowCardinality(String),
   customer_id Int64,
   review_id String,
   product_id LowCardinality(String),
   product_parent Int64,
   product_title String,
   product_category LowCardinality(String),
   star_rating UInt8,
   helpful_votes UInt32,
   total_votes UInt32,
   vine FixedString(1),
   verified_purchase FixedString(1),
   review_headline String,
   review_body String
)
ENGINE=MergeTree()
ORDER BY (time, millis)
SETTINGS index_granularity=8192

for i in *.tsv; do
    echo $i;
    tail -n +2 $i | pv |
    clickhouse-client --input_format_allow_errors_ratio 0.5 --query='INSERT INTO amazon(marketplace,customer_id,review_id,product_id,product_parent,product_title,product_category,star_rating,helpful_votes,total_votes,vine,verified_purchase,review_headline,review_body,review_date) FORMAT TabSeparated'
done

, , .

ClickHouse full scan , , , . HTTP- , HTTP: send_progress_in_http_headers=1. , Go , HTTP 1.0 ( 1.1!) ClickHouse, TCP- ClickHouse, GET /?query=... HTTP/1.0\n\n - , .

ClickHouse

ClickHouse ( 2019 ?) ORDER BY,

SELECT time, millis, message
FROM logs
WHERE message LIKE '%something%'
ORDER BY time DESC, millis DESC

, message "something", .

, , ClickHouse , , . cancel_http_readonly_queries_on_client_close=1.

SIGPIPE Go

, , some_cmd | head -n 10, some_cmd , head 10 ? : head , pipe , stdout some_cmd , , «». some_cmd pipe, SIGPIPE, .

Go , SIGPIPE "signal: SIGPIPE" , , SIGPIPE , , :

ch := make(chan os.Signal)
signal.Notify(ch, syscall.SIGPIPE)
go func() {
    <-ch
    os.Exit(0)
}()

, - (, , ), grep -A, -B -C, , , .

, ClickHouse, , , ( , ):

SELECT time,millis,review_body FROM amazon
WHERE (time = '_' AND millis < _) OR (time < '_')
ORDER BY time DESC, millis DESC
LIMIT __
SETTINGS max_threads=1

, ClickHouse , CPU ( ~6 ).

, () , , timestamp, .

logscli ?

Amazon, , :

#  ,    walmart
$ logscli -F 'walmart' | less

#    10 ,   "terrible"
$ logscli -F terrible -limit 10

#     -limit:
$ logscli -F terrible | head -n 10

#   ,   /times [0-9]/,   vine     
$ logscli -E 'times [0-9]' -where="vine='Y' AND star_rating>4" | less

#      "panic"  3   
$ logscli -F 'panic' -C 3 | less

#       "5-star"
$ logscli -F '5-star' -tailf

( ) github https://github.com/YuriyNasretdinov/logscli. ClickHouse.