In this article, I will tell a story about how you can face a rather simple problem on large installations of Atlassian products, in particular on Jira.
Methods for analyzing and finding bottlenecks:
statistics
sampling
profiling and tracing
In this article, we will analyze the request trace, which shows almost the entire segment of the request, namely from the beginning of the request from the client's browser to the transition to the reverse proxy, if it exists, to the application server and from it to caches, Lucene search indexes, DBMS.
An important point is that it is difficult to analyze the system on the part of the customer, since they had strict requirements for the use of the tools that are listed in the requirements.
Requirements
APM- ( ) , . — . , , open source . — , .
:
;
(On-Premise);
;
Elasticsearch (Opendistro);
JVM ( Atlassian , Java), javaagent;
HTTP c SQL-;
;
, , . , 12 , 1 . . , .
https://openapm.io/landscape, APM-. Glowroot, .
, , Cassandra ( ) Elasticsearch.
wget -c https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip
2. mkdir -p /jira/glowroot/tmp, Jira , , Jira
chown -R jira: /jira/home/glowroot
3. setenv.sh jira, /jira/current/bin. , /opt/atlassian/jira/bin :
JVM_SUPPORT_RECOMMENDED_ARGS="-javaagent:/jira/glowroot/glowroot.jar ${JVM_SUPPORT_RECOMMENDED_ARGS}"
4. :
systemctl restart jira
:
tail -f {jira_installation_directory}/logs/catalina.out
127.0.0.1, glowroot , , admin.json. bindAddress 0.0.0.0 . contextPath - /glowroot.
reverse proxy ( nginx), :
location /glowroot {
proxy_pass http://127.0.0.1:4000;
}
Agent-Installation-(with-Embedded-Collector)
, jira.example.com/glowroot .
, .
, , , .
,
glowroot , 30000, . trace, .
, PostgreSQL, PostgreSQL IOPS.
— Jira , thread dump , .
stacktrace , .
stacktrace , createTemporaryTablesIfNeeded .
. .
entity engine, , .
, .
:
unlogged cwd_users PostgreSQL
, ? , , ? , ?
, Atlassian .
, .