Analysis of vacancies using clustering

The article provides a method for analyzing vacancies for the search query "Python" using the Gaussian Mixture Model (GMM) clustering model. For each selected cluster, the most requested skills and salary range will be shown.



What was the purpose of this study? I wanted to know:

• In what applications is Python used
• What knowledge is required: databases, libraries, frameworks
• How much specialists in each area are in demand
• What salaries are offered

Loading data

Jobs downloaded from the site hh.ru , using the API: dev.hh.ru . At the request of "Python", 1994 vacancies were uploaded (Moscow region), which were divided into training and test suites, in the proportion of 80% and 20% . The size of the training set is 1595 , the size of the test set is 399 . The test set will only be used in the Top / Antitop skills and Job Classification sections.

Signs

Based on the text of uploaded vacancies, two groups of the most common n-grams of words were formed :

• 2-grams in Cyrillic and Latin
• 1-grams in Latin

In IT vacancies, key skills and technologies are usually written in English, so the second group included words only in Latin.



After selection of the first n-gram group contained 81 grams of 2, 1 and the second 98-gram:



It was decided to divide vacancies into clusters according to the following criteria, in order of priority:



Determination of which group of criteria the n-gram belongs to, and what weight to assign to it, occurred on an intuitive level. Here are a couple of examples:

1. At first glance, "Docker" can be attributed to the second group of criteria with a weight of 4 to 6. But the mention of "Docker" in the vacancy most likely means that the vacancy will be for the position of "DevOps engineer". Therefore, "Docker" fell into the first group and received a weight of 7.
   
   
2. «Java» , .. «Java» Java- « Python». «-». , , , «Java» 9.
   
   

n- — .

For calculations, each vacancy was transformed into a vector with a dimension of 179 (the number of selected features) of integers from 0 to 9, where 0 means that the i-th n-gram is absent in the vacancy, and the numbers from 1 to 9 mean the presence of the i-th n - grams and its weight. Further in the text, a dot means a vacancy represented by such a vector.

Example:

Let's say the list of n-grams contains only three values:

No.	n	n-gram	Weight	Vacancies
1	2	in python	eight	258
2	2	understanding of principles	eight	221
3	1	sql	five	490

Then for a vacancy with text.



Requirements:

• 3+ years of experience in python development .
• Good knowledge of sql

the vector is equal to [8, 0, 5].

Metrics

To work with data, you need to have an understanding of it. In our case, I would like to see if there are any clusters of points, which we will consider as clusters. To do this, I used the t-SNE algorithm to translate all vectors into 2D space. 



The essence of the method is to reduce the dimension of the data, while keeping the proportions of the distances between the points of the set as much as possible. It is quite difficult to understand how t-SNE works from the formulas. But I liked one example found somewhere on the Internet: let's say we have balls in three-dimensional space. We connect each ball with all other balls by invisible springs, which do not intersect in any way and do not interfere with each other when crossing. The springs act in two directions, i.e. they resist both the distance and the approach of the balls to each other. The system is in a stable state, the balls are stationary. If we take one of the balls and pull it back, and then release it, it will return to its original state due to the force of the springs. Next, we take two large plates, and squeeze the balls into a thin layer,while not interfering with the balls to move in the plane between the two plates. The forces of the springs begin to act, the balls move and eventually stop when the forces of all the springs become balanced. The springs will act so that the balls that were close to each other remain relatively close and flat. Also with removed balls - they will be removed from each other. With the help of springs and plates, we converted the three-dimensional space into two-dimensional, preserving the distance between the points in some form!Also with removed balls - they will be removed from each other. With the help of springs and plates, we converted the three-dimensional space into two-dimensional, preserving the distance between the points in some form!Also with removed balls - they will be removed from each other. With the help of springs and plates, we converted the three-dimensional space into two-dimensional, preserving the distance between the points in some form!



T-SNE algorithm used by me only for visualization of the set points. He helped choose the metric, as well as select the weights for the features.



If we use the Euclidean metric that we use in our daily life, then the location of vacancies will look like this:





The figure shows that most of the points are concentrated in the center, and there are small branches to the sides. With this approach, clustering algorithms that use the distances between points will not produce anything good.



There are many metrics (ways to determine the distance between two points) that will work well on the data you are exploring. I have chosen as a measure of the distance Jaccard , taking into account the weights of n-grams. Jaccard's measure is easy to understand, but it works well for solving the problem under consideration.

:

1 n-: « python», «sql», «docker»

2 n-: « python», «sql», «php»

:

« python» — 8

«sql» — 5

«docker» — 7

«php» — 9

(n- 1- 2- ):  « python», «sql» = 8 + 5 = 13

( n- 1- 2- ):  « python», «sql», «docker», «php» = 8 + 5 + 7 + 9 = 29

=1 — ( / ) = 1 — (13 / 29) = 0.55

The matrix of distances between all pairs of points was calculated, the size of the matrix is ​​1595 x 1595. In total, 1,271,215 distances between unique pairs. The average distance turned out to be 0.96, between 619 659 the distance is 1 (i.e. there is no similarity at all). The following chart shows that overall vacancies are few similarities:





When using the metric Jaccard our space now looks like this:





Four distinct areas of density appeared, and two small clusters of low density. At least that's how my eyes see!

Clustering

The model was chosen as a clustering algorithm Gaussian Mixture Model (GMM) . The algorithm receives data in the form of vectors as input, and the n_components parameter is the number of clusters into which the set must be split. About how the algorithm works, you can see here (in English). I used a ready-made GMM implementation from the scikit-learn library: sklearn.mixture.GaussianMixture .



Note that GMM does not use a metric, but separates data only by a set of features and their weights. In the article, Jaccard distance is used to visualize data, calculate the compactness of clusters (I took the average distance between cluster points for compactness), and determinethe central point of the cluster (typical vacancy) - the point with the smallest average distance to other points of the cluster. Many clustering algorithms use exactly the distance between points. The Other Methods section will talk about other types of clustering that are metric-based and also give good results.



In the previous section, it was determined by eye that there will most likely be six clusters. This is how the clustering results look with n_components = 6:







In the figure with the output of clusters separately, the clusters are arranged in descending order of the number of points from left to right, from top to bottom: cluster 4 is the largest, cluster 5 is the smallest. The compactness of each cluster is indicated in brackets.



In appearance, the clustering turned out to be not very good, even considering that the t-SNE algorithm is not perfect. When analyzing clusters, the result was also not encouraging.



To find the optimal number of clusters n_components, we will use the AIC and BIC criteria , which you can read about here . The calculation of these criteria is built into the sklearn.mixture.GaussianMixture method . This is how the criteria graph looks like:





When n_components = 12, the BIC criterion has the lowest (best) value, the AIC criterion also has a value close to the minimum (minimum when n_components = 23). Let's divide vacancies into 12 clusters:







Clusters are now more compact, both in appearance and in numerical terms. During manual analysis, vacancies were divided into characteristic groups for understanding a person. The figure shows the names of the clusters. Clusters numbered 11 and 4 are marked as <Trash 2>:

1. In cluster 11, all features have approximately the same total weights.
2. Cluster 4 is dedicated to Java. Nevertheless, there are few vacancies for the position of Java Developer in the cluster, knowledge of Java is often required as "will be an additional plus."

Clusters

After deleting two uninformative clusters numbered 11 and 4, the result is 10 clusters:





For each cluster, there is a table of features and 2-grams that are most often found in the vacancies of the cluster.



Legend:



S - the proportion of vacancies in which the trait is found, multiplied by the weight of the trait

% - the percentage of vacancies in which the trait / 2-gram is found

Typical cluster vacancy - vacancy, with the smallest average distance to other points of the cluster

Data Analyst

Number of Jobs: 299



Typical Job : 35,805,914

C ++ Developer

Number of Jobs: 139

Typical Job : 39,955,360

Linux / DevOps Engineer

Number of Job Openings: 126

Typical Job : 39,533,926

Python Developer

Number of Job Openings: 104

Typical Job : 39,705,484

Data scientist

Number of vacancies: 98

Typical vacancy: 38071218

Frontend Developer

Number of Jobs: 97

Typical Job : 39,681,044

Backend Developer

Number of Jobs: 93

Typical Job : 40,226,808

DevOps Engineer

Number of Jobs: 78

Typical Job : 39634258

Data Engineer

Number of Jobs: 77

Typical Job : 40,008,757

QA Engineer

Number of Jobs: 56

Typical Job : 39630489

Salaries

Salaries are indicated only in 261 (22%) vacancies out of 1,167 in the clusters.



When calculating salaries:

1. If the range "from ... to ..." was specified, then the average value was used
2. If only "from ..." or only "to ..." was indicated, then this value was taken
3. The calculations used (or were given) salary after taxes (NET)

On the chart:

1. Clusters rank in descending order of median salary
2. Vertical bar in box - median
3. Box - range [Q1, Q3], where Q1 (25%) and Q3 (75%) are percentiles. Those. 50% of salaries fall into the box
4. The "mustache" includes salaries from the range [Q1 - 1.5 * IQR, Q3 + 1.5 * IQR], where IQR = Q3 - Q1 - interquartile range
5. Individual points - anomalies that did not fall into the mustache. (There are anomalies not included in the diagram)

Top / Antitop skills

The charts were built for all 1994 uploaded vacancies. Salaries are indicated in 443 (22%) vacancies. For the calculation for each feature, vacancies were selected where this feature is present, and on their basis the median salary was calculated.

Job classification

Clustering could be made much easier without resorting to complex mathematical models: to compile the top names of vacancies, and divide them into groups. Next, analyze each group for top n-grams and average salaries. There is no need to highlight features and assign weights to them.



This approach would work well (to some extent) for a "Python" query. But for the request "1C Programmer" this approach will not work, because for 1C programmers in the names of vacancies, 1C configurations or applied areas are rarely indicated. And there are many areas where 1C is used: accounting, salary calculation, tax calculation, cost calculation at manufacturing enterprises, warehouse accounting, budgeting, ERP systems, retail, management accounting, etc.



For myself, I see two tasks for analyzing vacancies:

1. Understand where a programming language that I know little about is used (as in this article).
2. Filter new posted jobs.

Clustering is suitable for solving the first problem, for solving the second one - various classifiers, random forests, decision trees, neural networks. Nevertheless, I wanted to evaluate the suitability of the chosen model for the job classification problem.



If you use the predict () method built into sklearn.mixture.GaussianMixture , then nothing good happens. He attributes most of the vacancies to large clusters, and two of the first three clusters are uninformative. I used a different approach: 

1. We take the vacancy that we want to classify. We vectorize it and get a point in our space.
2. We calculate the distance from this point to all clusters. Under the distance between a point and a cluster, I took the average distance from this point to all points in the cluster.
3. The cluster with the smallest distance is the predicted class for the selected vacancy. The distance to the cluster indicates the reliability of such a prediction.
4. To increase the accuracy of the model, I chose 0.87 as the threshold distance, i.e. if the distance to the nearest cluster is greater than 0.87, then the model does not classify the vacancy.

To evaluate the model, 30 vacancies were randomly selected from the test set. In the verdict column:



N / a: the model did not classify the job (distance> 0.87)

+: correct classification

-: incorrect classification  



Total: 12 vacancies have no result, 2 vacancies - erroneous classification, 16 vacancies - correct classification. Model completeness - 60%, model accuracy - 89%.

Weak sides

The first problem - let's take two vacancies:

Vacancy 1 - "Lead C ++ Programmer"

"Requirements:

• 5+ years of C ++ development experience.
• Knowledge of Python will be an additional plus. "

Vacancy 2 - "Lead Python Programmer"

"Requirements:

• 5+ years of Python development experience.
• Knowledge of C ++ will be an additional plus "

From the point of view of the model, these vacancies are identical. I tried to adjust the weights of the features by the order of their occurrence in the text. This did not lead to anything good.



The second problem is that GMM clusters all the points in a set, like many clustering algorithms. Non-informative clusters are not a problem by themselves. But informative clusters also contain outliers. However, this can be easily solved by clearing the clusters, for example, by removing the most atypical points that have the greatest average distance to the rest of the cluster points.

Other methods

The cluster comparison page demonstrates the various clustering algorithms well. GMM is the only one that gave good results.

The rest of the algorithms either did not work or gave very modest results.



Of those implemented by me, good results were in two cases:

1. Points with a high density were selected in some neighborhood, located at a distant distance from each other. The points became the centers of the clusters. Then, on the basis of the centers, the process of cluster formation began - the joining of neighboring points.
2. Agglomerative clustering is an iterative merging of points and clusters. The scikit-learn library presents this kind of clustering, but it does not work well. In my implementation, I changed the join matrix after each iteration of the merge. The process stopped when some boundary parameters were reached - in fact, dendrograms do not help to understand the merging process if 1500 elements are clustered.

Conclusion

The research I did gave me the answers to all the questions at the beginning of the article. I got hands-on experience with clustering while implementing variations of known algorithms. I really hope that the article will motivate the reader to carry out his analytical research, and will somehow help in this exciting lesson.