🤦🏿 🔞 🦆 Certification. Is it worth using dumps and what does Markov chains have to do with it? 🏦 🥂 🚵🏾

More than 20 years ago I became interested in telecom. I started by reading books, one of which was the CCNA course. Then it was a rather thin little book with FDDI, Token Ring, ISDN and similar things that the new generation of networkers have only heard of. And then I first read about CCIE certification. I got the impression that the holders of this status are truly IT gods. Of course, I wanted to achieve this net samadhi.

Much time has passed since then. I received CCNA and then CCNP, I successfully expired all these certificates for uselessness, and after 10 years I became CCIE. I still do networking, and I must say I seem to love my job, but of course romanticism has passed. Twenty years ago, the attitude to all this was very different. It was a drive, it was a real adventure.

Now there is no such interest, and the attitude to certification among different engineers is also different, and I myself am ambivalent about this. I clearly realize that this is a kind of game, often not entirely fair game, and if you do not know the rules, then, even with excellent knowledge and skills, you have a fairly large chance of losing. In this article, we will use simple mathematics to consider these rules, which will allow you to create your own as honest and at the same time winning strategy.

An honest approach

Let's try to understand which approach to assessing our knowledge would be the most honest?

As the first condition, I would name a clear definition of the range of theoretical questions and practical skills that can be tested by testing.
Further, we would like to have reasonable detail and specificity. What is the point in knowing how well I know the insignificant details in my work, and whether I can accurately do / remember / explain what is useful to me a couple of times in my life and which is easily googled in one minute. I would like the questions to be substantive.
And the third component, which is quite important for this article, is that I would like my skills and knowledge that I need in my professional activity to be tested, and not the skills to pass this type of exam.

Let's assume the first two conditions are met. Let's examine the dependence of the "fairness" of our exam on the third component.

To be specific, we will talk specifically about testing with a binary result: passed (get a certificate) / did not pass (no certificate). Then we can present the exam results in one of the simplest versions in the form of the following diagram:

Arrows

p_{11}

$p_{11}$ ,

p_{12}

$p_{12}$ ,

p_{13}

$p_{13}$ denote the probabilities of transition from one state to another. So,

p_{11}

$p_{11}$ indicates the likelihood that you will try to pass the exam again after an unsuccessful attempt. If there is no transition between states, then this means that the transition probability is 0.

For example, the probabilities can be distributed as follows

The essence of this type of testing is that the test-takers do not have any insider information, or insider doesn’t improve the chances of passing the test. Of course, the requirements for practical skills as well as for the volume and depth of theoretical knowledge are known, but there is no useful information on the technique of passing the exam.

And it is precisely this situation that, in my opinion, is perfectly honest. In this case, your knowledge is tested, not how you learned to pass the exam.

1

, . . , , , , . , , , , , . , , , , .

Example 2

As another example, consider technical interviews for hiring employees. If your company is small, you have correctly drawn up an assignment and are interviewing only a small number of people, then, in principle, you can draw up a sufficient number of assignment questions so that they do not overlap or build your conversation interactively, based on the experience of applicants.

But if we are talking about popular certificates, then this approach does not work. There are several reasons for this, for example, I will give two:

such an individual approach is too expensive
need standardization

Fair approach

Usually we are faced with a situation where we must prepare specifically for the exam. I am not talking about theoretical material and not about the skills that we must possess and which are certainly important and useful in our daily work activities. I'm talking about the fact that we must develop skills specifically for this type of exam (I still assume that there are no dumps), and if you do not train this specific skill, then the probability of success is significantly reduced.

, CISSP . , , , , ( ), . ( , ). , , , , . , (, , , ...).

You can practice this skill ahead of time or by taking the exam several times.

This approach can still be considered fairly fair, because everyone is on a roughly equal footing here, which is why I called this type of exam the “fair approach”.

What will be the transition graph in this case?

We have the following states:

No experience. You decided to take the exam "absolutely honestly", that is, without specially training the technique of passing the exam. You know the theory and have the necessary practical skills. It seems to you that this should be enough to get a decent rating.
Did not pass. Experience. This is a state where you have additional knowledge and skills regarding the examination procedure. This knowledge was obtained either through an unsuccessful attempt or through purposeful preparation. Let me remind you once again that in this case we are not talking about dumps.
I passed it.
I gave up.

Arrows

p_{12}

$p_{12}$ ,

p_{13}

$p_{13}$ ,

p_{14}

$p_{14}$ , ... denote the probabilities of transition from one state to another. If there is no transition between states, then this means that the probability of the transition is 0.

And now let's ask ourselves the question: how much will the probability of your surrender decrease if you expect an "honest approach", and simply rely on your experience and knowledge and will not specifically prepare for technique for passing this exam (that is, we are interested in

p_{13}

$p_{13}$ )? Of course, if the difficulty of the exam is the same as in the case of "fair approach", then your chances will be the same. But it seems reasonable to me to make one important assumption that radically changes the situation.

Let's assume that the vendor tries to maintain a certain difficulty of the test, and does so by keeping the pass rate below a certain level. Suppose that the difficulty level of the questions is selected in such a way that the passing percentage would not be higher than 50%.

For definiteness, let us set the following conditions:

The vendor maintains the complexity of the task at such a level that on average no more than 50% of applicants would successfully complete the task. We will carry out calculations at this border.
, 5% (): $p_{14} = p_{24} = 0,05$
. , 2 30% , 1: $p_{23} = p_{12} + (1 - p_{12})*0,3$
, . .
, 50% ( 1) 50% ( 2). , , .

What percentage of successful delivery will be from state 1?

Now the probability of surrender will be only 36% versus 50% in the case of an "honest" approach.

The probability of surrendering from state 2 is 55%.

The complete graph will look like this:

You can find proof here

, :

p_{41} = p_{31} = p_{42} = p_{32} = 0, 5

$p_{41} = p_{31} = p_{42} = p_{32} = 0,5$ . .. 3 4, 1 2 , . , : (/) . 0,5 :

« , 50% ( 1) 50% ( 2)».

A = (\begin{matrix} 0 & 0 & 0, 50 & 0, 50 \\ 0, 59 & 0, 40 & 0, 50 & 0, 50 \\ 0, 36 & 0, 55 & 0 & 0 \\ 0, 05 & 0, 05 & 0 & 0 \end{matrix})

$A = \begin{pmatrix} 0 & 0 & 0,50 & 0,50\\ 0,59 & 0,40 & 0,50 & 0,50\\ 0,36 & 0,55 & 0 & 0\\ 0,05 & 0,05 & 0 & 0\\ \end{pmatrix}$

p = (\begin{matrix} p_{1} \\ p_{2} \\ p_{3} \\ p_{4} \end{matrix})

$p = \begin{pmatrix}p_1\\p_2\\p_3\\p_4\\\end{pmatrix}$

p_{1}, p_{2}, p_{3}, p_{4}

$p_1, p_2, p_3, p_4$ — 1,2,3,4

p_{1} + p_{2} + p_{3} + p_{4} = 1

$p_1 + p_2+p_3+p_4 = 1$

, : n , , 2. , ( ) .

, :

A p = p

$Ap = p$

, 1.

, , matrixcalc.org

λ = 1

$\lambda = 1$ :

p = p_{4} (\begin{matrix} \frac{400}{73} \\ \frac{1060}{73} \\ \frac{727}{73} \\ 1 \end{matrix})

$p = p_4\begin{pmatrix}\displaystyle\frac{400}{73}\\\displaystyle\frac{1060}{73}\\\displaystyle\frac{727}{73}\\1\\\end{pmatrix}$

p_{4}

$p_4$

p_{1} + p_{2} + p_{3} + p_{4} = 1

$p_1 + p_2+p_3+p_4 = 1$

1 2. , :

1: $\displaystyle\frac{400}{1060+400} \approx 0,27$
2: $\displaystyle\frac{1060}{1060+400} \approx 0,73$

27% ( 1) 73% ( 2). :

0, 27 * 0, 36 + 0, 73 * 0, 55 = 0, 50

$0,27 * 0,36 + 0,73*0,55 = 0,50$

.

, .

That is, if you are trying to pass the exam "honestly", which in the context of this article means that you rely on your theoretical knowledge and practical skills within the framework of this exam, but without specific preparation for the actual examination procedure itself, without trying to adapt your way of thinking to the logic of the test authors, without some insider information, then the probability that you will pass this exam is reduced by one and a half times. To pass in this way, your level of knowledge must be well above average and in fact well above the level required for this exam.

It is clear that with this approach, the proportion of people trying to pass "honestly" will decrease, the exam begins to get more difficult (to keep the 50% condition), which leads to the fact that the probability of passing from state 1 becomes less and less.

And yet I think this approach is fair. There is no unfair competition here. You just need to understand that you need to prepare specifically for the exam. Without this, even with excellent knowledge, the probability of passing the test is small (but there is).

Now let's say there are dumps. How will the picture change?

Dumps. Unfair competition

Now our schema has changed:

Another state of "Dumpa" appeared. This means that the applicant can use the exam dumps, which will give him a 100% chance of passing. Now let's set specific conditions.

- , , 50% ( ).
, , 30%.
4. ( $p_{14}$ ) 10%, $p_{24}$ , 5%. , , 1 ( ), , .
, 20% . 1 2 ( 40%).
We will assume that if the first attempt (from state 1 or 2) was unsuccessful, and the applicant nevertheless decided to continue, then exactly half of them will use the dump: $p_{12} = p_{15}$ and $p_{25} = p_{22}$

Let's see how the picture changes. Here's the solution:

You can find proof here

, :

p_{41} = p_{31} = p_{42} = p_{32} = 0, 4

$p_{41} = p_{31} = p_{42} = p_{32} = 0,4$

p_{45} = p_{35} = 0, 2

$p_{45} = p_{35} = 0,2$ . , . «» :

«, 20% . 1 2 ( 40%).»

A = (\begin{matrix} 0 & 0 & 0, 40 & 0, 40 & 0 \\ 0, 43 & 0, 31 & 0, 40 & 0, 40 & 0 \\ 0, 05 & 0, 34 & 0 & 0 & 1 \\ 0, 10 & 0, 05 & 0 & 0 & 0 \\ 0, 42 & 0, 30 & 0, 20 & 0, 20 & 0 \end{matrix})

$A = \begin{pmatrix}0 & 0 & 0,40 & 0,40 & 0\\0,43 & 0,31 & 0,40 & 0,40 & 0\\0,05 & 0,34 & 0 & 0 & 1\\0,10 & 0,05 & 0 & 0 & 0\\0,42 & 0,30 & 0,20 & 0,20 & 0\\\end{pmatrix}$

, :

A p = p

$Ap = p$

, , , www.wolframalpha.com

λ = 1

$\lambda = 1$ :

p \approx p_{5} (\begin{matrix} 0, 649 \\ 1, 344 \\ 1, 489 \\ 0, 132 \\ 1 \end{matrix})

$p \approx p_5\begin{pmatrix}0,649\\1,344\\1,489\\0,132\\ 1\end{pmatrix}$

p_{5}

$p_5$

p_{1} + p_{2} + p + 3 + p_{4} + p_{5} = 1

$p_1 + p_2 + p+3 + p_4 + p_5 = 1$

, , .

1,2,5:

1: $\displaystyle\frac{0,649}{0,649+1,344 + 1} \approx 0,22$
2: $\displaystyle\frac{1,344}{0,649+1,344 + 1} \approx 0,45$
5: $\displaystyle\frac{1}{0,649+1,344 + 1} \approx 0,33$

, :

0, 22 * 0, 05 + 0, 45 * 0, 34 + 0, 33 = 0, 494 \approx 0, 5

$0,22*0,05 + 0,45*0,34 + 0,33 = 0,494\approx 0,5$

We see that the probability of passing the exam from state 1 is only 5%, which means that you have almost no chance. It would seem that for state 2, everything is not so bad - 34%, but, in reality, for small values

p_{13}

$p_{13}$ such a calculation

p_{23}

$p_{23}$ not entirely correct. The fact is that our condition “experience reduces the probability of failing the exam by 30%” artificially limits the probability of

p_{23}

$p_{23}$ thirty percent (it cannot be lower), which, of course, is not entirely correct.

The only conclusion that can be made for now is that under our conditions we can forget about the possibility of surrendering from state 1 (no experience). Very soon, the community for which this exam is of interest realizes this, and the number of “idealists” trying to pass the exam without proper preparation becomes negligible. Then we get another diagram:

Under the same conditions as in the previous example, we get the following solution:

In this case, the distribution on the exam will be as follows:

State 1 (experience):

\approx 66

$\approx66$ %

State 2 (dump):

\approx 34

$\approx34$ %

I think the method of proof is obvious from the previous two examples, so I omit it.

This means that when dumps appear (and under the conditions that we have imposed), we will get a drop in the probability of passing for people who specially prepared for the exam or had an experience of unsuccessful passing and at the same time do not use dumps from 55% to 25%, that is more than 2 times.

Now let's remember the conditions.

At the first attempt, only 20% of us decided to immediately use the dump. After a failed attempt, only half of them decide to use the dump. And this rather mild condition has led to the fact that it becomes very difficult for applicants preparing for the exam without dumps to pass the exam. On average, they will have to take this exam 4 times. Preparation (time) and delivery costs are getting quite high. In this case, you can pass on the dump. This will require significantly less preparation time and will give an almost 100% chance of passing.

So let's take a look at the pros and cons that the applicant gets in both cases.

What do those who try to take without dumps have?

(-) time spent on preparation (money)
(-) money spent for unsuccessful attempts
(-) in case of failure, management and colleagues begin to doubt their competence
(+) good level of knowledge

Those who pass with dumps

(-) cost of the dump (optional: the dump can be freely available)
(-) perhaps an insufficient level of knowledge (optional: no one bothers you to prepare well while doing this)
(-) moral and ethical costs associated with unfair competition (optional and very individual)
(+) minimum spent resources
(+) respect for colleagues and management (you don't have to say that you used the dump)

In principle, theoretically, all this can be expressed in price and simply compare the prices of fair change and change with a dump. At the same time, what is marked with a minus (-) adds value, and what is positive, on the contrary, decreases. It is clear that some of the things on the list are very individual (for example, the moral and ethical costs), and therefore the price of dumping and the price of honest delivery will vary greatly for different people.

When the price of a dump becomes less than the price of fair change, a person starts using dumps.

Saturation state

With a low cost of a dump, this leads to a chain reaction: the more people use dumps, the more difficult the exam becomes and the less likely it is to pass without dumps, which forces more and more people to resort to unfair competition, and ultimately this leads to a saturation situation where on the first try, the number of applicants using the dumps already exceeds the given passing percentage (50% in our examples).

As a result, the vendor is forced to take some action. The simplest step is to increase the refresh rate of the test item. At these moments, the dumps become irrelevant, and due to this, the likelihood of delivery with a dump can be reduced.

Example

Suppose that the exam changes every 3 months, it takes 2 weeks for a new current dump to appear, then the probability of passing with a dump drops from 100% to $\approx 85$ %. But this may not be enough, so the exam becomes more and more complicated, different options appear, it is done in such a way that even with a dump, intensive training is required.

In this way,

the exam becomes incredibly difficult
chances to pass without dumps are few
even to pass with dumps, you have to seriously prepare, in fact train
exams and topics change frequently and unexpectedly. At these moments, those who prepared using dumps begin to fail, and those who try to pass the exam honestly get an advantage

That is, with this approach, there is still a chance of surrender for those who prepare without dumps, but the level of your knowledge should be very, very good, you should spend a lot of time. In fact, it turns into a sport, you have to really train to pass this exam. And this training is largely redundant.

Example

Such an exam, in my opinion, until recently was CCIE R&S. There are excellent CCIE preparation courses like ine.com . But, even after going through all the preparation (which will take you about six months, completely spent on it), it is still quite difficult to pass the exam. And in parallel with this, at least a few years ago, you could easily find free dumps on the Internet, both written exams and laboratory work.

Consequences

In fact, the consequences of low cost dumps (in the sense discussed above) are rather sad. On the one hand, we have sophisticated tests and labs that are really hard to pass. On the other hand, the obtained certificate does not guarantee an appropriate level of knowledge.

You will be surprised what questions the lucky owners of CCIE certificates do not answer. It is not uncommon for experts to be completely unprepared for an "honest" interview. And it's not that they once knew but forgot. The point is, they never knew it. Why teach, waste time, money, take risks when you can just memorize the answers to questions.

Of course, this leads to a drop in the credibility of expert certifications. Not only and not so much vendors suffer from this, but engineers who received these regalia honestly. There is a leveling of the competitive advantage that the status gained by sweat and blood gave.

I think many of you have seen the following picture: a strong engineer with good practical experience, having carefully studied the recommended literature, goes to take the exam and fails it. He is discouraged. The questions seem irrelevant (and this is often the case), the texts are too large and confusing, often the applicant simply does not fit into the time. Moreover, his colleague, with much less experience and knowledge, passes this test almost 100%. This is a typical result of unfair competition.

Dumps and vendors

Why don't vendors protect the credibility of their certificates? Is it impossible or very expensive?

I am sure that this is possible, and we can see it in the CISSP example.

To do this, you need to increase the cost of the dump. And this can be done in the following ways:

make a random sample of questions from a much larger list, for example, 10,000 questions
the knowledge that the applicant gains through honest preparation must be truly valuable to him (which reduces the overall cost of an honest / fair approach)
the moral-ethical component of the dump price should be high. There must be an unequivocal condemnation of the use of dumps in the community
it is also usually easy to find and disallow resources distributing dumps

According to our model, this should put tests back on track with a fair approach, with adequate complexity and depth of testing.

Unfortunately, from my experience, I see that vendors often limit themselves to what is absolutely free for them, namely, they tell you that you cannot use dumps. Basically, this is not effective. If you are preparing for an exam, then, of course, you are looking on the Internet for training questions, and, frankly, you do not know if some of the questions will coincide with the exam questions or not.

I would also like to note that dumps are not always something illegal, there are often quite official courses in which you are given the same dump, or the difficult points of the exam are explained, but, in any case, this is an insider that has little to do with the real increasing the level of knowledge, and, in my opinion, this is also unfair competition. If the vendor wants to really check and confirm the knowledge of the applicants, then this should be prohibited.

Another good way to increase the quality of your certificate is to simply add an interview.

I am sure that vendors are well aware of the situation. For example, it is easy to see this by the sharp drop in the percentage of passing the test or laboratory work, or, for example, by the number of people who passed a difficult written exam with results close to 100%. The fact that they don't do anything about it suggests that they are quite happy with this situation.

Conclusion

In this article, we looked at several models, which I named

honest approach
fair approach
unfair competition, dumps

You have to understand that these are just models. For example, we believed that there is a certain requirement of complexity, which in reality may not be so or not quite so. Nevertheless, my personal experience of passing exams, the experience of passing and conducting interviews and communicating with other engineers suggests that, at least for some certifications, the conclusions of this article are close to reality.

Each of these approaches imposes specific training requirements. An honest approach requires only a good knowledge of theory and good practical skills. A fair approach also requires specific preparation for the exam.

The case of unfair competition is somewhat more complicated; it includes, for example, a moral and ethical component, which is not easy to assess. But we can still draw some conclusions:

«» ( , , , )
, , , , , ,
. , . , , , . ,

It would seem, why play these games? It is clear that not everyone needs it, but there are several factors that are important:

sometimes certificates are needed to run a business (for example, if you work for an integrator)
the certificate actually increases our likelihood of finding a good job
certification is a great way to force yourself to learn something

For me, the last point has always been the most important, and certificates are pleasant and sometimes useful bonuses!

Certification. Is it worth using dumps and what does Markov chains have to do with it?