Wednesday, June 15, 2016

AUTOMATION 6 - Unit Check, Part 2: random distribution

We're looking at unit checking a Java dice rolling class - see last time for more information.


Previously by necessity we occasionally had to show some technical details, and raw Java code - I'm hoping those of you less technical are still with me.

Today will have less of that - I'll specify parts of code, so those of you experimenting with Eclipse yourselves can keep up and experiment, but it will be less important.

Distribution of numbers



Okay - I've got a test which assures me all the numbers 1-6 come up.  But that doesn't address my major concern, which is I'm worried that the number 1 will come up significantly less than numbers 2, 3, 4, 5 or 6.

How would I manually test this?  Well as before, I'd keep pressing "roll again" and keep a table of the results.  I'd expect them to look evenly spread, 1/6 each.  Ish.

Again - great test, but there's a clause there like above "I'd keep doing until I'm satisfied / dissatisfied".

Automation needs a hard and fast rule.  Welcome to the world of computers - things have to either be a 1 or a 0, a pass or a fail.  And "until I'm satisfied" is not hard or fast.

Now ... there's a something which is often quoted about random numbers.  If you sample over a large enough sample set, then you should see on your dice rolls a perfect split of 1/6 for each number using a 6 sided dice.

When I've manually tested this before (believe me I did statistics at University and I did this one night with my best friend David Farish), and notice that the results do smooth out a bit.  However I'd usually get bored at about 200 dice rolls.

However in my automated unit check, I have the power to sample over 6 million dice rolls.  Over such a huge number, everything should even off to a cool million each.  So I'm going to rename and reuse that code I had yesterday.


@Test public void randomNumberSpread() { System.out.println("Confirming even distribution"); int totalRolls = 6000000; ...

  //Confirm each number rolled 1/6 of the time for(counter = 0; counter < numSides; counter++) { assertTrue("Dice rolls for " + (counter+1) + " not perfect 1/6", (countArray[counter] == (totalRolls/6) ) ); }
}


Let's see how it works shall we?




Oh - that failed.  I thought over such a large number, it'd all even out.  When I ran this, I genuinely thought it might be within 1 or 2 of the target number.

Maybe I need to roll more.  Let's set our total rolls for 600 million, and see what happens?

It takes about 21 seconds to run, but ...

Maybe if I try again?



Well, obviously I've not run enough dice yet, let's set it to 60 billion!!!  And that dear readers is where I break my software completely because


Basically I've pushed the number of rolls so high, that Java can't handle unless I change a lot of the data types I'm using.

I could ... but I'm reminded of the third iron law,


And I'm starting to go down the path of making this check way too complex.  Just adding more dice rolls doesn't seem to help.

Here's the frustrating thing - look at the results from these three runs

Run 1


Run 2


Run 3


In Run 1, the dice rolls for numbers 1, 4 and 6 fall just under the 100 million "perfect" number.  In Run 2, the dice rolls for numbers 1 and 6 are just above 100 million.  Whilst on Run 3, the dice rolls for the number 4 is above 100 million.

Intuitively, using manual judgement, that says to me that I've seen numbers fall above and below the average, but only a little bit.  So that looks okay to me.

I've tried this test several times - sometimes it take 2 goes, sometimes 3, sometimes 4.  Fundamentally though the challenge is "how do I create a hard and fast rule for this to test?".

When I posed this problem on Twitter, Iain Mc pointed out to me


I tried to get a good calculated way to determine an expect deviation for this which led me to Yahoo answers and this ...

d^2 = [1/(N-1)]∑(xi - m)^2 = [1/(N-1)]*{∑[N*vi*P(vi)]^2 - 2*N*m*∑v*P(vi) +N*m^2}

d = √ [1/(N-1)]*{∑[N*vi*P(vi)]^2 - 2*N*m*∑vi*P(vi) +N*m^2}

At which point, I remembered that I've not done "hard maths" for a while, so maybe Iain had a point.

What I've decided to do is to modify the test.  Obviously running 600 million rolls doesn't superbly define the problem for me, and at 21 seconds, takes a bit too long.



What I've decided to do is to test a threshold of plus or minus 1%, which should be good enough.  And I'll do it over 600,000 dice rolls, which will be much faster (couple of seconds including compilation), but still nicely divisible by 6.



@Test public void randomNumberSpread() { System.out.println("Confirming even distribution"); int totalRolls = 600000; int minVal, maxVal;
...

//Work out min/max threshold as +/- 1%
minVal = (int) (0.99 * totalRolls / 6);
maxVal = (int) (1.01 * totalRolls / 6);

   //Confirm each number rolled 1/6 of the time for(counter = 0; counter < numSides; counter++) { assertTrue("Dice rolls for " + (counter+1) + " below threshold of " + minVal, (countArray[counter] > minVal ) ); assertTrue("Dice rolls for " + (counter+1) + " above threshold of " + maxVal, (countArray[counter] < maxVal ) ); }
}

And that seems to run nicely ... and so far repeatedly ...


Okay - that's a lot of maths and thinking.  We still have a little more to play with before we sum up, for now, we'll wrap up.

Recap

So we managed to create an important automated check - but it was a little harder than initially expected, because as the first iron law said, we needed to summarise what we wanted to check into a hard pass/fail criteria.  And that turned out to be harder than we intuitively first assumed.



Extension material

Have a think about what other tests you think might be worth running.  If you're using Eclipse, have a go with it.

But also try and think about problems with this for you as a tester.

2 comments:

  1. material handling equipments | Automated Storage & Retrieval System | Conveyors | Automated Guided Vehicles | Warehouse Management System | SCADA Supervision Software

    ReplyDelete
  2. Ass Wr Wb Saya ingin berbagi cerita kepada anda bahwa sy seorang TKW dari malaysia dan secara tidak sengaja saya buka internet dan saya melihat komentar IBU LUSI yg dari singapura tentang AKI yg telah membantu dia menjadi sukses dan akhirnya saya juga mencoba menghubungi beliau dan alhamdulillah beliau mau membantu saya untuk memberikan nomor Togel/lottrey 4D dr hasil ritual/ghaib dan alhamdulillah itu betul-betul terbukti tembus dan menang RM.270.000 Ringgit ,kini saya kembali indon membeli rumah dan kereta walaupun sy cuma pembantu rumah tanggah di selangor malaysia, sy sangat berterimakasih kepada AKI MAULANA dan tidak lupa mengucap syukur kepada ALLAH karna melalui AKI saya juga sudah bisa se sukses ini, pesan AKI yg slalu sy ingat setiap manusia bisa menjadi kaya, hanya saja terkadang mereka tidak tahu atau salah jalan, Banyak orang menganggap bahwa miskin dan kaya merupakan bagian dari takdir, Takdir macam apa? Tuhan tidak akan memberikan takdir yang buruk terhadap kita semua, cobaan yang Tuhan berikan merupakan pembuktian seberapa kuat Anda bertahan di dalamnya. Tuhan tidak akan merubah nasib Anda jika Anda tidak berusaha untuk merubahnya sendiri, Jadi teman2 yg dalam ke susahan jgn pernah putus asah, kalau sudah waktunya tuhan pasti kasi jalan asal anda mau berusaha. AKI MAULANA adalah guru spiritual terkenal di indonesia yg bisa melakukan ritual ghaib seperti:
    1.Pesugihan bank ghaib
    2.Ritual tembus togel/lottrey
    3.Transfer janin
    4.Pelaris usaha, jodoh DLL
    jika anda ingin mengubah nasib seperti saya silahkan KLIK DISINI PESUGIHAN DUNIA GHAIB



    THE STORY OF SUCCESS MOTHER RATNAH PUSPITA

    I want to share a story to you that sy a migrant worker from malaysia and accidentally I open internet and I see the comment LUSI mom from singapore about AKI who has helped him become successful and finally I also tried to contact him and alhamdulillah he wanted Help me to give the number Togel / lottrey 4D dr ritual / unseen and alhamdulillah it really proved translucent and won RM.270.000 Ringgit, now I am back indon buying house and train even though sy just housemaid in selangor malaysia, sy very thankful To AKI MAULANA and do not forget to give thanks to ALLAH because through AKI I also can be this successful, AKI message which slalu sy remember every human being can become rich, only sometimes they do not know or wrong way, Many people assume that poor and rich Is a part of destiny, what kind of fate? God will not give a bad destiny to us all, God's temptation is proving how strongly you endure it.God will not change your fate if you do not try to change it yourself, So teman2 deep into tusah jgn never despair, when it's time god must be the road to where you want to try.AKI MAULANA is a famous spiritual teacher in Indonesia who can perform the occult rituals such as:
    1.Presugihan bank ghaib
    2.Ritual translucent togel / lottrey
    3.Transfer fetus
    4.Paris business, matching DLL
     If you want to change the fate Please visit the website CLICK HERE WORLD WIDE GHAIB




    ReplyDelete