Friday, January 27, 2017

AUTOMATION 28 - Some checks are faster than others ... surviving The Pyramid Of Doom

This is part 28 of my automation series - for our story to date, check out the list of articles here.

Technical level: **

All the way back in part 4, I talked about a zoology of automation types.  My lingo has evolved a bit, and I sometimes talk about "the automation spectrum" or yesterday I talked about "the automation toolbox".  Different terms for the same thing.  😉

There are three basic types of automation we've been talking about to date,

  • Unit testing, which occurs on parts of the code itself.  It's really great for checking business rules.
  • API testing, which involves the integration usually of an application layer to a lower layer, but can be used to drive a front end as well.  It works by inserting commands and scenarios for a system to react.
  • GUI testing, which is testing from the user interface.  Entering data, pushing buttons etc.


A decade ago when we were talking about automation within the context of testing, we were almost always thinking about a GUI test tool.  Typically the notoriously unreliable record-and-playback tools.

To really be smart in their use, it helps to not just choose one, but to learn to use them all.

Why?  What's the advantage?



One of the key things is speed!

Try using this online dice roller here.  When you click to roll the dice it,

  • sends a request back to the server
  • the server rolls a dice
  • the server renders a new page
  • your browser has to load up a new page (I know because I'm on a slow connection and can see it loading)
  • the page has to download an image corresponding to the number (it takes a moment for this to load too)
  • done

All told it can take about 2 seconds for a roll.  There is another site which is as slow, and there's an animation which shows a rolling dice, so it really can't go faster.

When we were looking at dice rolling under our unit test, we were able to run about 600,000 dice rolls in the same time.  All that test needed to do was to call the method which generated the random number.  So the round-trip is considerably faster (it helps it doesn't have to travel half way around the world and back for sure).

This is generally what we find in automation - unit testing is several orders of magnitude faster than GUI testing, with API testing somewhere in-between.



In my consultation with different automators, a great piece of advice they've given is all around thinking about testing at the appropriate layer.  This was summed up by a quote "don't test the database with a GUI tool".  A bit like the online dice roller, you have to take a long trip through a lot of layers, which you need to do once in a while, but if there's a lot of test you need to do, you want to move them much closer to the area you're testing.

An example came out last week when we were talking about ways to test a csv file upload.  I showed a method I used, we have line-by-line error messaging for uploads, so I make a file full of friendly uploads.  Then I make one where the first line is good, but line by line I break a rule - whether too many characters here, or a number in a text only field etc.

It makes for a great manual test, and we were thinking about how to automate it, originally as a GUI test.  Then we realised that each line where I'd kept or broke the rules really belonged as a JUnit test - it'd be much faster that way.

Something else which came out when we discussed this further at our company technical test study group was there was another reason to push for more JUnit testing.  It's also far easier to write a JUnit test for complicated data than it is to write a GUI one,

  • In a GUI test you tend to have to create a data item going through a series of processes.  For instance if you needed a suspended customer account, you might have to create it, validate it, then go in as an admin and suspend.
  • In a JUnit test you can just declare an item, and have the ability inside the code to set certain fields.  For instance, "declare a customer object, and set the status to suspended".


This makes for much simpler tests, but it brings challenges.  Developers and testers need to be able to look more at what's in place in terms of unit tests, and to make things visible.  We tend to have lovely dashboard for status of GUI and API tests, but the JUnit checks can be a little less visible to a tester (although you'd hope if a test fails, it won't build, so it'd be always 100% right?).

Beware The Pyramid Of Doom! *

The idea that to take best use of the speed of unit and API tests (which also tend to be more robust as well because data and interface items tend to change much less than GUI elements) you need more of them than GUI tests has been represented in a model you've probably heard of called the automation pyramid which I first read about in Lisa Crispin and Janet Gregories Agile Testing.


Simply put, because they're so much faster and easier, you'll want more unit tests than API tests than GUI tests.

Hopefully you'll see from the above why - faster and simpler!

Lisa and Janet's book was written in 2009, but as we discussed in our study group, go back just a few years around New Zealand, and we pretty much had a test pyramid then ... it just went the other way ...


As we've embraced more agile methodology and particularly the idea of automation as a toolbox over a tool, we're taking huge leaps to embrace this pyramid more in our use of automation.





* Unless you haven't noticed, The Pyramid Of Doom in automation looks somewhat like this ...


44 comments:

  1. Saya ingin berbagi cerita kepada anda bahwa saya IBU HERAWATI seorang TKW dari malaysia dan secara tidak sengaja saya buka internet dan saya melihat komentar IBU DARNA yg dari singapur tentan AKI SYHE MAULANA yg telah membantu dia menjadi sukses dan akhirnya saya juga mencoba menghubungi beliau dan alhamdulillah beliau mau membantu saya untuk memberikan nomor Togel/lottrey toto 4D dr hasil ritual/ghaib melalui jarak jauh dan alhamdulillah itu betul-betul terbukti tembus dan menang RM.230.000 Ringgit ,kini saya kembali indon membeli rumah dan kereta walaupun sy Cuma pembantu rumah tanggah di selangor malaysia, sy sangat berterimakasih kepada AKI SYHE MAULANA dan tidak lupa mengucap syukur kepada ALLAH karna melalui AKI MAULANA saya juga sudah bisa sesukses ini. Jadi kawan2 yg dalam kesusahan jg pernah putus asah, kalau sudah waktunya tuhan pasti kasi jalan asal anda mau berusaha, ini adalah kisah nyata dari seorang TKW, AKI MAULANA adalah guru spiritual terkenal di indonesia. jika anda ingin seperti saya kunjungi situs/website KLIK DISINI RITUAL PESUGIHAN DUNIA GHAIB



    SUCCESS STORY OF HERAWATI MOTHER

    I want to share the story to you that I am HERAWATI MOTHER a TKW from malaysia and accidentally I open internet and I see DARNA MOTHER comment from singapur about AKI SYHE MAULANA who has helped him become successful and finally I also tried to contact him and alhamdulillah he Want to help me to give the number Togel/lottrey toto 4D dr ritual results / unseen and alhamdulillah it really proved translucent and won RM.230.000 Ringgit, now I am back indon buying house and train although sy Just a housemaid at selangor malaysia, sy very thankful To AKI SYHE MAULANA and do not forget to give thanks to ALLAH because through AKI MAULANA I also can be as successful as this.So kawan2 who in distress jg ever break down, when it's time god must be the road of origin you want to try, this is the true story of a migrant worker, AKI MAULANA is a famous spiritual teacher in Indonesia. If you want to like me visit the website CLICK HERE RITUAL GHAIB TEMBUS LOTTREY


    ReplyDelete
  2. Saya ingin berbagi cerita kepada anda bahwa saya IBU HERAWATI seorang TKW dari malaysia dan secara tidak sengaja saya buka internet dan saya melihat komentar IBU DARNA yg dari singapur tentan AKI SYHE MAULANA yg telah membantu dia menjadi sukses dan akhirnya saya juga mencoba menghubungi beliau dan alhamdulillah beliau mau membantu saya untuk memberikan nomor Togel/lottrey toto 4D dr hasil ritual/ghaib melalui jarak jauh dan alhamdulillah itu betul-betul terbukti tembus dan menang RM.230.000 Ringgit ,kini saya kembali indon membeli rumah dan kereta walaupun sy Cuma pembantu rumah tanggah di selangor malaysia, sy sangat berterimakasih kepada AKI SYHE MAULANA dan tidak lupa mengucap syukur kepada ALLAH karna melalui AKI MAULANA saya juga sudah bisa sesukses ini. Jadi kawan2 yg dalam kesusahan jg pernah putus asah, kalau sudah waktunya tuhan pasti kasi jalan asal anda mau berusaha, ini adalah kisah nyata dari seorang TKW, AKI MAULANA adalah guru spiritual terkenal di indonesia. jika anda ingin seperti saya kunjungi situs/website KLIK DISINI RITUAL PESUGIHAN DUNIA GHAIB



    SUCCESS STORY OF HERAWATI MOTHER

    I want to share the story to you that I am HERAWATI MOTHER a TKW from malaysia and accidentally I open internet and I see DARNA MOTHER comment from singapur about AKI SYHE MAULANA who has helped him become successful and finally I also tried to contact him and alhamdulillah he Want to help me to give the number Togel/lottrey toto 4D dr ritual results / unseen and alhamdulillah it really proved translucent and won RM.230.000 Ringgit, now I am back indon buying house and train although sy Just a housemaid at selangor malaysia, sy very thankful To AKI SYHE MAULANA and do not forget to give thanks to ALLAH because through AKI MAULANA I also can be as successful as this.So kawan2 who in distress jg ever break down, when it's time god must be the road of origin you want to try, this is the true story of a migrant worker, AKI MAULANA is a famous spiritual teacher in Indonesia. If you want to like me visit the website CLICK HERE RITUAL GHAIB TEMBUS LOTTREY


    ReplyDelete
  3. Selamat Datang di Blog Nagasaon Datu Bolon Izinkan kami membantu anda semua dengan Angka ritual Kami Kami dengan bantuan Supranatural Bisa menghasilkan Angka Ritual Yang Sangat Mengagumkan Bisa Menerawang Angka Yang Bakal Keluar Untuk Toto Singapore Maupun Hongkong Kami bekerja tiada henti Untuk Bisa menembus Angka yang bakal Keluar..dengan Jaminan 100% gol / Tembus Tapi Ingat Kami Hanya Memberikan Angka Ritual Kami Hanya Kepada Anda Yang Benar-benar dengan sangat Membutuhkan Angka Ritual Kami Kunci Kami Anda Harus OPTIMIS Angka Bakal Tembus Hanya dengan Sebuah Optimis Anda bisa Menang…!!! Apakah anda Termasuk dalam Kategori Ini 1. Di Lilit Hutang 2. Selalu kalah Dalam Bermain Togel 3. Barang berharga Anda udah Habis Buat Judi Togel 4. Anda Udah ke mana 2 tapi tidak menghasilkan Solusi yang tepat Jangan Anda Putus Asa Anda udah berada Di blog yang sangat Tepat Kami akan membantu anda semua dengan Angka Ritual Kami Anda Cukup Mengganti Biaya Ritual Angka Nya SajaJika anda Membutuhkan Angka Ghoib Hasil Ritual Dari DATU BOLON 2D,3D,4D di jamin Tembus 100% Silahkan Hub : Eyang Guru 0852-3162-7257 Adapun Besar Pergantian Biayanya Tergantung Paket Yang Kami Berikan 1. PAKET 2D : Rp.300.000 2. PAKET 3D : Rp.500.000 3. PAKET 4D : Rp.700.000 Semua Biaya Diatas Tidak Sebanding Dengan Angka Yang Kami Berikan Kepada Anda…Tapi Ingat Setelah Anda Succes Dengan Angka Ritual Kami..Sisikan Sedikit Buat Orang Yang Membutuhkan Nya Dan Untuk Bisa Mendapatkan semua Angka Kami Silahkan Anda Selasaikan Paket Yang Ada Diatas Call / Di Nomer : 0852-3162-7257 EYANG GURU – DATU BOLON FORMAT PENDAFTARAN INFO BOCORAN SGP & HK; Kpd Yth!! Bpk/ibu Yang Sedang Dililit Utang Kami Atas Nama Team Togel Yang Terbaik Bersedia Ingin Membantu Anda Dgn Angka GOIB, Hasil RITUAL Sendiri, Cara Bergabung Jadi Member ? KETIK: Nama Anda#Kota Anda#Kabupaten#Togel SGP/HKG# LALU Kirim Ke No Hp Kami : ( 0852-3162-7257

    ReplyDelete
  4. I want to share the story to you that I am a Web Service Provider, Online Casino Play Baccarat Online Casino Via Mobile Phone, IPhone, IPad and Roulette Slots...etc "


    จีคลับ
    goldenslot
    gclub online casino


    PLZ, come & join with us, Thanks for your many tutorials!

    ReplyDelete
  5. Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D








    Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D








    Terimakasih banyak AKI karna melalui jalan togel ini saya sekarang sudah bisa melunasi semua hutang2 orang tua saya bahkan saya juga sudah punya warung makan sendiri hi itu semua berkat bantuan AKI JAYA yang telah membarikan angka 4D nya menang 275 jt kepada saya dan ALHAMDULILLAH berhasil,kini saya sangat bangga pada diri saya sendiri karna melalui jalan togel ini saya sudah bisa membahagiakan orang tua saya..jika anda ingin sukses seperti saya hubungi no hp O85-244-015-689 AKI JAYA,angka ritual AKI JAYA meman selalu tepat dan terbukti..silahkan anda buktikan sendiri. 2D 3D 4D 5D 6D



    ReplyDelete
  6. Looking for
    best selenium training in Chennai, Credo Systemz is the no 1 selenium Training institute in Chennai offering professional selenium course by selenium experts.

    ReplyDelete
  7. The risk is superior.
    จีคลับ The option to open online casino games that will bring you like stepping into a real casino. Many online gambling games are ready to offer the best possible bets and meet every need. With a new generation of online games designed for today's gamers. The number of gamblers who choose to gamble online. A lot of glee of gambling. We are a great online service provider that is ready to give you full enjoyment. Bet on gambling online, bet fully on all kinds of bets. Lots of fun that we have brought a lot of games to you to bet on the mobile phone screen. Get the most out of your site, and get instant access to our site and expertise. Only players have the Internet, they risk the chance to win online every time. We offer fast and agile online betting. It is convenient to gamble on all online gambling games. If you do not want to miss out on the great online gambler that offers its users the money to profit from the game. We provide you with unlimited bets and entertainment to players at any time. บาคาร่า

    ReplyDelete
  8. Entertainment gives you 24 hours with online casinos.
    Ruby888It is another online casino games site that can provide you with a real risky online betting experience. Online horoscope has been broadcast online from the famous gambling source of Cambodia. Casino Poipet Casino Resort Enjoy the fun and full of fun to win. Because of the new type of gambling games, both the easy understanding. The challenge is to deliver on the experience of gambling. Guarantee the fun of a long-established website as well as the recognition of the player who bets. We are the leading online gambling site in Thailand. Realistically open to players all day and all night. There is a team of experts who provide advice. Ask for opinions and ask questions. About Betting Games Anytime Online gambling online that will give players a chance to challenge the game. Moreover, the money spent on the service game. All games are ready to make you rich. The opportunity of online casino gambling that most users wait to use the service every click. IBCbet

    ReplyDelete
  9. Real Play Free Play
    Gclub SlotBetting on Online Casinos is a great way to earn money online. Many new online casinos that will make you gambling. With the kind of risky online games that play comfortably. The opportunity of gambling to win online gambling. With hundreds of online games to choose from. Online gambling games are popular among gamblers around the world. This is because they can bet on the full scale and earn money from the games. The atmosphere of online casino games is more realistic by live casino games online. With sharp images, play without interruption. Giving the best online gamble to the consumer. Not only that, they can find tips and entertainment with other buyers. Entertainment is not a holiday to open the risky online games that meet the needs of customers. If you are interested or are looking for ways to make extra money to customers. Win a game online is a way to choose which can meet all users needs. Holiday Palace

    ReplyDelete
  10. Online gambling has better entertainment.
    Goldclub SlotIt is a great way to start playing online gambling games at the best online gambling sites. Win a chance to win no limit online. The customer can risk the online fortune of our website whenever you want. Online gambling betting opportunities suitable for all types of gamblers. Gold to the joy. Every game is a gamble to be sure that the players are ready to bet. And can meet the needs of online gambling games are open all the time. And online gambling games where you choose to gamble. Online casino games more than 100 games to bring joy. Excited to you constantly. Online casino games and players will be more than entertainment. We offer online casino services that are all so popular and popular. Do not miss the chance to win live online and also promote. Special offer for new members and get up to 500 free credits to gamble. All bets are free to play. คาสิโนออนไลน์

    ReplyDelete
  11. Think of the service at once. Remember to remember our online casino.
    viva9988 Our online casino site. It's a fast-paced web site hosting service. And the speed of live broadcast through the Internet, covering all locations, unlimited. No matter where you are We have developed programs and systems all the time to accommodate more and more modern technologies and to meet the wishes of the players. With no play. You can play on your mobile just by having the internet. No need to waste time downloading the program. In addition, each game is a virtual game developed from casinos in foreign countries. Please join us by signing up to become part of our website. I guarantee that. You will not be bored anymore. And your game is not a normal game. That's your dream. Unless you are having fun then. You also have the opportunity to win prizes from participating games. Come back home We have a Jackpot to distribute each day. From various games You will only be eligible if you join our site. If you are interested in joining us at our website. จีคลับ

    ReplyDelete
  12. The blog are the best that is extremely useful to keep.

    ดูหนัง

    ReplyDelete
  13. The most popular destination for Hadoop Training in

    Noida
    in NOIDA is CIITNOIDA in Sector-2

    You can Master the various components of Hadoop ecosystem like Hadoop, MapReduce Architecture, Pig, Hive, HBase, Sqoop, Flume by their Senior Experts

    from Industry with good Industrial experience of handling big projects.

    You can even Get hands-on practice with live projects and ORACLE certification at CIITNOIDA.

    This Big Data Hadoop Training in Noida is best

    suited for CS & IT Engineering Students, professionals looking to gain expertise in Big Data and work for Fortune 500 Software Companies.

    Hadoop Training in Noida
    Big Data Hadoop Training in Noida

    ReplyDelete
  14. Our digital marketing course in Chennai is targeted at those who are desirous of taking advantage of career opportunities in digital marketing. Join the very best Digital Marketing Course in Chennai. Get trained by an expert who will enrich you with the latest digital trends.

    Digital Marketing Course in Chennai
    Digital Marketing Training in Chennai
    Online Digital Marketing Training
    SEO Training in Chennai
    Digital Marketing Course
    Digital Marketing Training
    Digital Marketing Courses

    ReplyDelete
  15. The Great Info for Thank You!!
    We are an Auckland based digital marketing agency comprised of ad enthusiasts who strive to find the right marketing mix for our clients.

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. As reported by Stanford Medical, It is really the one and ONLY reason women in this country live 10 years longer and weigh an average of 42 pounds less than we do.

    (By the way, it has totally NOTHING to do with genetics or some secret exercise and absolutely EVERYTHING to do with "HOW" they eat.)

    P.S, What I said is "HOW", and not "WHAT"...

    Tap on this link to see if this little test can help you unlock your real weight loss potential

    ReplyDelete
  18. Wow What A Nice And Great Article, Thank You So Much for Giving Us Such a Nice & Helpful Information about Java, keep sending us such informative articles I visit your website on a regular basis.Please refer below if you are looking for best Training Center.
    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  19. Thanks for provide great informatic and looking beautiful blog, really nice required information & the things i never imagined and i would request, wright more blog and blog post like that for us. Thanks you once agian
    we offer services delhi birth certificate which inculde name inclusion in birth certificate and birth certificate correction complete process is online and we offer
    birth certificate apply online and we offer this birth certificate online same service offers at yourdoorstep at birth certificate in ghaziabad our dream to provide birth certificate in india and other staes like birth certificate in bengaluru and birth certificate in gurgaon book service with us birth certificate in noida also, service at yoursdoorstep only birth certificate in india or

    ReplyDelete
  20. Thank you so much for your efforts for this valuable blog, keep it up for more updates. Visit Ogen Infosystem for professional website designing and SEO Services in Delhi, India at an affordable price.
    SEO Service in Delhi

    ReplyDelete
  21. here you can get more information about joining the pakistan navy
    Join Pak Navy

    ReplyDelete
  22. Very much informative, thanks for posting.
    SEO Training In Hyderabad

    ReplyDelete
  23. Nice sharing information regarding automation. I hope you will share something information next time. It's helpful for our life.
    if you needs graphics design with uniqueness & conceptual then visit my site show my design portfolio.

    ReplyDelete
  24. “Airlinespolicy247.com” tells about policies of different airlines under one hood. Passengers usually get confused about Flight Change Policy, Cancellation Policy, Baggage Policy, Check-In Policy, and Pet Policy. We cover all of these policies of different airlines and you will find everything in one place. With us, you enjoy the benefits of getting knowledge of every policy of each airline which you must know before traveling with an airline.
    For more details, visit at https://www.airlinespolicy247.com/
    Southwest Flight Change Policy

    ReplyDelete
  25. Forex Website Seo are a quick and easy way to increase the visibility of your website on the internet. There are many types of backlinks that you can purchase, including forum posts, blog posts, articles, press releases, social media posts, and more. We have the experience and expertise to create the perfect Forex Website Seo for your needs.

    ReplyDelete
  26. Tradingzy Is The Best And Most Reliable Seo Company And Offers A Wide Variety Of Forex Trading Seo Packages , Including An Affordable Monthly Subscription Service, One-time Links, And Blog Comments. With Any Of These Services, You Can Be Assured That Your Site Will Rank Higher In The Search Engine Results Page And Receive An Influx Of Targeted Traffic.

    ReplyDelete
  27. Online Stock Broker is a free service that aims to provide consumers the best online forex broker reviews. It includes unbiased comparisons of a large range of leading brokers and trading platforms, highlighting their strengths and weaknesses.

    ReplyDelete
  28. At Market Forex Net Our Aim Is To Provide You With The Information Necessary To Make An Educated Choice When Finding The Right Trading Service Provider. We List A Variety Of Brokers Offering Many Different Platforms And Services. Gaining Access To A Wide Range Of The Best Online Brokers Will Allow You To Compare Them Side By Side And Make The Right Decision For Your Trading Needs.

    ReplyDelete
  29. Never miss a stock price change from Gmhi Stock with our Live, Real Time Stock Market Overview! Most of the data is updated every 5 minutes to ensure that you are getting all of the latest Acam Stock change alerts as soon as they happen.

    ReplyDelete
  30. Track your stocks in all major markets instantly with our unique live stock overview. See Ecdp stock live prices changes as they occur and view change details, including volume and share changes.

    ReplyDelete
  31. As a new forex trader you need to know what the best brokers offers and their How To Start Forex Broker before you begin trading and investing, Use this guide to learn about trading costs, leverage and leverage limits, spreads and execution times, custodial services and market maker spreads .

    ReplyDelete
  32. Forex trading is one of the fastest growing markets. White Label Forex Meaning will teach you how to trade forex like a pro. It contains some of the best strategies for learning a six-figure income in your spare time with very little financial risk.

    ReplyDelete
  33. "Forex Seo Services will help you increase your business by reaching the maximum number of potential customers through social media, Search Engines and direct advertisement.

    ReplyDelete
  34. Here are our top picks for the Best MT4 Forex Brokers In UAE . We chose only regulated and reputable brokers who have low fees and offer reliable trading platforms to their traders.

    ReplyDelete