Test automation is a powerful tool in the software development process. It allows teams to quickly and accurately check their code for bugs and other issues, saving time and money in the long run. But many businesses are still hesitant to embrace this technology, not sure how it can benefit them or how they can implement it effectively.
This article will share our experiences with implementing our agile test automation process.
Free code review checklist and best practices
DownloadOf course, as much as any other company, we have made mistakes along the way; however, by learning from them and applying best practices to our approach, we hope you can avoid making similar mistakes in your endeavors.
We all recall the thrilling moment when our business scored its first major contract. Years back, we were a humble development team yearning to move beyond mundane tasks and manifest our potential for greatness internationally. With that ambition in mind, we have made considerable strides toward demonstrating excellence in our work and showing the world just how talented we are!
The client was a luxurious jewelry marketplace with high demand. They had recently acquired an impressive investment and wanted to advance the process further. At this point, their existing contractor desired to take another route; they sought out a company that could propel their software toward subsequent rounds of investments.
Although all four of us were highly experienced software engineers, none of us had ever managed a project as large and complex as this one. Nevertheless, Adam was astounded by our courage and character traits, so the decision took no time to make. We were nestled in our recently decorated office in Kyiv, observing a large screen as Adam, located thousands of miles away in California, enthusiastically signed the contract.
Later that evening, Adam called us across the time zones. "Have you done something? My checkout page isn't working," he inquired. And so began our journey of high-quality software development.
The issue appeared trivial. Before departing, Douglas – the former developer – had uploaded his final work to our system. He swiftly remedied the problem, yet we were astonished by an unforeseen revelation. Douglas was compelled to roll out his local changes straight onto the production server without a test and staging environment.
Given that we could not instantly deploy a novel system from our local machine, we resolved to launch servers with continuous integration tools. This automated test suite would streamline the deployment process altogether.
As software engineers, we were well-versed in unit tests and regularly practiced test-driven development. Going through the previous developers' work revealed an abundance of unit test files, which made us acknowledge that running tests in the testing pipeline with each deployment to our servers would be wise.
What happened was expected and unavoidable: many existing unit tests failed. As seasoned software engineers, we've encountered this situation numerous times before.
As Yoda famously proclaimed, "If once you start down the Dark Path, forever will it dominate your destiny." When developers succumb to the pressure of meeting tight deadlines and bypassing minor failing tests, they open the door to an onslaught of issues. The path back from this point is often long and challenging, shrouded in frustration and despair.
The primary aim of a reliable test suite primary aim is to deliver instantaneous feedback every time the code changes. Identifying and resolving problems quickly saves time and conserves resources in the long run.
The most frustrating dilemma we faced was not knowing why a specific unit test failed. Was it something that needed to be fixed that had been overlooked, or were the requirements modified at some point and should no longer exist?
Despite Adam's desire to have some critical features completed by this point, we devoted a couple of weeks to reviving the test suite without having begun any of them. Starting this new contract with Adam's doubts cast a shadow of uncertainty over our future together. To prove that he made the right decision in hiring us, we must demonstrate all of our talents and expertise to reassure him.
Over the next couple of weeks, we made remarkable progress. The old unit tests were fixed, and our team released new features on an accelerated two-day timetable while ensuring that our updates remained consistent.
With every bug we caught in the test and staging environment, we created a new unit test to protect against it. This allowed us to grow overall test coverage effortlessly over time, thus dramatically reducing manual testing efforts.
Adam was delighted with the speed of delivery, which allowed him to tackle business tasks efficiently. Moreover, there were no more issues arising from manual deploys; consequently, his average conversion rate dramatically escalated!
One fateful day, Adam reached out to us in a desperate state, bluntly exclaiming:
"We've been receiving thousands of complaints! Customers are paying their invoices, but the system keeps prompting them to do it again, and we're already facing terrible reviews on review websites. We need this problem solved right away!"
The firewall was improperly configured, prohibiting the checkout system from updating the database and confirming orders. Consequently, customers were still receiving follow-up emails after their purchase was finalized. Luckily, this issue was rapidly addressed; however, it caused great distress for the business involved.
The main issue was within our testing strategy. Unit tests are independent and assess single individual modules, so they're speedy but unable to examine the collaborative communication between various parts of the system.
Unit testing is the foundation of any reliable test strategy, but integration tests and UI tests, also known as end-to-end tests, should also be employed to ensure a comprehensive approach. These additional types of tests allow for detailed diagnostic analysis between individual modules, core business logic, and external components, as well as more thorough testing on the entire system.
Driven by business needs and overconfident in our abilities, we overlooked the crucial integration tests and end-to-end tests. Even though unit testing is usually under the purview of developers, it's often Quality Assurance personnel who take on integration testing duties. This prompted us to approach Adam and implore him to add a skilled QA person to our team for greater assurance that nothing was left unchecked.
We were so lucky to find Andrew, the new QA engineer, on our team. His previous experience testing large-scale enterprise applications made him an invaluable asset. Our process was encompassed by numerous test cases, comprising functional tests, service tests, checklists, security testing, system testing, usability testing, and other concepts which group software tests into categories.
Andrew was our guard against potential bugs, so we felt completely secure. Unfortunately, a new issue arose; since we disregarded integration tests, Andrew only had to conduct manual testing. Later, we realized that this antipattern was called the reversed test pyramid or simply an ice cream cone. The number of automated tests in this type of testing pyramid is limited, so QA engineers must primarily rely on executing test scripts manually.
Andrew often found our bugs due to the rebellious nature of human error. As soon as a mistake was fixed, we had to restart testing again, which took extensive time and led to delays in release dates. It was genuinely agonizing!
In the end, disaster struck, and Christmas Eve was ruined for all of us. Adam had planned to launch a special holiday promotion, but Andrew discovered an annoying bug at the eleventh hour before its release.
However, we knew that if we didn't put in some extra hours now, our work would have gone down the drain after weeks of effort - so there wasn't much choice left for us but to do overtime!
Upon our reflection, it became apparent that Andrew's previous projects had adopted an entirely different framework. Instead of utilizing a testing process embedded within the entire development cycle, they followed a waterfall approach with dedicated months solely for testing.
Additionally, instead of having regular releases throughout the project, they chose to have a single large release at the end. Unfortunately, trying to use that same testing strategy was detrimental to us.
Recognizing the value of swift feedback, we opted to automate our test suite as much as possible. Only exploratory testing cannot be automated for all types of tests.
After we started to write tests, it quickly became evident that automated execution times would skyrocket. The feedback was definitely not instantaneous. Our software can run thousands of unit tests in mere seconds. Still, for integration tests, we need good-quality test data and the ability to connect external services or even execute scripts through a user interface for end-to-end tests.
By devoting a few days to writing tests, we can create a much more reliable test suite that takes hours to run. Andrew valiantly salvaged the situation by proposing a concept of the test automation pyramid. This is an acclaimed technique for structuring test suites. The idea is straightforward: if you can verify a feature with a quick test, don't bother utilizing the slower one.
If the feature can be verified through unit tests, use that instead of integration tests. Only opt for end-to-end tests if necessary because the integration cannot provide sufficient evidence of correctness. If your current automation process is too slow and complicated to be cost-effective, manually executing the script is a worthwhile alternative.
After thoroughly inspecting the agile code review checklist, it was evident that the QA scripts for corner cases practically doubled our unit tests, and specific actions could be completed through API calls rather than UI. It became a momentous victory! The completion time for the entire test suite diminished to less than 20 minutes - an incredible success!
We no longer had to wait days to launch a new release; with such fast execution times, deploying new features several times each day is now feasible.
Adam was overjoyed with the changes we instituted in our process; previously, he had to wait an interminable amount just to run an experiment or release a minor feature. Now he can witness the result in production the following day.
The following Christmas was much brighter, as our new methodology allowed us to move quickly and confidently break things. As a result, we ensured that our software was free of any significant bugs and safely launched on time.
Thanks to his swift action, Adam was able to outpace the competition and secure fresh funds for his venture. He then went on working with us in harmony.
Ultimately, software development has no universal truth; each project has challenges. However, one can always find ways to apply Agile principles and tailor them to its needs. We have learned a great deal from our experience and are confident we can develop more creative solutions in the future.
VT Labs does agile development for Shopify stores, to move faster than competitors. You get store improvements every week and can change priorities anytime.
