August 30th, 2017 by Craig Schwarzwald

Flakiness of Corporate Selenium Suites and how to get rid of it


If you follow the principles in this article and utilize the open source framework I refer to as a base to your selenium suites, you will find your tests to be STABLE, MAINTAINABLE, and EASY TO READ / CREATE, throughout your entire enterprise! These 3 key pillars are something that every company strives for in their automation, but it seems like few actually achieve. WIth the patterns I lay out below and the open source framework as a starting point, you can get a leg up on all your competitors in the test automation space!

Background:


Flaky tests are the number one cause of death to automation efforts in large (or small) corporations. Usually, team members and management get fed up with the unreliability of the test results, and see the maintenance costs rise to uncontrollable levels until the entire initiative goes under. Also, other issues besides true Selenium Flakiness tend to all get lumped together under this general concept and have given Selenium the tenuous reputation it seems to currently have in the industry today. For this reason, let’s first define selenium flakiness as well as what will be covered in this article vs. out of scope for this article.

A flaky selenium test simply means that the test is indeterministic, or in other words will fail or pass seemingly at random thus giving the tester absolutely no confidence in either correct behavior of the production code or significant proof there is a defect. A true flaky selenium test will do something like pass on the first run, then fail on the next, then pass again on the third run, all while executing the same exact flow on the same exact environment/region/setup/config/etc. In many ways a flaky Selenium test can actually be worse than no test at all as it gives about the same information to the tester, only now the test engineer must spend hours or days to debug the existing flaky test.

One more criticism I’ve heard a lot that tends to get lumped into selenium flakiness is that when any tiny change is made to the production application, it causes hours or days (or more) worth of work to fix every test throughout the entire corporate selenium suite because they’re so “flaky”. This is really a script maintenance issue, and not an issue of true flakiness, however this issue will also be addressed in this article.

Out of scope for this article will be region availability or region inconsistency issues. Since those types of issues would be too specific to each individual corporation. However, we will go over some techniques that will help with small variations in network traffic delays within this article as they relate to flakiness.


Getting rid of Selenium flakiness, in a corporate setting (or elsewhere), really comes down to two key principles. First, use explicit waits as your waiting strategy any time you need to wait for anything. Second, make sure your using appropriate patterns to make your tests easily maintainable. Throughout this article, I’ll be referring to an open source framework I’ve started that helps you do both. [link to Vanguard GitHub when available]

Note: All examples in this article will be in Java, but the concepts apply to any of the Selenium language bindings.


Stability:


Defining The Different Wait Strategies

The 3 ways you can wait with Selenium:
Hard Wait: ex: Thread.sleep() – Just Don’t. Never!
Implicit Waits: ex: driver.manage().timeouts.implicitlyWait(3, TimeUnit.SECONDS); – Bad. Can be extremely bad if ever used along with explicit waits
Explicit Waits: ex:
WebDriverWait wait = new WebDriverWait(driver, 3); wait.until(ExpectedConditions.elementToBeClickable(locator));
– This is what you want to do all the time.

Nearly ALL Selenium Flakiness stems from not waiting the right way for the right amount of time somewhere in the test script. Let’s explore each of these 3 options and go over their pros and cons as we explore the following common example situation:

The Example

I enter a value into a form where the page runs some validation on that input and then enables a continue button if the input value was valid. It typically takes ¼ second for the continue button to become enabled, 30% of the time it takes ½ second, but on 10% of the time can take 2 seconds (or very occasionally more) due to bad sporadic latency.


The Hard Wait

This is usually the first tool relatively inexperienced Selenium developers go to in their typically limited tool belt. It will simply always wait for the entire time specified in the Hard Wait call. From our example, we can just tell the script to hard wait for 2 seconds every time and problem solved, right?

Hard Wait Pros:
  • It’s the simplest option to understand, and will usually give the engineer the result they’re looking for on at least the next couple of runs so they can mark the script as “fixed”. Even though it’s not really fully resolved (see Con section)
  • Hard Wait Cons:
  • The script is always going to wait a full (in this case) 2 seconds! It’s going to wait all that time even if the continue button becomes enabled in ¼ second for a particular run. Waiting an extra 1 ¾ seconds for an operation might not sound like a big deal, but if you multiply that by 5 or 6 similar type operations within that same test, then multiply that by hundreds or even thousands of tests across your organization, and your enterprise test suite is now taking hours (or more) longer than they should. That time really adds up.
  • Some flakiness is still likely to occur within our example. On rare occasions the continue button actually take more than 2 seconds to become enabled and so the script will still fail in those rare cases with this solution. This approach is similar to addressing the symptom rather than addressing the true issue. The quick fix of course is to up the hard wait to 3 seconds, or maybe 4 or 5, but now you’re wasting even more time on the average run. This also leads to a slippery slope where you have to use the maximum hard wait it might EVER take, and then continue to waste that added wait time in nearly all runs of the test.

  • Implicit Waits

    This strategy takes nearly any selenium command, and if it’s not successful, continues to try again for up to the amount of time the implicit wait is set to. This means if the implicit wait is set to 2 seconds, it will continue the script in our example as soon as it sees the continue button is enabled, be that after ¼ second, ½ second, or 2 full seconds depending on the run, instead of waiting the full time every time like the hard wait. On its face, that sounds like a great solution, and it’s also the next easiest wait strategy to implement. This is because it’s usually done once at the beginning of each test, or set once at a framework level applying to all tests in your entire corporate suite. In theory engineers never have to worry about any wait strategies throughout their tests because the implicit wait that’s already set should take care of everything. In theory at least…

    Implicit Waits Pros:
  • Will continue once condition is met instead of waiting the entire wait time every time.
  • Gets applied to ALL waits within the test(s) with a single command.
  • Implicit Waits Cons:
  • The same pro of getting applied to ALL waits can also be a con as it gives us a great deal of coupling things we may not actually want to be coupled together. What if there’s another point within our test that requires a database update and a batch program to run before results are returned which can take up to 30 seconds? Now we have to up our implicit wait to 30 seconds or more, and remember all implicit waits are shared whether we want them to be or not. Now when we run our test with the 30 second implicit wait, if there’s an issue with the continue button, instead of knowing that after 2 seconds, we’re going to wait a full 30 seconds before the test fails alerting us to our issue because of other waits that are needed in our test (which should be unrelated).
  • Implicit waits can be set anywhere in the script, and there’s no current way to know what value it’s set to. This means it can be virtually impossible to know what the implicit value is set to when a particular line is being executed. Upping the implicit wait value in the one place (you know of where) it’s set might not actually do what you want if there happens to be another implicit wait call that’s made anywhere else. Because each implicit wait call will completely override whatever the value was previously set to, this can make modifying the implicit wait on the 1 line you care about (like the enabling of our continue button) difficult to do sometimes. One possible solution is to constantly reset the implicit wait wherever any wait is needed, but this just compounds this issue, and makes the pro we had listed above about only needing to set the implicit wait once for all tests no longer applicable.(Note: In the W3C spec they are adding a method to return the current implicit wait value, in part due to complaints from myself and other known colleagues).
  • Lastly, using implicit waits do NOT mix with explicit waits. If you use explicit waits ANYWHERE, you should never set an implicit wait before that or the explicit wait will not function correctly. The selenium documentation states emphatically to not mix the two strategies. For an in-depth answer as to why, see this StackOverflow link

  • Explicit Waits

    This strategy is similar to implicit waits in that it will only wait up to the amount of time specified (unlike hard waits), but unlike implicit waits, applies to only a single selenium command. While this is the most difficult wait strategy to employ, it has none of the downsides that the other wait strategies have. Let’s explore how Explicit Waits work and their benefits, and then we can go into how we can make them easier to call within our scripts to mitigate their only flaw.

    Explicit waits take a time, and an ExpectedCondition to satisfy. Selenium will continue trying to satisfy that condition until it’s satisfied (and will then continue the script immediately), or until the time specified is hit (and will then throw a TimeOutException that should be caught and dealt with).

    Explicit Wait Pros:
  • The script will continue as soon as the condition is met, even if the wait time was set to something longer.
  • If other places in the script also need to wait, they can use their own Explicit Wait that is completely decoupled from each other as they should be.
  • If changes are needed to the wait time for any one Explicit Wait, they are easy and straightforward to make because the Explicit Wait call applies only to that action.
  • Explicit Wait Cons:
  • This is the most difficult wait strategy to employ. Each call requires the user to choose an Explicit Wait from a large list and know which is best in each situation. Also a specific max wait time must be specified for each and every command you might need to wait for.

  • Now that we’ve gone over what the different wait strategies are in Selenium, and which one we should be using (always Explicit Waits), what can we do to mitigate the con of that choice? It turns out, we can actually mitigate the difficulty of using Explicit Waits to almost nothing if we set up the right base framework. Like the one here: [link to Vanguard GitHub when available]


    Maintainability:


    Before we get into the base framework, there are some basic Selenium topics we must discuss first. It should come as no surprise to any that has ever done any research around writing Selenium scripts that one of the best approaches to giving us maintainable Selenium test suites is to use the Page Object pattern.


    Page Objects:


    Page Objects have been around and talked about by countless industry experts for over a decade, perhaps one of the best examples being Martin Fowler’s blog post. In short a Page Object (in Java) is represented as a class object containing all elements you might interact with, and action methods for how a user would interact with those elements. As Martin Fowler points out, a Page Object may represent an entire web page, or a section/tab/component of an entire page. Finding the balance of when to create a new PageObject and when to simply add to an existing one can be difficult and better decisions will come in time with more experience. In general, if your Page Object class (or any class for that matter) is more than 200 lines, you may want to start thinking about how to break that down.

    Note: 200 lines is relatively arbitrary. The point is we don’t want our classes to get “too big”. Best practice is to set a level for your corporation that makes sense for your organization. Then use a static code analysis tool to fail any build if any class gets “too big” or “too complex”.

    Creating Page Objects for all your user interactions across your entire corporate Selenium test suite will solve most of your maintenance issues all by itself. Let’s go back to our definition of flakiness and the brittleness factor that typically gets lumped into the ‘flaky’ category.

    Another Example

    Let’s suppose we now have an example where you have thousands of scripts within your organization and something changes within the login portion of your production app. It’s likely that all (or at least most) of your thousands of tests will be impacted. Without a Page Object model, it may require going into each and every test, one by one, to correct them which is where this ‘flaky’ Selenium reputation comes from. However, if you employed a Page Object pattern for all user actions, all of your thousands of tests would be calling the same logon method from a common Logon Page Object. In that latter scenario, you simply have to make 1 change to that commonly used Page Object’s method, and all of your thousands of tests are then instantly fixed!

    So Page Objects completely solve any maintenance issues you may have with your own corporate application updates, but what about potential updates to the Selenium framework itself, or fixing the issue we wanted to address of making Explicit Wait calls easier for engineers writing our Selenium tests? It turns out we can do all that and more with our base framework by extending a Base Page Object. We can further encapsulate the Selenium commands, like the Explicit Wait and others, within our framework so that our test engineers don’t need to worry about them.

    Let’s examine a small sample of some of these encapsulation methods within our framework now.


    Framework Encapsulation Methods:

    From SeleniumElementFinder:
    public WebElement getElement(final By locator, int maxTimeInSec) {
    	WebElement returnElement;
    	if(maxTimeInSec < 0 || maxTimeInSec > ABSOLUTE_MAX_WAIT_TIME_SECONDS){
    		maxTimeInSec = ABSOLUTE_MAX_WAIT_TIME_SECONDS;
    	}
    	WebDriverWait wait = new WebDriverWait(driver, maxTimeInSec);
    	try {
    		wait.until(ExpectedConditions.elementToBeClickable(locator));
    		returnElement = driver.findElement(locator);
    	} catch (TimeoutException e) {
    		returnElement = null;
    	}
    	return returnElement;
    }

    First we do some simple validation checks if the maxTimeInSeconds (the Explicit Wait time we want to use) needs to change based on prior properties set within the project from what was passed in. Next we set up our Explicit Wait call and either return the WebElement if one is found to be clickable (or intractable) within the time specified, or a null object if no valid element was found after looking for the full waitTime. Notice how this method completely abstracts the use of the Explicit Wait from the engineer who will use it within their corporate Page Object methods. Now the engineer simply calls into finder.getElement() and pass a locator and a time to wait and the framework will do all the complicated Selenium logic of the Explicit wait for them.


    From SeleniumActionMethods:
    public void type(final String text, final By locator, final int maxTimeInSec) {
    	WebElement element = finder.getElement(locator, maxTimeInSec);
    	if(element!=null){
    		element.sendKeys(text);
    	} else {
    		throwOrLogError(locator.toString(), getMethodNameFromStackTrace(0));
    	}
    }
    
    public void click(By locator, int maxTimeInSec) {
    	WebElement element = finder.getElement(locator, maxTimeInSec);
    	if(element!=null){
    		element.click();
    	} else {
    		throwOrLogError(locator.toString(), getMethodNameFromStackTrace(0));
    	}
    }

    Expanding on our finder method, we can create actions methods that the engineer can utilize in creating their own user action methods within your corporate PageObjects. Above we see the framework code for type() and click() methods. In both, we can see we are utilizing our findElement() method to see if the element we’re looking to interact with exists. Then if it does, we take the appropriate Selenium action on it. And if not, we log an error to the console, or throw the error which will outright fail the test, based on project properties.

    Let’s now explore more of the framework code so we get a better idea how these simple methods above are used eventually in your corporate PageObjects. First, at the core of our framework, we have the SeleniumPageObject


    Selenium Page Object:


    SeleniumPageObject
    public abstract class SeleniumPageObject {
    	protected WebDriver driver;
    	protected SeleniumElementFinder finder;
    	protected SeleniumActionMethods actions;
    	protected SeleniumBrowserMethods browser;
    	protected TableUtilities tableHelper;
    	
    	public SeleniumPageObject(WebDriver driver) {
    		super();
    		this.driver = driver;
    		this.finder = new SeleniumElementFinder(driver);
    		this.actions = new SeleniumActionMethods(driver, finder, this.getClass().getSimpleName());
    		this.browser = new SeleniumBrowserMethods(driver);
    		this.tableHelper = new TableUtilities(driver, finder, actions);
    	}
    
    	public abstract boolean isLoaded();
    }

    This root object is quite simple. It implements the AutomationPageInterface which just means that any/all SeleniumPageObjects must declare a method for isLoaded() which will signify to your test whether or not the browser is currently on the page you are expecting it to be on. Next any/all SeleniumPageObjects have access to finder, actions, browser, and tableHelper objects. Each of these contain the Selenium commands respective to their categories which we will utilize in our creation of our user action methods within our corporate Page Objects. We’ve already seen some examples of the finder and actions objects above.

    From here, we can look at our SeleniumBasePage which extends our SeleniumPageObject above. This is the class that nearly all of your corporate Page Objects should extend from.


    Selenium Base Page:


    SeleniumBasePage
    public abstract class SeleniumBasePage extends SeleniumPageObject{	
    	public SeleniumBasePage(WebDriver driver) {
    		super(driver);
    		if(!isLoaded()) {
    			throw new SeleniumNavigationException("Page " + this.getClass().getSimpleName() + 
    				" was not loaded in the browser when trying to construct the object.", driver);
    		}
    	}
    }

    This abstract (or partial) class simply has some logic in the constructor. Namely, whenever we create a new object that extends from the type of SeleniumBasePage (remember this will be nearly all of your corporate Page Objects), we will verify that the browser is on the page we expect at the time of the PageObject’s creation. If the page was determined to be loaded then we continue with the test, and if we determine the page “not isLoaded()” then we throw an error that stops the test, and takes a screenshot of what was loaded so we can better debug what went wrong.

    Let’s review an example of how we would create corporate Page Object using our framework as a base.


    Corporate Page Object Example:
    public class MyFormPage extends SeleniumBasePage{
    	private static final By mainform = By.id("myForm");
    	private static final By nameField = By.id("myForm:name");
    	private static final By addressLine1Field = By.id("myForm:addr1");
    	private static final By addressLine2Field = By.id("myForm:addr2");
    	private static final By phoneField = By.id("myForm:phone");
    	private static final By submitButton = By.id("myForm:submit");
    	private static final By errorField = By.cssClass("highlight-error");
    	public MyFormPage(WebDriver driver) {
    		super(driver);
    	}
    	public boolean isLoaded() {
    		return actions.isDisplayed(mainform, 5);
    	}
    	
    	public void fillOutForm(String name, String addr1, String addr2, String phone) {
    		actions.type(name, nameField, 3);
    		actions.type(addr1, addressLine1Field);
    		actions.type(addr2, addressLine2Field);
    		actions.type(phone, phoneField);
    	}
    	public NextPage submitForm() {
    		actions.click(submitButton);
    		return new NextPage(driver);
    	}
    	public void submitInvalidForm() {
    		actions.click(submitButton);
    	}
    	public boolean hasAnyFieldsCurrentlyInError() {
    		List<WebElement> allErrorElements = finder.getElements(errorField, 3);
    		return (allErrorElements.size() > 0);
    	}
    }

    There are several rules to follow when creating each corporate Page Object. These rules are designed to give readability and thus maintainability of the Page Object class.

    1. The Page Object should extend SeleniumBasePage. This is to ensure we gain all the benefits that have been implemented for us from that framework class.
    2. Alternatively, you may want to create a CorporateBasePage which extends SeleniumBasePage, and then have each of your PageObjects within your corporation extend that. One key benefit would be to allow you to create corporate specific methods that could be written once and then shared.
    3. The first lines in your Page Object class should define all the elements this PageObject might interact with. Coming up with the best locators to use for each of these elements is again something that generally comes with experience. As a rule, you want locators that are (a) Unique, (b) Descriptive, and (c) Unlikely to change.
    4. Next we have the constructor for the page. Notice how we simply call super() which calls the constructor for SeleniumBasePage which does our logic of determining if the browser currently has this PageObject loaded or not.
    5. After the constructor, each corporate Page Object must implement an isLoaded() method. In most cases, the implementation should look similar to our example. Pick an element that is unique and significant to the Page Object and make sure it’s displayed within a reasonable amount of time.
    6. The last thing within each corporate Page Object are the action methods the user can take on that page. In our example above of MyFormPage, there are several things the user can do on the page.
      • fillOutForm – Can take in string values for each of the values we want to populate into the form, and types them into the correct fields.
      • submitForm – Will click the submit button on the form and take you to the NextPage. Notice that if NextPage extends the SeleniumBasePage as it should, then by calling the constructor at the end of this method like we do will ensure that the NextPage isLoaded(), or we will get an exception thrown that will fail the test at this point and take a screenshot of what was in the browser so we can debug what went wrong.
      • submitInvalidForm – Will also click the submit button, but will not attempt to create a NextPage object because by calling this method we are assuming the form has invalid input and will stay on this page, highlighting the fields that are in error.
      • hasAnyFieldsCurrentlyInError – This is a predicate method that returns some state information about the page. In this case if there are any fields that match the locator that was set for errorField.

    Notice how some of our method calls off of actions and finder have integer parameters at the end, and some do not. The ones that do are exercising explicit waits from the framework waiting up to that wait time (in seconds). The ones that do not have numbers at the end are calling overloaded methods that do the same functionality, but have a “default wait time” of immediately.

    Example Corporate Test Case code snippit
    @Test
    public void testValidFormSubmission() {
    	MyFormPage myFormPage = ...//Get to the page somehow
    	myFormPage.fillOutForm("Bob", "123 Main St", "Apt. 12", "123-4567");
    	NextPage nextPage = myFormPage.submitForm();
    	...//Continue Test
    }
    @Test
    public void testInvalidFormSubmission() {
    	MyFormPage myFormPage = ...//Get to the page somehow
    	myFormPage.fillOutForm("Bob", "123 Main St", "Apt. 12", "abc-defg");
    	myFormPage.submitInvalidForm();
    	Assert.assertTrue(myFormPage.hasAnyFieldsCurrentlyInError());
    }

    Notice how are tests are extremely clean, simply making calls to our Page Object. Now that we’ve gone over how to make our test Stable and Maintainable by using Page Objects, and extending the SeleniumBasePageObject from our tests, let’s dive into our last goal of making our tests extremely Easy To Read/Create.


    Rules to make tests Easy to Ready / Create:


    Page Objects:
  • Contain all locators.
  • Contain methods for any user actions a person can do on the corresponding page / section of page. They use only the locators they know about to implement those user actions within Page Object action methods. Only methods off the framework actions, browser, finder, or tableHelper objects (or CorporateBasePage methods/objects if you’ve implemented that) should be used within these Page Object user action method implementations.
  • Contain state/predicate methods to return information to a test case about the current conditions of the page.
  • NOTE: Page Objects should never assert anything themselves, that is the role of a test.

    Tests:
  • Perform a assertions.
  • Call Page Object action method to progress the test.
  • (Assign variables to make more readable) – This is strictly for easier readability. You could just as easily perform all actions inline and not assign any variables, but the code would be harder to read.
  • NOTE: Tests should never directly reference a locator or perform any type of user action outside of calling a Page Object action method.

    NOTE: Nowhere in either Page Objects, nor Tests, do we specify performing Selenium commands (those are done at the framework level only).


    Putting it all together


    Let’s explore a new example of an amazon order like application. The following are example page objects and tests to help illustrate everything we’ve gone over in this article.

    ShoppingHome Page Object
    public class ShoppingHome extends SeleniumStartPage{
    	private static final By searchBar = By.id("searchInpt");
    	private static final By itemList = By.id("filterItms");
    	private static final By purchaseButton = By.id("submitBtn");
    	
    	public ShoppingHome(WebDriver driver) {
    		super(driver);
    	}
    	public boolean isLoaded() {
    		return actions.isDisplayed(searchBar, 3);
    	}
    	
    	@Override
    	public String getPageUrl() {
    		return "www.myShoppingExample.com";
    	}
    	
    	public BuyItemPage buyAnItem(String itemName) {
    		actions.type(itemName, searchBar);
    		actions.click(itemList, 3);
    		actions.click(purchaseButton, 3);
    		return new BuyItemPage(driver);
    	}
    }

    Notice this particular Page Object is special. Instead of extending SeleniumBasePage like most of our Page Objects, we’er instead extending SeleniumStartPage. These two objects are very similar, with the only differences being that StartPages don’t perform our logic check for isLoaded() when the page object is created, and that any/every StartPage must implement a getPageUrl() method. The SeleniumStartPage in our framework implements a navigateToPage() method which will drive the browser to the url specified in getPageUrl() and verify the page isLoaded() after that. This means our ShoppingHome PageObject gets that method implementation for free even though you can’t see it in the code above because that functionality comes from the framework within the SeleniumStartPage.


    BuyItem Page Object
    public class BuyItemPage extends SeleniumBasePage{
    	private static final By item = By.id("purchaseItm");
    	private static final By amount = By.id("purchaseAmnt");
    	private static final By submitButton = By.id("submitPurchase");
    	
    	public BuyItemPage(WebDriver driver) {
    		super(driver);
    	}
    	public boolean isLoaded() {
    		return actions.isDisplayed(item, 3);
    	}
    	
    	public boolean isItemCorrect(String expectedItem) {
    		return expectedItem.equals(actions.getText(item));
    	}
    	public boolean isAmountCorrect(String expectedAmount) {
    		return expectedAmount.equals(actions.getText(amount));
    	}
    	public ConfirmPage submitPurchase() {
    		actions.click(submitButton);
    		return new ConfirmPage(driver);
    	}
    }

    Confirmation Page Object
    public class ConfirmPage extends SeleniumBasePage{
    	private static final By item = By.id("purchaseItm");
    	private static final By amount = By.id("purchaseAmnt");
    	private static final By confirmNumber = By.id("orderNum");
    	
    	public ConfirmPage(WebDriver driver) {
    		super(driver);
    	}
    	public boolean isLoaded() {
    		return actions.isDisplayed(confirmNumber, 5);
    	}
    	
    	public boolean isItemCorrect(String expectedItem) {
    		return expectedItem.equals(actions.getText(item));
    	}
    	public boolean isAmountCorrect(String expectedAmount) {
    		return expectedAmount.equals(actions.getText(amount));
    	}
    	public boolean isConfirmNumberPresent() {
    		return (actions.getText(confirmNumber).length() > 0);
    	}
    	public void logConfirmationNumber() {
    		//Simulate user "writing the confirm number down" for potential later use
    		//Log to DB, or Spreadsheet, etc., for example just log to console
    		System.out.println(actions.getText(confirmNumber));
    	}
    	
    }

    Notice how most page’s isLoaded() methods are set to look for only 3 seconds, but the Confirmation page we look for up to 5 seconds because the transaction has to process first which might make it take longer than standard pages to load. Now that we see our sample Page Objects, what would our corporate tests look like?


    Framework SeleniumBaseTest
    public class SeleniumBaseTest {
    	protected WebDriver driver;
    	
    	@Before
    	public void baseTestSetup() {
    		System.setProperty("webdriver.gecko.driver", "/location/of/geckodriver");
    		driver = new FirefoxDriver();
    	}
    	@After
    	public void baseTestTeardown() {
    		if(driver != null) {
    			driver.quit();
    		}
    	}
    }

    First, we can utilize our framework’s base test class to ensure we have an instance of WebDriver, and that each test will launch and close a Firefox browser. This way we don’t need to worry about those things with each and every corporate test class we create.
    Note: Just like best practice could be to implement a corporate Base Page Object, creating a corporate Base Test class can also allow you to set up standards in a single place to be implemented throughout your organization. For instance connecting to a selenium grid (or 3rd party grid provider) instead of launching a Firefox browser at the beginning of every test.

    Example Test
    public class ExampleTest extends SeleniumBaseTest{
    
    	ShoppingHome shoppingHome;
    	
    	@Before
    	public void setup() {
    		shoppingHome = new ShoppingHome(driver);
    		shoppingHome.navigateToPage();
    	}
    	@Test
    	public void testPurchasingAnAppleFor89Cents() {
    		BuyItemPage buyItemPage = shoppingHome.buyAnItem("apple");
    		Assert.assertTrue(buyItemPage.isItemCorrect("apple"));
    		Assert.assertTrue(buyItemPage.isAmountCorrect("0.89"));
    		ConfirmPage confirmPage = buyItemPage.submitPurchase();
    		Assert.assertTrue(confirmPage.isItemCorrect("apple"));
    		Assert.assertTrue(confirmPage.isAmountCorrect("0.89"));
    		Assert.assertTrue(confirmPage.isConfirmNumberPresent());
    		confirmPage.logConfirmationNumber();
    	}
    }

    Notice how our test is extremely easy to read. There are no selenium commands anywhere, and we’re only making calls to, or assertions on Page Object methods. Note, most test classes should start with creating an object of a SeleniumStartPage, and call navigateToPage() on it within an @Before like we’ve done above. This guarantees that as we start each and every @Test in the class we already have a browser open, and it’s already been navigated to the application under test. We can keep the test succinct to only the actions we need to take in order to test on our application.


    BONUS

    In addition to making our test suites STABLE, MAINTAINABLE, and EASY TO READ/CREATE, this framework offers numerous significant benefits even beyond these main pillars we were after.
    1. We future proof our corporate test suites against potential updates to the selenium framework itself. Because of our added encapsulation, (notice there were no selenium commands in our example Page Objects or Tests). This means if the Selenium development team decides to modify how to call WebDriver.get() or WebElement.isDisplayed() commands, we would only require a single change to a common method within our framework and no changes to any corporate Page Objects or Tests throughout our entire organization would be required! For anyone thinking this sounds nice but would never come into play, think back to Selenium RC (or Selenium 1.0) days. Selenium WebDriver (2.0) was a complete re-write of the API and everything changed. Google has been a big supporter of Selenium, even back in the RC days, and had thousands of RC test cases which they spent millions of developer hours converting to the new WebDriver API. Had they used a framework like this (if one was around then), all that work could have been avoided.
    2. Similar to #1, not only do we future proof against future Selenium API changes, but we even allow ourselves to move to a completely different tool/technology with again virtually no changes to our corporate Page Objects or test cases! We could replace the WebDriver object and all selenium implementations with a completely different tool if we wanted to. We could simply update the framework to use Tool X implementations instead of Selenium implementations, and all of our tests should continue to work just as they had before completely oblivious to the underlying tool/technology change.
    3. We can give all corporate Page Objects some base functionality like getting a cell of a table containing text we’re looking for. Any method that would be useful for all corporations to have can be coded once in our open source framework and shared, so more functionality can be continuously added to your corporate Page Objects over time as well.
    4. It gives a unique opportunity for corporations to implement any checks or validations they might want to add in a single place where those checks are guaranteed to be run across every command in every script throughout the enterprise when using a Corporate Base Page Object. It is also a perfect place to throw any custom logging, etc. that might make debugging tests easier throughout your organization.
    5. Some of the most common and difficult to fix Selenium errors like NoSuchElementException and StaleElementReferenceException are completely handled by the framework and should NEVER appear within your corporate suite test runs!

    I hope you’ve found this information, and our base framework [link to Vanguard GitHub when available] useful. Please post any comments you have below.

    Posted in OpenSource

    5 Comments

    1. […] Flakiness of Corporate Selenium Suites and how to get rid of it […]

    2. wonderful put up, very informative. I’m wondering why the opposite
      experts of this sector don’t realize this. You must proceed your writing.
      I am sure, you have a huge readers’ base already!

    3. Thanks for finally writing about >Selenium Flakiness
      | OCPsoft <Liked it!

    4. JD says:

      Great article. Also appreciate the follow on post "Running the same suite across multiple OSes". Can the link to the base framework be added please?

      1. Trying to get all the required permissions to open source the library I talk about in this blog, is taking far longer than I was hoping for. I am actively involved in continuing to work through my companies open source policy for this library. I will update this article with the link once it’s finally available. Sorry for the long wait.

    Leave a Comment




    Please note: In order to submit code or special characters, wrap it in

    [code lang="xml"][/code]
    (for your language) - or your tags will be eaten.

    Please note: Comment moderation is enabled and may delay your comment from appearing. There is no need to resubmit your comment.