Why using JavascriptExecutor in WebDriver can be dangerous

I have occasionally seen people recommending running javascript using the JavascriptExecutor implementation. For example, I was working on a project where a test would pass on the developer's computer but would fail on the functional test machine. The line of code which failed was:

driver.findElement(By.cssSelector(".save")).click();

So the developer changed it to:

((JavascriptExecutor)driver).executeScript("$('.save').click();");

It seemed harmless and it worked.

The GUESS was that WebDriver found the button because it was always in the DOM with style="display: none;" but the click failed because it wasn't waiting for the button to become visible.

The idea was that the test was REALLY:

- open the dialog
- wait for the button to become visible
- click the button

If we used javascript we didn't have to wait for the button to be visible. So javascript seemed faster but was still testing the right thing.

No problem, right? Wrong. It turned out that the save button was on a javascript dialog which did not scroll. If you scrolled up and down the page the dialog remained still; the dialog ALWAYS stay exactly 120px from the top of display. When all the tests ran it created data which made the dialog 830px high. This placed the bottom of the dialog at 950px. The developers machine was 1600x1200 display. It made the dialog fully visible on the display. The functional test machine was set to 800x600. This forced the bottom of the dialog off the bottom of the display. Even when you made the browser full screen you could not see the save button.

So on the developers machine WebDriver would see and click the save button because it was visible. On the functional test machine (set to 800x600 because that was the minimum requirement) it was not visible and you could not scroll it into view because the dialog did not scroll.

When they switched to the javascript code it clicked the button regardless of whether the button was visible or not. If we shipped this application, any customer who was using a computer with 800x600 display (a significant number of people based on analytics) would be unable to click the save button even though the functional test passed.

The moral of the story is, if the javascript solution doesn't test EXACTLY the same requirement as the WebDriver solution, you could be making your tests report a false positive.