Monday, December 27, 2010

Lies, damned lies, and statistics

Mark Twain told once, There are three kind of lies : lies, damned lies and statistics. There are websites who take this analysis far and show how statistics are used to lie about something. There is a book named How to Lie with Statistics (

Basic concept is anyone can use the statistics to present a different picture than what actually is. Check this video ( mathematician Peter Donnelly  explains how statistics used to fool juries in the criminal cases. 

How is that related with software or testing? Lot of things we do is related with numbers. Let me give you some examples.

First, code and test coverage numbers. Let us say 10 bugs per KLOC, i.e., 10 bugs in every thousand lines of code. However, if you look at module wise, branch wise, loop wise data, this average won't fit. And there will differences between code written by experienced developer and fresher. So, 10 bugs per KLOC may tell that we need to put same testing effort for all modules, but in reality some modules may need more testing effort than other modules. If you can categorize modules based of effort needed, then it is easier to test. 

Another example is number of test cases executed data. Say, testing team executed 10k cases and found 1k bugs. Is that mean 1 bug per 100 cases? Maybe. But can we conclude the most if not all of the modules tested by these? Nope. 

That data need to be correlated with test coverage numbers. If 100 test cases check the same code logic or use different data to test, it is waste of time and effort. Chances of finding a bug goes to none after the first test. So if  these unnecessary test cases removed, then, we will get an estimation of how many test cases needed to test the product. 

Aware of this kind of averaging, because, it is not good in most of the times.

Sunday, December 26, 2010

Change in Blog description and writing

When I started writing in this blog, I thought, I should write only about technical things not about everything. But after some time, I started thinking of writing non-technical but work related subject too. So from today, there will be posts on other related subjects too.


Monday, October 4, 2010

Testing Flex with Selenium

The purpose of this post is to explain the differences between Flex and Flash w.r.t to testing and how to test the same with Selenium. This follows the previous post of Testing Flash with Selenium.

Flex and Flash both based on ActionScript. Flash coded using AS3 and complied into SWF file. In Flex, actions are coded in Java and converted into AS3 and complied into SWF file. For Java developers, Flex offers a option to code in Java rather than learning AS3. Actually, building a Flex app is complex than Flash app, however, discussing that is out of scope of this post.

As we saw in here, External Interface is the only JavaScript can interact with ActionScript. In Flex, it is possible to add ExternalInterface call backs for all the actions when compiling with a FlexUISelenium API. So it takes away the burden of manually adding and removing the call backs in the case of Flash.

The following steps describe how to install and use FlexUISelenium  in a few simple steps.
  1. Download and install Selenium RC  
  2. Download Selenium Flex API, and rebuild your Flex application with SeleniumFlexAPI.swc 
  3. Download FlashSelenium and add to your test project 
  4. Download FlexUISelenium  and add to your test project 
  5. Write and run your test cases against your Flex applications 

So if you use Flex instead of Flash, is that possible to test that in Selenium easily? No. If the API can't add ExternalInterface ability to all of you actions in the UI, then you need to review the code  manually and add all those things. Again, we left with the manual option of adding the interaction between ActionScript and JavaScript. How do we know, this method didn't add those callbacks? Only when recording the tests.

In the end there is no silver bullet for testing Flex/Flash apps in Selenium. Manual work is needed to add the ability of interaction in these apps.

PS: SauceLabs Flex Recorder claims that there is no need to recompile the Flex app in order to test. I haven't tried that so can't comment about that.

Thanks to my friend Vinod for helping me in this.

Happy Testing

Tuesday, September 14, 2010

Using AutoIt in Selenium

First about AutoIt

AutoIt is a freeware Windows automation language. It can be used to script most simple Windows-based tasks (great for PC rollouts or home automation).

Using AutoIt, it is possible to handle the tasks that Selenium can't do. It is possible to upload/download files using AutoIt in Selenium.

Here are the ways to find a window,

Advanced Window Descriptions
A special description can be used as the window title parameter. This description can be used to identify a window by the following properties:
  • TITLE - Window title
  • CLASS - The internal window classname
  • REGEXPTITLE - Window title using a regular expression (if the regular expression is wrong @error will be set to 2)
  • REGEXPCLASS - Window classname using a regular expression (if the regular expression is wrong @error will be set to 2)
  • LAST - Last window used in a previous AutoIt command
  • ACTIVE - Currently active window
  • X \ Y \ W \ H - The position and size of a window
  • INSTANCE - The 1-based instance when all given properties match

Check the below docs,

Window Titles and Text (Advanced)

ControlCommand Reference

Now, How to use this Selenium?

1. Install AutoIt and write a simple script. AutoIt bundles the ScitTE editor for this. Code Editors such as Notepad++ supports AutoIt syntax.
2. Compile that script into an EXE or make a batch file to run that script.
3. Call that EXE/Batch file from your code.

I've encountered an issue when trying to run AutoIt script. It throws an DLL alert before executing the script. Installing AutoIt, solved that one. So, you may need to install the machines you want to run.

Happy Testing.

Saturday, August 28, 2010

Testing Flash With Selenium

Is it possible to test Flash with Selenium? Yes, but it is difficult. Before seeing how much difficult it is, let us see how to test Flash in Selenium.

Testing the Flash is possible using External Interface Command in ActionScript. External Interface allows the JavaScript to interact with the ActionScript. Here is the list of actions we can do,

From ActionScript, you can do the following on the HTML page:
  • Call any JavaScript function.
  • Pass any number of arguments, with any names.
  • Pass various data types (Boolean, Number, String, and so on).
  • Receive a return value from the JavaScript function.
From JavaScript on the HTML page, you can:
  • Call an ActionScript function.
  • Pass arguments using standard function call notation.
  • Return a value to the JavaScript function.
Flash Player does not currently support SWF files embedded within HTML forms.

So, if you call a ActionsScript function via JavaScript, that action will be performed in the Flash. This is the method used by the Flash-Selenium and similar projects to test Flash using Selenium. 

To test a Flash app using this method, all Flash functions should be exposed via ExternalInterface to JavaScript. If it is not, that function can't be called via JavaScript. If the Flash app has a lot of functions needed to be tested, exposing all of them using ExternalInterface is cumbersome or sometimes may not be possible. 

So, if your app has less functions that needs to be tested, you can use this approach, if not, testing manually will be the better option.

Also check this article for more info,

Thanks to Vinod for helping in this. 

Wednesday, August 18, 2010

FireQuery-FireFox Addon for Checking jQuery code

I was using the Firebug and FireFinder add-ons to know the XPath for knowing the locators in Selenium. CSS code also can be checked using FireFinder. When I am searching for a some kind of add-on that can offer this in Firefox, I found FireQuery doing that.

Here are the steps to get that working.

1. Install Firebug add-on
2. Install FireQuery add-on.
3. Activate the Firebug for the webpage and go to the console.
4. If the webpage doesnt have the jQuery.js added, you need to jQuerify the page.
5. Run the needed jQuery code in the console as you do for the JS.

Easy and simple. Thanks for Binaryage who developed this addon.

Wednesday, July 21, 2010

A New Start

I've stopped blogging in Aug 09. After that, I've read and worked in lot of things. Worked in Selenium, FitNesse, jQuery etc. Will start blogging about all these links along with what I read.


raja's shared items

There was an error in this gadget

My "Testing" Bundle