Wednesday, December 28, 2011

List files in a Directory-Pyton code

How to list the files along with path in python?

for root,dirs,files in os.walk('C:'):
    for file in files:
        print root+'\\'+file
This will print the file name along with absolute path.

Sunday, November 6, 2011

Threading and UI test automation


If there is one thing that is the most difficult part of test automation is waiting for some action to finish then proceed. I am not talking about ajax/waiting for UI element, waiting for the AUT do something or waiting for some time to elapse.

Most of the tools are use only sequential execution. I should say all, but I think there may some tool that has this feature. As this different from parallel or grid execution. Let me explain with steps,


Typical Test Script does the below functions,

1. open a page
2. Click something
3. go to next Page
4. do something
5. check for something

Above steps are perfectly good except when there is need to wait. See the below steps

1. open a page
2. Click something
3. go to next Page
4. do something
    < Wait for sometime>
5. check for something

If the <wait for something> time to is few seconds or few minutes then no need to worry unless it affects the execution time significantly.

What if that time is hours or day?
How do I automate an app/webpage that needs periodic wait time?

For ex, testing an Calender apps is a good example.



So, what needs to be done the automation tools to work better?

Programmers who reading would have shouted Threading/Concurrency. Yes, that is the obvious answer. Having a thread scheduler or task notifier will make this job easier. I am wondering, why this obvious things are not available in Test Automation tools? Why testers need to be satisfied with basic sequential execution?

Yes, I can use the threading in API tools such as Selenium. This why I love API based tools rather than Record and Playback or off the shelf tools. See my previous post here Test Automation using Tool or API (http://vrajasankar.blogspot.com/2011/08/test-automation-using-tool-or-api.html)

Further, this waiting for action can separated into two categories.

1. Checking/waiting for outside action (nothing to do with UI)
2. Checking/waiting for UI action.

Examples for trying to check a action outside of the UI is check for mails, checking the values in DB, check if files are available etc. For this, actions can be checked in another thread while UI execution move next script thus speeding up the test execution.

For the second option, either those actions can scheduled or another instance(head/headless) can be started to finish the test.

I think, without this threading/concurrency capability it is not possible to automate the complex enterprise applications.

Tuesday, September 27, 2011

Productive Tester-I

Long time ago I read Productive Programmer by Neal Ford, after reading that I wondered what could be a book named Productive Tester consist of.

Really? Is testing can be done like that?

I am not interested in Software Test Cycle kind of plans, procedures, test scripts, etc to do testing. I am interested in how a small member team can test or do that testing without those fancy documents and million dollar products?

I take IDE used by programmers for analogy. So, if IDE helps developers to create code in better way or just speed ups what could be the one for testers?

One thing immediately comes to my mind is Log parsing and analysis. Log watcher by James Bach and Log parser utility by Microsoft.

I will start posting my thoughts in the coming days in the series.


Monday, August 29, 2011

Test Automation using Tool or API

This debate between Record-Playback  and Scripting is going for long. I have written about that too. Now, after Selenium 2 release, some people asked me, why should one needs a framework such as Selenium? Comments range from, "it is hard to understand", "can this be simple?" or "I want to get started in easy way".

Those comments are seems to valid if you're not burned your finger in testing an enterprise app or writing automation code for thousands of tests. Because, only after automating 10 or 20 % of the cases, you'll come to know that, some actions can't be automated because there is no option in the tool to do that. Then, finding out that there is no way you can do that, will be an another big surprise.

So, if we go in the Selenium way i.e the framework way, it is possible to do that? Yes. Because, as a Framework, it doesn't try to predict the permutations or combinations for all the test automation scenarios.

There are lot ways, I want to test my application. I may use, keyword driven testing, data driven testing or scenario driven testing or some other driven testing. I may want my tests developed in BDD or do a acceptance testing. So can one tool/software give the options for all? It will actually make the product bloated.

Take the analogy of WYSIWYG editors for HTML. Ican write html in a text editor or use those editors to speed up the work(drag and drop). It offers the flexibility and best of both worlds. Same can be said to IDE for programming languages. If something doesn't work in IDE, you can edit the file using text editor. If that possible and good for developing software, why it is not good to do that in Test Automation?

Another example is using BDD. I want to use BDD but I want to use it my way or extenstion of something I already used, not the way like Rspec or JBehave/easyb. So tying these wont work for me.

Now, if we deduce, the issue is "what to do" is combined with "how to do" causing most of the issues in test automation rather than technical issues such locating the elements in the UI.

Here,

What do to= Actions done in the UI such as Click, Select etc

How to do= Sequential use of the above actions

Separating "what to do" and "how to do" will bring lot of benefits. I would like to add another example to emphasis the point. Catching the JavaScript errors is a valuable improvement in automation. If I can tweak the Click function or extend, so that, I can check for every click, is there any JavaScript errors.

Here are the requirements for a ideal automation framework

1. All actions should be available independently . No dependencies .

2. Should only do UI actions. Operations such as I/O, connecting to DB should be written by users.

3. Should be extensible.Adding a new action should be easier.

4. Should use standard languages and standard methods to locate the elements. Ex. Java/Python to script. CSS/DOM/XPATH to locate the elements. No vendor scripts.

 5. Running the tests should be independent.

Tuesday, July 19, 2011

Selenium and Waiting for Ajax

Keith Sterling posted a post, how to avoid Thread.sleep in automation code in Selenium Smells


Once of the biggest issues I see with people new to Selenium ( or any number of the other web automation tools ) is the liberal scattering of Thread.sleep(xxx) through out the code. This should be considered a bad smell introducing timing specific code into your script.
As this waiting for a element is one of biggest headaches in automation, I thought, I should share my experience.

In Sizzle.js(of jQuery), there is simple code snippet to check if there is an element present

$(window.document).find('locator').length>0

In ajax, the element may be there, but it is may not be visible to the users. So, checking for length alone wont do. For that, checking for visibility will do.

$(window.document).find('locator').is(':visible')
Call these two snippets in the waitforcondition and see the magic. Selenium can automate any ajax or dynamic element actions.

These tricks for only Selenium 1.0. I think, in Se2, these issues may not be there. I need to check and confirm this.

Wednesday, June 8, 2011

IPv6 Day

Today is IPv6 Day.

Check the official site http://www.worldipv6day.org/

Also run this test to check if your org is ready for IPv6 change, http://test-ipv6.com/


Wednesday, June 1, 2011

Firefox Addon Collection for Test Automation

I've created the following collection for easy find and use. Hope it will help you also. 

https://addons.mozilla.org/en-US/firefox/collections/rajasankar/ui-test-automation/

Another my collection for personal browsing, https://addons.mozilla.org/en-US/firefox/collections/rajasankar/personal-browsing/


Saturday, May 14, 2011

Transparency and Business

 

I tried to catch up my unread blogs and read a blog by Seth Godin.  In Why you might choose to be in favor of transparency, he made the case for how businesses can benefit from transparency. I would like to add some more points.

Businesses take extra care for not leaking their business plans, that is the conventional view every body agrees. It is based this following argument, in a cut throat competition, knowing what the competitor going to do, can be a big advantage. I also had that view until I read Ricardo Semler’s Maverick : The Success Story Behind the World's Most Unusual Workplace. Semler showed to world that if you keep your plans dynamic, knowing what they could do, can’t bring an advantage to the competitor. When competitor working on that plan, it could be changed in Semelr’s factories. Workers debate and update the plans every day, every week so there no static plan to rely on.

Seems to be illogical, isn’t it?

Think about open source code. Lot of people thought that by making the source code open, security will be compromised. It was proved otherwise.

If we extend this argument as Godin did in that blog, Internet removed the these barriers and opened the flood gates of knowledge. Transparency always brings benefits. 

Sunday, May 8, 2011

Standard Scripting languages in Test Automation

 

By standard scripting languages, I mean, JavaScript, Python etc. To know more about how non standard scripting languages used in test automation, please read Hey Vendors, Give Us Real Scripting Languages by Bret Pettichord. Even, though, it was written in 2001, the points made is still valid today.

Another aspect is using the standard locators in UI element locators. Let me explain with one example. I had a OR condition in  the element locator. In non standard locators, I am forced use this following script

Check if Element A is present

if A is present click A

else

Click Element B

Which means I need to make at least two calls to check if the particular element is present.

Consider the multiple selector option and multiple attribute selector in jQuery,

Multiple Selector (“selector1, selector2, selectorN”)

Multiple Attribute Selector [name="value"][name2="value2"]

Click (Element A,Element B)

If I can use the above selector formats, just one step is enough. Add to this how many lines I don’t need to type to check this. Instead of four lines, script will have only one line. I choose jQuery as example. Those formats are actually CSS selectors.

Next time if you’re still using a old custom format for element locators, think about CSS.

Thursday, April 28, 2011

Software Estimation - Understanding

 

As I work in software testing using automation, giving a figure of how long it will take to finish something is hard. Sometimes the figures are correct, some times not. When I started reading about that, there some rules of thumb given. Get the time from developers, double it add a little bit is one of them. Other reason  is software developers work to create new knowledge. I have problem with these kind of maxims.

If you think about this conclusion, software estimation is hard because it is work to create new knowledge, you would obviously need to ask what about other sectors, engineering disciplines are doing in this area.

Is producing new cars is hard because they work to create new knowledge?

Is inventing new drugs/medicines is hard because they work to create new knowledge?

However, one can argue that, they are just one or two exceptions, others sectors are same.

How about designing a new home? 

How about exploring for oil?

The point is, all kind of estimations are actually predictions. Even if it called as forecasting, it is still prediction about future. The problem is no one can predict what they don’t know. If they know, there is no need to predict it.

Here is quote from my favorite author Nassim Nicholas Taleb in Black Swan,

Consider the wheel again. If you are a Stone Age historical thinker called on to predict the future in a comprehensive report for your chief tribal planner, you must project the invention of the wheel or you will miss pretty much all of the action. Now, if you can prophesy the invention of the wheel, you already know what a wheel looks like, and thus you already know how to build a wheel, so you are already on your way. The Black Swan needs to be predicted!

Read Black Swan about the highly improbable.

Let us take another simple work. Delivering mails (not emails), packages, parcels to customers. There is no thinking involved in it, right? If it is so, why we have a tracking feature to know where is the package now.

Coming back to where we started, if estimation is hard how do we know when the project/development will complete?

The short answer is there is no fail proof method to know.

The long answer is, we use what we know to know when approximately it will complete, keeping in mind that something may go for toss at any time. Use number of options such tracking the progress, following standards etc may solve the some of the issues.

If you consider, software test estimation, the picture is gloomy. In developing software, the work is cut. You have basic understanding of what the end product will look. However, that is not possible in test estimation. I am not going for clich├ęs here.

Yes, there are lot of strategies one can use to find bugs in the software. However, even if testing for lot of possibilities may still leave bring disaster because the tester haven’t tested for one possible input.

Understanding that we can’t predict what we don’t know will be a first step to solve these issues.

Sunday, April 17, 2011

Testing Vs Automation Checking

    The argument that, testing is not checking has generated lot of content. While reading the whole debate, I thought, we’re missing some pieces. Hence this post.

See the Testing vs Checking posts here,
http://www.developsense.com/blog/2009/08/testing-vs-checking/
http://www.developsense.com/blog/2009/09/tests-vs-checks-motive-for/
http://thetesteye.com/blog/2010/03/the-testing-vs-checking-paradox/
http://www.testingreflections.com/node/view/8282
http://www.testingreflections.com/node/view/8243
http://www.infoq.com/news/2009/12/testing-or-checking
http://www.stickyminds.com/sitewide.asp?ObjectId=16684&Function=edetail&ObjectType=COL

A simple summary of the discussion. Testing is a learning process, that can’t be automated. Testers can only automate checking not testing. Checking means verifying a button/link/functionality is there or works expected. Other words, human intelligence can’t be automated. Yes, that is the point. However, it missed an other point. Human intelligence can be used in creative areas rather than doing mundane repetitive actions.

Am I saying automated testing can be called as Tool Assisted Testing? Nope.

Let us take Programming. A programming language has control flow, data structures, memory management,error reporting etc. If you look at the abstract level, all the actions done by the programming language is actually checking.
Get an input ---> Check the input –> Do Some actions based on that ---> Show/Pass the output
Which means, if a programmer learns all these things, can he write a great application using this? Nope.
Wisdom of what to put where and how, will differentiate between a good programmer and bad one. Handling errors, designing a UI will make a difference.

Coming back to the point, the purpose of testing starts when tester sees an error and decides what to do with that. That particular error may be a bug or UI needs some improvement or may lead to finding an another bug.

Enter automation. If automation can capture the errors in UI, shows a nice report along with screen shots, log entries etc, tester can analyze it and chart a future course of action. In essence, time spent on producing those errors may be or can be saved. We know the computer is dump, can’t think. So why assume it will assist us in finding the bugs?

Using automation tools in right way and doing more work in less time (productivity) will give us more time to do testing. Human intelligence can be used to do more productive and creative works.

Lets us see a quote from Seth Godin, to understand what I mean.
And email is not a faster fax. And online project management is not a bigger whiteboard. And Facebook is not an electronic rolodex.
Play a new game, not the older game but faster.
In the same way, automation is not faster testing. For example, take Static code analysis. It can find flawed code that may be buggy. Using it to scan the entire repository will make job of the coder easy to find bugs. More over, if he makes some mistake it can be found using this. Code review will be a easier with static code analysis.

Another example is continuous integration tools. It can start the testing when new code checked in.

Test Automation is not faster testing. It enables more testing.

Saturday, April 16, 2011

Moving Thunderbird Mails to web mail



I am using Thunderbird for long time. For storage and other reasons, I want to move my Thunderbird mails to web mails such as Zoho mail. There are some solutions to this issue. You can convert the mbox mails to eml, upload it to web. There is a solution to read this mbox and forward it one by one. Trying all these, I found a easy solution. IMAP come to rescue.

IMAP will sync the folders and mails between client(Thunderbird) and Server(Zoho mail). So, I created IMAP account in Thunderbird, transferred the mails from local folder to Zoho mail folder. Bingo. I can see all the mails in the web interface.

Here is the step by step instructions

1. Download and install Thunderbird

2. Create a Zoho mail IMAP account in Thunderbird. See this link for more info, Configure in Thunderbird

3. Create a folder under the Zoho mail account. Let us say that Test folder.

4. Move/Copy your mails in the local folder to Test folder.

5. Sync the account using get messages.

Now, you can see all your moved mails in web interface.

Thursday, April 7, 2011

Tools used in Manual Testing

Common test tools useful in Manual testing

1. SQLYog
    SQLyog MySQL GUI is the most powerful MySQL manager and admin tool, combining the features of MySQL Administrator, phpMyAdmin and other MySQL Front Ends and MySQL GUI tools.

2. DBVisuvalizer :
    DbVisualizer is the universal database tool for developers and database administrators. It is the perfect solution since the      same tool can be used on all major operating systems accessing a wide range of databases.

3. MAT (Eclipse Plugin)
    The Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.
4. MicroSoft LogParser
    Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®.

5. Microsoft SysInternals Suite
    The Sysinternals Troubleshooting Utilities have been rolled up into a single Suite of tools. This file contains the individual troubleshooting tools and help files. It does not contain non-troubleshooting tools like the BSOD Screen Saver or NotMyFault.

6. Putty
    PuTTY is a free implementation of Telnet and SSH for Windows and Unix platforms, along with an xterm terminal emulator. It is written and maintained primarily by Simon Tatham.

7. WMIDiag
    A new utility to help system administrators diagnose and repair problems with the WMI service.

8. WMI Administrative Tools
    WMI Tools include: WMI CIM Studio: view and edit classes, properties, qualifiers, and instances in a CIM repository; run selected methods; generate and compile MOF files. WMI Object Browser: view objects, edit property values and qualifiers, and run methods.

9. Unlocker
    Tool to remove the folder locking

10. WinDirStat     WinDirStat is a disk usage statistics viewer and cleanup tool for Microsoft Windows.

11. WinMerge
     WinMerge is an Open Source differencing and merging tool for Windows. WinMerge can compare both folders and files, presenting differences in a visual text format that is easy to understand and handle.


Wednesday, January 26, 2011

Selenium and Choosing a Framework

         After seeing some people asking questions about choosing/developing a framework to use Selenium, I thought, I could offer few tips. This may look like ways to creating a new automation product, however, customizing the tool to the needs of app and the team, will decide the success of test automation. It is like bespoke vs ready-made.
       I've separated this into two categories, one is how to run and other is how to have a robust, reusable Selenium tests. If you choose to go with this way, forget about Se IDE. This is not to discredit the IDE but to understand that record and playback, wont work in this approach. If you think that you can record and put the cases in the script, sorry to bust the bubble. It wont work.

I. Running the Selenium scripts.
      I will list the available choices. These are specific to Java.

  • I can live with the default main method as I do in my app - No need for any existing frameworks. You can decide on your own or use the Java Reflect API.
  • I want run with minimum work. - Choose JUnit or TestNG, again depends upon how you want to organize tests.
  • I want to run using CI - Plug-ins are available for Maven/Hudson
II. For a robust Selenium tests.
     This portion will also cover general test automation best practices, however, I will highlight how to get that using Selenium.


      A. Understand the App you're going to automate:
           I've seen a number of people trying to automate, without understanding the UI/App. It will be a spectacular failure because, that is wrong way to start with. UI can be a heavily ajaxified , may have lot of dynamic ids or app may be in early stages so UI may change infrequently.

          For ajaxified pages using CSS selectors is the best option. Going with Sizzle/jQuery based CSS selectors is nice. If the app has lot of dynamic elements, talk to your developers to understand the underlying pattern. Having a custom locator using that pattern will solve lot of issues. Don't use Xpaths with div or span in the locators, if that div is changed to span, your tests are broken. There wont be any visible changes but tests still wont run. So think before using those locators.

         B. What is testing requirement now and what are the plans for future?
              Now the hardest part. Deciding what you're going to do and what else will be needed in future will decide time and difficulty of the your test automation. Let me list few options.

  • Access to database/csv/excel etc
  • Integration with logging framework if you need to print info message wherever needed
  • Integrating with speed load testing frameworks such as JMeter
  • Integration with HTML/JS validation frameworks such as HtmlUnit. This way you can do unit testing of the page.
Once you've decided on these, taking care of options to handle exceptions/alerts, capture screen shots, generating test results/reports come to play. If the framework/model doesn't handle these things then it will not reduce work or add value to the testing.

III. Page Objects or BDD?
       As you know in Selenium 2, Page Objects come with it. However, by separating element locators from code using properties or xml file you can get benefit of the Page Objects without going for it. You can point out that Page Objects can do much more that, however, my fingers crossed on that issue.
      
       If it so, why BDD? Using BDD model you can separate flow of actions from the code, that extends the concept of Page Objects. Moreover, creating and maintaining code written using BDD is easy. And if you extend that write your own framework for BDD, you can even create the customizations you need without trying to predict what you will need in future.  


In the end, creating tests which will bring value to testing, app and the team matters. So choose wisely.

Sunday, January 2, 2011

Issues with Page Objects Pattern

From Selenium 2, comes the Page Objects Pattern, it is hailed as a way to solve the issues of code maintenance and duplication in test automation.

Let me describe the Pattern as I understood. Then criticism and why it can't solve the issues it promises.

Here are three sample test cases,

Case A.
1. Login to Blogger.com
2. Click Dashboard link
3. Click New Post

Case B

1. Login to Blogger.com
2. Click Dashboard link
3. Click New Post
4. Click Comments

If we write a cases in a traditional way, it will look like this,

public void test_caseA(){
     selenium.type("username","test");
     selenium.type("password","test");
     selenium.click("login");
     selenium.click("link=Dashboard");
     selenium.click("link=NewPost");
}

public void test_caseB(){
     selenium.type("username","test");
     selenium.type("password","test");
     selenium.click("login");
     selenium.click("link=Dashboard");
     selenium.click("link=NewPost");
     selenium.click("link=Comments");
}
Here we can move  same actions into a another function, like this,

In ClassA,

public ClassA() { 
     public void login(String username,String password){
          selenium.type("username",username);
          selenium.type("password",password);
          selenium.click("login");
     }
In ClassB,


public void test_caseA(){
     ClassA.login("test","test");
               selenium.click("link=Dashboard");
     selenium.click("link=NewPost");
}
However, this is an simple example. It is possible to have all the functions in related to one page in a class and call that using method names. In a nutshell, even if the page changes, change is needed in only one place. That is the purpose of this Page Objects Pattern. But, what is the issue with this?

This approach doesn't address the change in sequence steps. UI automation has two issues which needs maintenance of code.  

1. Element changes in UI: Changes such div element changed to span or radio button becomes a check box. 
2. Design or flow changes:  Let say, need to automate a three step sign up process, each step is a page. After releasing, all the three steps combined into one page. If the tester scripted for three step signup, it need to be changed. 

The first issue is addressed by Page Objects Pattern, but not the second. If the second issue occurs, then using this pattern will not solve the purpose. 

So, what can be done? I think, using Behavior driven development model can be answer to this issue. More on that later. 

Rajasankar. 

ShareThis

raja's shared items

There was an error in this gadget

My "Testing" Bundle