How to Write PHP Tests

The FOSSology project has decided to use the simpletest framework to write PHP tests. Browse the link for documentation, or use these links SimpleTest API or on sirius: Simpletest documents on sirius internal link. There are multiple API's for simpletest depending on what you are doing. Focus on the web-tester or scriptable browser API's. The web-tester documents document a set of assertions and pattern api's to use as well as api's for testing a web app. The scriptable browser api's overlap with web-tester api's. I often have both sets of docs up in tabs in a browser.

In the future, for difficult to test web testing the selenium plugin and selenium Remote Control might be used. For now simpletest meets our needs.

Fossology uses a common WebTestCase class that extends the simpletest WebTestClass class. This allows for project specific methods and helper functions on top of what simple test gives you. All the tests are designed to be run from the source tree. Please keep it that way. This helps keep the setup and configure steps lean and mean.

All unit tests should extend the fossology class instead of the simpletest WebTestCase class. The fossology class is found in svn at:

...trunk/fossology/tests/fossologyTestCase.php.

There is a class template that can be used to create a test. It's found at:

...trunk/fossology/tests/templateTest.php.

Make a copy of it and use the comments inside of it to get started.

There is also some test specific classes that can be used by the tests. These classes are found at:

...trunk/fossology/tests/testClasses/*.php.

The api's for these classes can be viewed in the source or at: Fossology Test API this is an internal link

• Simpletest Documents on sirius this is an internal link
• simpletest web site
• FOSSology Test API this is an internal link
• How to configure and run UI Tests

It is a really good idea to read the other sections below. In the interest of getting you started on actually writing a test we will just jump in.

The simpletest framework consists of a set of classes that provide an api for creating tests and then an api/utility to run the tests in either a cli or a web framework. Using the simpletest framework, all tests extend the simpletest classes in some way. That's how the framework works.

All fossology tests use a common php include file. This file is used for setting the URL to test and to supply the DB user name and DB user password. The template already includes the file and makes the 3 variables global. All tests need the URL, only the login test needs the user name and password.

• For this example we will use the templateTest.php
• make a copy of the template, let's create mytest.php
• the template tries to be self documenting, follow the steps outlined in the template. edit mytest.php and change the following according to the the instructions.
  • remove abstract from the class, any abstract classes are NOT run by simpletest.
  • give the class a name ending in Test, e.g. myfirstTest. Notice we extend the fossologyTestCase class rather than the standard WebTestCase or the fossologyTest class.
  • add any needed checks in the setUp method after logging in. For example if the test needs to make sure a folder exists it should check for it in setUp.
  • change the name of the method (function) from testsome…. we will use testmytest.
  • if your test needs to clean up after itself (remove files, folders etc…) then use the tearDown method.
  • save your test as mytest.php

There is a small amount of test environment setup that should be done before running any test. This is documented in this wiki at How to configure and run UI Tests. All the steps below are automated with scripts. Here is a brief summary of the steps used to set up the test environment.

Running the Install.php as sudo does the following

• set up the test users on the system and in the UI.
• copy over some test data into the test users home directory

These steps must still be done by hand, but take very little time.

• set the test environment by running configTestEnv.php to create the TestEnvironment.php include file.
• check the link for fo-runTests

If all you need to use is the test environment for logins then just setting the test environment should be enough. But at some point the new tests should be run with the other tests and for that all the steps will need to be done anyway. The setup is designed to be lightweight and quick to run/setup.

There are multiple ways to run a test. All methods use a simpletest run script. During the initial construction of the UI test suite, separate run scripts were constructed for each test suite. This is no longer needed. When tests are created, that method can still be used, but is no longer recommended.

A generalized run script has been created called fo-runTests . Fo-runTests can run 1 or more tests. Usage is:

• *fo-runTests** -l “list of tests space seperated”

If you ran the Install.php script or ran lnfo-runTests.php then a sym link should exist in /usr/local/bin/fo-runTests back to the source tree that you plan to test from. Cd into the where the test is, run it with fo-runTests. You can use fo-runTests to run everything in a directory or that you can wildcard with the shell.

fo-runTests -l "`ls`"
run one test: fo-runTests -l ./test-name

If you wish to create your own run script, then start with the template or copy fo-runTests.

...trunk/fossology/tests/runtemplate.php

Use the comments in the file as a guide. Replace the line $test→addTestFile('atest'); with $test→addTestFile('mytest.php'); Save the run script: myrun.php. Make it executable.

Make sure the test environment has been set.

Use the run script to run your test ./myrun.php

For a discussion of how to interpret test output and failure please see How to configure and run UI Tests.

Before you can write a test, it's a good idea to have a test case in mind that needs testing. Everyone on this team already does this when they test informally, they just might not think of it that way. For example, I'm going to run X, and I expect Y from the data that it will process. That's a test case. Of course test cases can get much more complex than that.

Test cases are usually written down, either informally or formally. Writing them down in a simple text test plan is usually good enough. Wikis are great places to store test plans. There are several advantages to having your test cases written down. The first is the it provides a record of what you plan to test. Hopefully all the documented test cases will get implemented, but if not there is a record of them so that when time allows they can be implemented. The second advantage is that when a defect occurs after your code passes all the tests, the test plan provides an easy way to see examine why didn't the tests catch the defect and provides a place to document new test cases added to fix defects. The third advantage is that a test plan allows one to look at all the test cases and pose the question what else needs to be tested? This is hard for most humans to do if there are more than about 3-7 items to consider and we have to hold them in our head. I'm sure you can think of other great reasons why it makes sense to have a written test plan. (A test plan in a wiki is a written test plan).

The sources that enable test case creation come from user requirements, External reference documents (man pages, api specs, etc.), internal specifications and defect reports. So for example if the man page says the program always creates a log file then one can test if that is true. You can read code to help generate test cases, but that is not very efficient.

This sounds very old school but it works. Write the code, write the man page, write the tests, run tests/add test till all tests pass and user requirements are met. Release! Even Xtreme and Agile methods use this, sometimes with a different order.

All methods assume that the user requirements are understood and will be understood before final code. Rapid prototyping for example allows one to learn requirements as one goes but still starts with an initial set of requirements. The point here is that if the requirements are captured in some way it allows the programmer or tester to use the requirements to generate test cases.

If Selenium is used to web test, it is very useful to only do one test case for playback. This helps keep the tests from being too brittle and one test failure has a smaller chance of cause a failure down stream.

Definition: a brittle test is a test that breaks easily due to environment changes or dependence on prior tests or test results.

• each test stands alone (if possible). This means it does not depend on a prior test to set something up or leave a condition around for the test to use. Each test should try to set up what it needs to test and then remove it when done.
• setup/tear down is fine, try to minimize. A test should always try to clean up after itself. Or create a cleanup script that cleans up the testing cruft as the last step.
• keep tests small. Don't try to test too much at once. This keeps you sane. Also helps with debugging when a failure occurs.

This is just a stab at some the larger areas that need to be tested in fossology in no particular order or priority.

• License Matches
• unpack
• uploads
  • url
  • file
  • server
• UI
• CLI

Under Construction……

• use a code example
• write a test for it
• show how to run the test
  • cli
  • web
• Web tester