The tests are designed to be run from the FOSSology source tree. This cuts down on installation and configuration time.

The installation process has been automated. As sudo, just run the Install.php script found in the tests directory.

$ sudo Install.php

If for some reason parts of the install fail, each step can be done by hand and usually there is a separate script for each step.

The initial install only needs to be done the first time the tests are installed. After that, the tests can be run as often as needed.

Installation is really a two phase step. First the Install.php script is run. Then a small set of manual steps are performed. When all of the steps have been completed with no errors, the tests are ready to run (see below for the complete list).

TODO: Rando to write a check script to check on the configuration….

The initial install script is run when the tests have never been installed on a system or when a retest is needed and the system was upgraded to a new OS or the test users and data were removed from the system.

To run the install script:

cd <path-to-sources>/fossology/tests
sudo ./Install.php

Usually the Install script is run just once. It does not hurt to run it multiple times.

The steps below are required when first installing the test suite.

• as a user with sudo privledges, check out the sources, let's call this user TG (TestGod).
• as TG run the Install.php script
• address any failures in the install.
• in another window, login as the fosstester user created by the Install script.
• as fosstester, check out the sources again, and cd to the tests directory.
• as user TG cd over to the fosstester's sources. cd to the test directory.
• as sudo, reestablish the link in /usr/local/bin to this copy of the sources

sudo ./lnfo-runTests.php

• as fosstester run the configTestEnv.php script to create the test environment.

fosstester$ ./configTestEnv.php 'http://mysys.domain/repo/' fosstester fosstester

Run the tests

./testFOSSology.php -a

See Install Steps Detail section below for more details on the install process.

There are times when parts of the test install need to be redone after an initial install. This document can't detail all of them but will try to capture the tasks that seem to need to be re-done as the test system is used.

• after a Database drop. There will no longer be UI users fosstester and noemail. To fix this rerun the createUIUsers.php script using fo-runTests. First the TestEnvironment.php file must be changed to the fossy user or any user allowed to create accounts.

./configTestEnv.php 'http://mysys.domain/repo/' fossy fossy
fo-runTests -l createUIUsers.php 

or

./fo-runTests.php -l createUIUsers.php
./configTestEnv.php 'http://mysys.domain/repo/' superuser superduper

• After an OS cold install, the initial install process should be run again.
• After a failure in the Initial install script. (See below)

• easiest to just pull the sources and create the test users and load the test data on each agent.
• email must be configured on each agent for email notification to work.
• See the note below for upload from server options. If you use the 1st bullet above, there should be no issues.

Important Note for Multi-System Setups - make sure the scheduler.conf file is configured correctly or the upload from server tests will fail. Either the test users home dir needs to be mounted under /home on all the agents or restrict where fosscp can run (either confine it to the server or the server and a single agent). If you don't want to mess with NFS just confine fosscp to the server and 1 agent. Then on that designated agent, check out the sources and install the test user and the test data.

All of the tests use the &nopopup=1 parameter when logging in. This makes the java script popups turn into a H3 heading instead.

The fosstester user and user data should be created/installed before running the suites. The tests are fully automated. Use the -a option to run all of the tests (normal usage). Using the -a option, the script will wait for the uploads to complete and then run the verify tests. If for some reason the automation does not work, the verify step can be run by hand see Running the Test Suites below for an example.

All suites can be run from a master script testFOSSoloyg.php located at

..path-to-sources/fossology/tests/ 
./testFOSSology.php -a

The testFOSSology.php script when running will log it's output in …path-to-sources/fossology/tests/FossTestResults-yyyy-mo-day-hh:mm:ss-[am|pm]. When -a is used, then all the results are logged in a single file. When the verify step runs it moves the test results to ~fosstester/public_html/TestResults/Data/Latest.

The tests have finished. How do you tell if they passed? The easiest way is to grep the log file:

grep Passes FossTestResults-2009-05-14-10:57:21-am
 
results in:

Test cases run: 1/3, Passes: 30, Failures: 0, Exceptions: 0
Test cases run: 19/21, Passes: 290, Failures: 0, Exceptions: 0

The above will show you all of the results lines from each suite that was run. As long as Failures is 0 for everything, the tests passed. If there is a failure, examine the log file and use the sections Common Test Failures and Test Output and Investigating Test Failures to help investigate the failure.

When developing a test and debugging it often just that test is all that needs to be run. There is an easy way to do that now using fo-runTests. To run a single test, cd to the test directory where the test file is:

fo-runTests -l test2run.php

The above example assumes that the tests were installed. Part of the installation is to create a symlink in /usr/local/bin for ease of use. If the link didn't exist, one would have to put a relative path in to the php file.

For example:
cd ui/tests/BasicTests
../../../tests/fo-runTests.php -l mytest.php

The moral of the story is that if the link isn't there, use the relative path to the tests directory where fo-runTests.php is located.

All suites can be run from a master script testFOSSoloyg.php located at

..path-to-sources/fossology/tests/ 

Make sure the testFOSSology.php script executable, (it should be that way already).

The usage for testFOSSology.php is:

Usage: ./testFOSSology.php [-l path] {[-a] | [-b] | [-h] | [-s] | [-v -l]}
-a: Run all FOSSology Test Suites
-b: Run the basic test suite. This runs the SiteTests and any Tests that don't depend on uploads
-h: Display Usage
-l path: test results file path 
-s: Run SiteTests only (this is a lightweight suite)
-v -l: Run the Verify Tests.  These tests require uploads to be uploaded first. 
See the test documentation for details You must specify a log file when using -v, best to use the same 
log file that  was used in a previous run for example -a -l foo, then -v -l foo when    all the files 
have been uploaded with -a.

The tests are fully automated. Use the -a option to run all of the tests (normal usage). Using the -a option, the script will wait for the uploads to complete and then run the verify tests. If for some reason the automation does not work, the verify step can be run by hand, like the example below.

./testFOSSology -v -l <previous run logfile>

The testFOSSology.php script when running will log it's output in …path-to-sources/fossology/tests/FossTestResults-yyyy-mo-day-hh:mm:ss-[am|pm]. When -a is used, then all the results are logged in a single file. When the verify step runs it moves the test results to ~fosstester/public_html/TestResults/Data/Latest.

Each suite when run separately will create it's own log file with the format FossTestResults-yyyy-mo-day-hh:mm:ss-[am|pm]. These results will not be moved to the TestResults area.

• TestEnvironment.php file is not correct. The most common mistakes is to forget the /repo/ part of the url. The other common mistake is getting the hostname wrong. Either of these mistakes will cause all the tests to fail due to login failures. All of the tests may pass, but may not be run by the user you expect, check the file before running the tests.
• Simpletest not installed. The Install script checks and attempts to install it if it's not there. If simpletest is not installed in /usr/local then the tests will appear to run very fast but the results files will be empty. The tip off is it will say waiting for jobs to finish but will continue executing.
• UI-FOSSology users not created or not correct. The tests attempt to login to the UI using these logins, if the UI-FOSSology users do not exist or are not correct the logins will fail.
• fo-runTests link is not correct (it's either not the correct name or it's pointing to the wrong source tree). If it points to the wrong source tree, there may be permission problems or you might not run the correct set of tests.

Feel free to add your own to the list!

• lots of ok's when they pass.
• most/all tests print they are starting this should help when things fail
• if the tests fail there is often output to try to help understand the failure. There is a trace that reports the line number where the failure occurred, and the module it occurred in. There is also a method trace. If a php excepton occurs they are also reported.
• Failures will occur in groups. This grouping occurs because once an assertion fails, the following assertions will most likely also fail. So, focus on the first failure. When that is fixed, the others should pass.
• If there is a failure
  • Determine the test and what the test was doing at the time of the failure. Usually the failure message will give a lot of hints.
  • IMPORTANT: go to the web and display the page that was under test. This is a quick way to help determine if it's a defect in fossology or a test failure. For example, a test failed, the failure made no sense, why couldn't it pick a folder in the selection pull down? Hey, look at that!, the pull down menu is broken! The test found the error. Since the menu was broke, there was no folder_pk's to find in the XHTML.
  • Can the failure be reproduced by typing in what the test was doing?
• Tests are not very interesting if there is not some uploads to examine. So, some tests do depend on things being uploaded. Is the test failing because the material is not uploaded? Good tests will check first, but you can't count on it.
• The test could be broken.

The initial install script performs the following steps:

• Creates a symbolic link in /usr/local/bin for fo-runTests. This symlink will point back to the sources where the install script is run. This is needed by the install script itself. If this step fails, parts of the install will fail.
• Checks to see if simpletest is installed in /usr/local, if not attempts to wget it and install it.
• Creates the test users fosstester and noemail. The fosstester user is used to run the test suite.
• Loads test data for use by the tests from the sources and Web into the fosstester's home directory into three sub directories, archives, licenses and public_html.
• Other support files are also loaded into the fosstester account. The install script attempts to adjust the servers file in the .subversion directory so that fosstester can check out the source. This is required.
• UI users called fosstester and noemail are created in FOSSology.
• Runs the configTestEnv.php script. This creates a php include file that the tests need to run and is a required step.

The steps below are for use when the steps need to be run individually or if something does not work in the inital install and needs to be redone.

This step must be done by a user with sudo access. Make sure to cd to the correct source location and to the tests directory in the sources (not the ui/tests). Run the script.

sudo ./lnfo-runTests.php

• The Install.php script checks to see if simpletest is in /usr/local and if not it attempts to install it there. If for some reason that does not work, use the example below to install in into /usr/local by hand.

• Install simpletest into /usr/local. For now we will use the version from the site as the debian version is older. When the debian package is updated, we'll switch to that. For example:

wget http://downloads.sourceforge.net/simpletest/simpletest_1.0.1.tar.gz
sudo cp simpletest_1.0.1.tar.gz /usr/local
cd /usr/local
sudo tar -xvf simpletest_1.0.1.tar.gz

As sudo run the script in tests:

sudo ./CreateTestUser.sh

As a sudo user run the script in tests

sudo installTestData.sh

Before the script can be run, the TestEnvironment file must be changed so the user is fossy, then the script is run and then change the TestEnvironment file back (either edit directly or reurn script).

./configTestEnv.php 'http://mysys.domain/repo/' fossy fossy fo-runTests -l createUIUsers.php or ./fo-runTests.php -l createUIUsers.php ./configTestEnv.php 'http://mysys.domain/repo/' fosstester fosstester

NOTE This section needs some work. It's correct, but was written before the Initial Install section was written. Kept in here for now.

• check out the fossology source. Typically it can be checked out to a repo machine. The tests should be run on the machine where email will be delivered locally or the email notification tests will fail. This is typically the repo or web server machine.

• Run the config scripts to create the test users (fosstester and noemail) and load test data into the test user account.

cd ….path-to-sources/fossology/tests

sudo ./Install.php

• log out and login as the test user fosstester. Check out the sources into the fosstester home directory.
  • make sure you are in the test directory and run configTestEnv.php <http to repo> user passwd Important if this step is not done, most if not all of the tests will fail, as they will not be able to login to the repo.

For example:

cd ...path-to-sources/fossology/tests/
configTestEnv.php http://fluffy.ostt/repo/ fosstester fosstester
the url must end in a / and the path should be to the repo you want to test.  For example, repo/ could be ~rando/ui/

The above step sets up an include file called TestEnvironment.php. The script puts the file in the current directory.

Make sure that the file TestEnvironment.php exists in the directory …path-to-sources/fossology/tests.

• reinstall fo-runTests to point to this copy. This step will adjust the symlink in /usr/local/bin to point to the source tree in ~fosstester. This step will need to be done by a user who has sudo access. The test user fosstester is not in the sudoers file and so can't make this link. (This is the chicken and egg issue in test setup).

<as a user with sudo access...>
cd ...path-to-sources/fossology/tests/
sudo lnfo-runTests.php

The first suite is call SiteTests and are very light weight. SiteTests are designed to run right after an install and initialization of the repo. They verify that the site can be logged into and that the main important menus are operational. It does no real functional testing other than logging in and verify you can get to a real sub menu. To run the suite:

cd to ...path-to-sources/fossology/ui/tests/SiteTests
runSiteTests.php 

BasicTests are functional tests for the UI that do not depend on uploaded files. To run them:

cd to ...path-to-sources/fossology/ui/tests/BasicTests
runBasicTests.php

NOTE the step below is no longer needed, testFOSSology script takes care of it.

Important Before running verify tests upload the required material.

cd to ...path-to-sources/fossology/tests/
./uploadTestData.php

The script above will upload all the files that the Verify tests depend on. Uploading from a url must work for this script to complete successfully. The script is actually a simpletest, so if it fails it will tell you why.

VerifyTests are functional tests for the UI that depend on uploaded files. The tests should first check to see if the archive they depend on is uploaded. If it is not, the tests should fail. To run them:

cd to ...path-to-sources/fossology/ui/tests/VerifyTests
runVerifyTests.php

The verify tests are not very smart. They assume that all the test suites were run. There may be failures if only selected suites are run and then the verify suite is run.