Software System Development Strategies v1.1

1.     System Design

A good protocol or system design document is IS-41 Rev B standard used for TDMA wireless telecom network.

System designers should use diagrams to show flow of data between internal components/subsystems similar to HLR, MSC, RBS, MS, etc. in the IS-41 standard.

Each sub-system must be well defined for functionality and offered access to its functionality via an interface module.

- The interface should be well declared as in the API section if traditional programming languages were used in software development.

- If a protocol like programming language were used such as Ericsson’s PLEX-C, the interface would be like a dispatcher, which also offers the access to its functionality. See the dispatching interface for details.

2.     IS-41 Rev B standard

IS-41 protocol spec should be a reference to design interface and data exchanges. The specification has a good way to present how elements have passed data as well as details of each data in a message in another chapter. A diagram also offers an overview of a feature.

If a designer understood how the IS-41 spec organized, they would be able to present their system in a similar way, i.e. similar structure of documentation.

3.     API for traditional programming languages

API for traditional programming languages should describe function interface as well as meaning of parameters, variable types, and input and output data types

- An interface module would be the gateway to sub functions. All requests to access its functionality would go through the interface module. This is like a sub-system. A library of function would include only those interface modules for other developers to use.

- Instead of modifying an existing function to add more data/parameters, it would be easier to add a new function in the interface. It would be less work to retest existing functionality involving the "modified" functions.

4.     Dispatching interface for protocol programming languages

The entry would be a signal receiving interface with specified/required signal data, where the request would be dispatched to a specific block or module within its sub-system. However the returning data or result would be done by another signal or returning function via the same interface. How data would be communicated, users must refer to the protocol specification. Of course, the interface would also be documented with the meaning of each data in a signal. Developers may attempt to specify the possible returning signal(s) as there may be many possible outcomes for a request. Protocol specification would be a better option.

If an interface module is large, developer could introduce another interface module to access its sub-system.

The interface module should be updated to add more entries or returning signals for other sub-systems. However modifying an existing entry or returning signals to incorporate more data/results would require regression testing or updating of all existing functionality, i.e. lots of work and time as well as potential bugs.

This interface module could be used to be an interface to common functions. This is similar to building a library of functionality.

For PLEX-C, developers could add more new signal data at the end of a signal to carry new information; however modifying an existing signal data would require more investigation to exiting functionality or programming codes. The corresponding protocol specification and dispatching interface’s documentation must be updated for new message data.

5.     TEST CASE DOCUMENT

Test case document could be used with Microsoft Word to detail actions and results for each test step. The document also includes diagram(s) to illustrate the system or subsystem under tests.

- The first page of this document would briefly describe the test object as well as its revision. The main reference of the test object such as requirement specification, function specification, function description, technical specification, or protocol specification should be stated.

- Basically a test case has heading to include: requirement/specification section title including its document revision; brief description of a test case's goal - this could be described by a good test case heading. This is to ensure that all requirement or functionality have been covered in a test specification.

- The body of a test case would be 2 columns, i.e. a column for action steps and a column for corresponding results to verify.

- The requirement or technical specification’s section title would help in tracking of specification down to testing phases. If developers didn’t change a section title in future revisions, the test case’s reference would be valid for a long time.

6.     TEST CASE HEADING

A test case heading must be carefully written, thus tester would know what to do by reading the test case heading in the Test Report. Basically it should describe the action end-to-end. Tester would need to read the test case for detailed of data to be checked, btw.

7.     End to End Test

- Based on feature specification, we could come up with test cases to run end to end test to make sure that all key paths were corrected.

- Smaller test cases would cover "error handling" or logging in cases of unexpected errors happened.

8.     Sampling of check points at key interfaces

- During an end-to-end test, if an error happened, testers should quickly check all key interfaces to ensure that data had been passed correctly at that point or not. This is a method to narrow down issues.

- This could be done by checking message data based on a protocol specification or data logged in database at each stage.

9.     TEST REPORT

Test Report must be created for a test case document above. This document would include a column for the headings of the test case document, i.e. all of test cases to be executed. Another column would be the result P/F (passed/failed). Another column would be comments or notes about a test case such as failure reasons, special environment setups, etc. This document could be documented in an Excel format.

- Each test report would have a main page, e.g. use Excel, to include the number of test cases to be tested as well as number of test cases passed/failed.

- The main page would also give a brief description of the document.

- This main page could also summarize issues and remaining tasks.

- All failed test case should include the ID of a defect report in the comments/notes section.

- After a defect fixed, the tester should retest the corresponding failed case and update the Test Report document accordingly, e.g. changing F indicator to P indicator.

- All failed cases above would be logged in a defect tracking tool with test case ID or section heading and revision of a test case document.

10.                       DOCUMENT VERSION CONTROL

All documents must be stored using a revision tracking tool such as SVN. On the description upon checking in users should specify the revision as well as a brief description of the document. This would help in browsing of documents in the SVN to search/download a correct revision for reference quickly.

11.                       TEST PLAN

Test Plan document should base on a requirement specification to specify all new test case documents as well as re-test, regression test case documents. It should briefly specify what should be tested based on requirement. Testers would need to read specification to come up with detailed test cases. Discussion with developers to come up with regression test cases could be required.

12.                       SIMPLE FEATURE TEST

For a simple feature, we could create a master Test Report document in Excel. There should be only 1 new test case document and a few regression test case documents. The main tab of the Test Report would be headings of the new test case document, and subsequent tabs include headings of regression test case documents.

- Basically all new test cases must be executed. The remaining tab should mark the regression test cases. Thus this would show the total number of new and regression test cases in the main page for time allocation.

- Usually a new test case would require more time to figure out of environment settings and trouble shooting. Regression tests should go smoothly with all execution steps validated earlier.

- If we could estimate time required per test case (new or regression), the total time required for a test object could be obtained from a Test Report.

- The main page of the Test Report should include statistics of the test case progress of results.

13.                       SELECTION OF REGRESSION TEST CASES

Selection of regression test cases could be done by both testers and developers of related functions impacted by the new feature. Based on experience, testers and developers would select key test cases to be regression tested. Some companies asked for 30% of test cases to be regression tested.

- Basically testers should look into the flow charts or sequence diagrams corresponding to the regression test object, the test case covered "end-to-end" of the flow chart must be selected. The cases covered other branches could be randomly selected unless developers suspected something may happen to a specific case.

14.                       HANDLING OF A DEFECT REPORT

In case of resolving a defect report, developer should state "related test case(s)" should be regression tested. Of course developer shouldn't say to retest the whole test object.

- Anyway a bug reported in the field would be bad for both developers and testers, thus both sides should work together. Having many test cases retested would require more test time to cause delay of a product deployment that would trigger complaints by managers too.

15.                       DEFECT TRACKING REPORT TOOL

Any defect must be logged in a defect tracking tool. Based on the system design diagram, many categories could be created to log a bug into a correct sub-system or a module.

If a system is large, and tester was not able to track down a defect into a module (or black box tests), tester should log their found defect against a subsystem. There should be a test tool to help tracking down a bug into a subsystem or a module such as dispatching interface or an API. With PLEX-C, tester could use test system to trace data passing between modules/blocks or sub-systems; thus it’s easier to log a bug against a target.

If a bug is logged against a sub-system, it is up to developers to move or redirect it to a correct programming module/block.

Of course, testers could log bugs found in documentation as well as a protocol specification.

16.                       DEFECT MANAGEMENT

As a manager, all open/close defects must be extracted into a report. With all defects categorized against sub-systems and modules, you should be able to tell which part of the entire system had been the most trouble part during development phase. With that information, action plan would be developed such as breaking up a sub-system into smaller sub-systems, better/detailed protocol specification, restructuring of a block or programing module, training necessary for a subsystem, etc.

At the end of a test cycle, manager could review all opening defect reports and evaluate those with developers and business analysts, if needed. All critical bugs that would make customers yelled or inconvenient must be fixed. Bugs that impact many subsystems must be fixed, so other subsystems could function as expected. Minor bugs or cosmetic bugs with work around could be delayed to next product development phase. Ideally all bugs must be fixed, but time is limited.

If system or products have been deployed to many customers, common bugs or complaints would have higher priority to be fixed. This is called Pareto Chart in 80-20 philosophy.

-         All field bugs must be logged by a technical support or a tester to an appropriate module or a sub system. It should include steps to reproduce.

-         To make a field bug different than an internal bug report, the subject field of the field (defect) report could include a tag such as [FR] before its usual heading.

-         If a special category used in a defect report tool, then a field bug must be registered with that category. It’s better to register a field bug along with regular bugs for ease of evaluating or estimate work to be done in a phase.

-         With the management or statistic function of a defect tracking tool, a manager could extract all bugs and figure out if a module or subsystem had many field bugs attached.

-         Usually field bugs or customer complaints are at higher priority to fix.