Wednesday, January 18, 2017

Packet, Transaction, or Protocol Programming V1.7

1.     The main difference between traditional and packet/protocol programming

The protocol programming language like PLEX-C doesn’t offer function calls. A program module/block sends a SIGNAL with data to another block explicitly or all blocks that waits for this particular SIGNAL with expected data.
SIGNAL with data is like a PROCEDURE in traditional programming languages such as JAVA, C++. However the signal, when sent, is logged in the CPU processing queues and waited to be dispatched to destination blocks. Thus there is no looping at a block to wait for an expected function/procedure calls with data.

For example, at an Internet PORT’s interface, where a packet comes; we could dispatch the packet to correct programs by sending a SIGNAL with data; in order to do this, an application must register a SIGNAL at the interface, thus the module would dispatch the packet automatically to its destination as long as the conditions met. This SIGNAL could wake up an application in the form such as a process. Currently a program must be looping at a PORT indefinitely to catch a packet, i.e. using a lot of processing power of a PC.

2.     Sample of a packet analyzer and filter

Diagram 1. Examining incoming packets before delivering to a destination program

In the diagram above, all incoming packets would be dispatched to a relevant subsystem for processing in order to determine if the packet is legitimate. If the packet was harmful, it would be logged in database as well as triggering a report to attendance to take appropriate actions.
Basically the Packet Filter Manager would examine types of an incoming packet, and it sends a SIGNAL with relevant data to correct subsystem for validation. The same packet would also be sent to Unpacker module/block for unpacking in the form of a SIGNAL.
Of course programmer could implement
-         Dispatching Interface block to make their subsystem well defined and declared to other subsystem using OOD or API concepts.
-         Library with clear Dispatching Interface to share common functions.
The unpacker’s functionality should be a common task used by many of the subsystems in the diagram. Thus it could use a common library function.

3.     Dispatching interface for protocol programming languages


The entry would be a signal receiving interface with specified/required signal data, where the request would be dispatched to a specific block or module within its sub-system. However the returning data or result would be done by another signal or returning function via the same interface. How data would be communicated, users must refer to the protocol specification. Of course, the interface would also be documented with the meaning of each data in a signal. Developers may attempt to specify the possible returning signal(s) as there may be many possible outcomes for a request. Protocol specification would be a better option.
If an interface module is large, developer could introduce another interface module to access its sub-system.
The interface module should be updated to add more entries or returning signals for other sub-systems. However modifying an existing entry or returning signals to incorporate more data/results would require regression testing or updating of all existing functionality, i.e. lots of work and time as well as potential bugs.

This interface module could be used to be an interface to common functions. This is similar to building a library of functionality.

For PLEX-C, developers could add more new signal data at the end of a signal to carry new information; however modifying an existing signal data would require more investigation to exiting functionality or programming codes. The corresponding protocol specification and dispatching interface’s documentation must be updated for new message data.
4.     Sample program to receive an incoming packet

Assuming that the unpacker had removed all irrelevant headers of a packet before delivering to a program, the company XYZ had registered an expected SIGNAL at Registration Interface Manager as XYZReportingTask.
In a dormant process module, it could be programmed as (samples to demonstrate the structure or format of receiving and sending SIGNALs)
-------------------------------

RECEIVE XYZReportingTask OriginatingAddress,
                                                        NumberOfReport,
                                                        NameOfReport,
                                                        FieldData1,
                                                        FieldData2,
                                                        Etc;

GOTO ProcessingReport; ! redirect signal processing to a label called
                                                     ! ProcessingReport
                                                     ! by using label, programmers could place all expected
                                                     ! receiving SIGNALS and data at one place in a block. Just
                                                     ! for organizing and ease in debugging and reviewing a
                                                     ! programming block

ProcessingReport:    ! this is the label in this block to process data coming
                                       ! from the signal XYZReportingTask


-         Processing the report data or further unpacking in forms recognized by XYZ system.
-         Log data in database as needed
IF all data were as expected and acceptable
SEND TO PrinterBlock StartPrintingReport CompanyName, Address, PhoneNumber, InvoiceID, etc;
      ELSE  
SEND SpecialAnalyzer CompanyName, Address, PhoneNumber, InvoiceID, ErrorCode, etc;
      ENDIF;

EXIT;  ! Release the central processor for other processing tasks
-------------------------------
The XYZReportingTask, StartPrintingReport, and SpecialAnalyzer are SIGNAL names. All variables followed those SIGNALs are signal data. The program would process signal data with programming statements similar to any traditional programming languages including INSERT those in database.
The XYZReportingTask signal came from the central processor queue, which was originally inserted in by the Delivery Manager block. This program didn’t loop for an incoming message at an Internet PORT. The Delivery Manager must include the destination Block Name, Signal Name as well as signal data to this programming block.
This programing block sends the signal StartPrintingReport to the central processor queue, which would be delivering to the programming block PrinterBlock for printing a good report.
This programing block sends the signal SpecialAnalyzer to any programming block that expects this signal with data for analyzing errors.
All internal/block SIGNALS had been registered in the registration system at compilation, thus the central processor knows where or the block should receive a particular signal correctly.

5.     Central Processor

Basically the central processor has signal queues with associated data. The queue is FIFO rule. There are several priority queues to process data according to level of importance, e.g. system may insert their signals in Priority Queue A. Queue Priority B is for normal tasks. Priority Queue C is for lower level tasks such as printing.
The queued SIGNALs would be delivered to a particular programming block or be broadcast to many blocks waiting for the same signal with data.
There must be syntax for a packet/protocol programming language to specify a method to send or receive a SIGNAL.

6.     Tracing/debugging issues on live system and patching

Ericsson’s APZ (central processor) offers a packet programming language called PLEX-C. This language also uses a specific syntax for SIGNALs sending and receiving. I used to work over there, but I didn’t remember the syntax.
The system offered a tool called TEST SYSTEM. Testers could activate this tool to see signal names and signal data passing between programming blocks. By analyzing passing data in a signal, testers could determine if a block analyzed and passed data correctly or not.
By checking signal data at some particular points, testers could narrow down issues quickly and figure out a solution by writing assembly patches/codes to load on a live switch. Other testers could use these patches as a temporary solution to continue their testing activity. The permanent solution would be implemented by high level programming language such as PLEX-C, compiled into assembly codes, and then loaded on the central processor (APZ) later.
Usually the process to implement a solution in high level language would take more time and scheduled at a determined phase. Thus temporary patches were useful to keep testing activity going.

7.     Parallel processors

The Ericsson’s APZ has 2 main processors, which has many CPU in each processor. One of them is the executive processor and the other is standby processor.
The executive processor is the live node to handle real time data and processing. Data is also passed to the standby processor as a mirror database and backup. If the executive processor is faulty for a reason, the standby processor would take over as the executive processor.
In normal circumstances, new programming codes would be compiled into assembly languages and loaded on the standby processor. Technical support would trigger a process to transfer real-time data on the executive processor to new codes and database on the standby processor before an automatic switching processors happened, i.e. standby processor with new codes becomes a live node.
-         The old codes and database were still on the standby side. If the executive side was faulty for a reason, technical support would switch the standby side with previous codes back to executive side as a live node.
-         Both processors would be brought back to run in parallel later if everything was normal.
APZ is used in telecom switches, which couldn’t be down for more than 20 minutes (decade ago allowed time) a year. Thus the process to switch processors over was short and carried out in maintenance windows, e.g. 12AM to 5AM.

8.     Sample of tracing or verifying the SMTP subsystem in the diagram above


Tester could initiate TRACE of
-         Incoming signals to the Packet Filter Manager, i.e. incoming packets

-         Incoming signals to the interface block of SMTP Subsystem, i.e. delivered packets from the Packet Filter Manager

The SMTP Subsystem could log some data into database by a programming block, e.g. by block UpdateData.

-         Trace a variable in the block UpdateData, e.g. External_OPC variable, that would change value before committing data into database

-         Outgoing signals from the block UpdateData

-         Outgoing signals from the interface block of SMTP Subsystem, i.e. packets out from the SMTP Subsystem

-         Incoming signals to the interface block of Report Manager

-         Trace a variable in a block of Report Manager, e.g. ErrorCode, that would change value before committing data into database

-         Outgoing signals from the interface block of Delivery Manager

Use another PC connecting to this system using an Ethernet cable or WiFi in order to send packets.

There should be a tool for testers to assemble a file of packets in a proper protocol format. Testers could create many packets with different scenarios. Send a good/bad packet or a set of packets in a file with different errors.

Based on the packet sent and signal data collected in the above traces, testers should be able to determine if the overall SMTP Subsystem have been functioned correctly.

-         There should be signal descriptions associated with the system to interpret the meaning of each signal data in a signal to/from a block.

-         If the values of the traced variables above were set before or after a signal leaving/coming to a block as expected, this would mean the data had been committed to database at proper time. Verify database to ensure its value and format.

Testers could also initiate a TRACE on the Outgoing Packet Manager in case the PC request the originating node to resend an erroneous packet.
If errors or unexpected behaviors have been detected, testers could trace many blocks in the subsystem to figure out the location of bug(s). Of course, assembly patches could be implemented to load on the node to correct issues.

9.  Notes about trace tools Ericsson’s APZ or APY and a packet analyzer

-         Couple with special commands, APY could be used to implement the packet analyzer above to track down illegitimate activity, to record, and to bring those nodes down. Notes at https://peterboroughtechservices.blogspot.ca/p/tracking-adevice-using-internet-a.html
The detailed syntax must be provided by the equipment provider.

10.     Packet Analyzer

I had developed the concept of packet analyzer or special firewall earlier. Notes could be found at https://peterboroughtechservices.blogspot.ca/p/my-special-firewall.html. There is a link in the notes to download my documents.

The packet analyzer could be implemented to protect a PC or network. It is a little bit different than the current antivirus programs in the market.  The concept was based on analyzing incoming and outgoing packets as well as the protocol specification for each Internet protocol and current activity within a PC or networks.
I like the unattended mode in my document as hackers had threatened to enter my PC when I was away from my PC, which was connected to the Internet while I walked away.

-         Basically the Packet Analyzer would detect if a PC or network had no user activity such as typing on a keyboard.

-         Current activity such as downloading files using http or ftp

-         If there was no activity or PC’s screen saver was activated, the Packet Analyzer could shutdown Internet connection, but allowing current [pre-defined] activity to continue until completed. After completion, the Internet was completely off. Some activity wouldn’t be allowed while users were away from their terminal, btw, as a pre-defined rule.

-         If user unlocked the screen again, the Packet Analyzer would restore Internet connection.
Personally I don’t want any software applications that I am using access or open a Port of the Internet communication, e.g.
-         Microsoft Office or Adobe’s PDF unless it is opened on a web browser. End users didn’t buy an Internet or server version of MS Word, thus all edits should be local.

-         Updates could be done periodically, but shouldn’t have a permanent connection to an Internet port. Perhaps at installation, user would accept the update feature, thus an Internet defender, e.g. Packet Analyzer, would let it open port sometimes for checking and downloading. Start of downloading updates should ask for user’s permission. We don’t want hackers faking a company to upload harmful software to our PC.


-         Having a Port open permanently would give hackers an opportunity to enter our PC.


-         Internet games, Internet applications, and Internet streaming applications are within another category.


Recently hackers had faked downloads from well-known companies to a user’s PC. The actually downloads contained harmful virus or Trojan Horses.
The Packet Analyzer concept is simple, but software product providers must analyze a protocol and legitimate activity in order to provide a best Internet defender. Antivirus companies could use this concept as well as their existing features such as detecting known hacking programs to enhance a PC security.
Law enforcement would be able to develop all features of a packet analyzer including special measures such as tracking and bringing down attackers.

Private packet analyzer for users would focus on the features to defend user’s PC or networks. It could log intruder’s data and reported to local authority later. Unless the private companies had permissions by law enforcement, they shouldn’t attempt to launch special measures against intruders as this may involve special rules and agreement among local law enforcement and many countries.

11.     Features should be included in a personal PC’s Internet Defender



The Internet Defender or Packet Analyzer should offer an interface for users to do the following simple things:

-         Block all Internet ports



-         Open http port



-         Start a program within the PC – of course an application that allows accessing Internet using http, e.g. a web based game or a web based chat room.


-         If the application worked fine. Only http port is needed.


-         If it didn’t work? The Internet Defender should know which port an external application is trying to open or communicate, i.e. check to see if the target server relayed answers or response to another port of this PC.

-         Internet Defender would prompt the user to open that port manually, if a request or outgoing TCP/IP packet had been detected earlier.
If the Internet Defender could also offer blocking an (troublesome) Internet protocol, it would be easier, too. In this case, users would block some protocols regardless of ports. If a user allowed a protocol regardless of ports, it may open many gates for intruders.
Note that some applications must be originated in a PC in order to get response from an external source, i.e. responses or messages came to a PC without a PC’s originating request would be illegitimated.
In contrast, some process could be started by an outsider such as incoming email or use of incoming SMTP.

12.     System Security

The Internet Defender or Packet Analyzer are screening and validating Internet messages at packet layer coupled with current activity in a PC or network. Thus the system is quite secure.
The only concern would be lower layers of an OS such as transmission layer. Those lower layers should only be accepting electrical signals to pack those into TCP/IP or UDP packets for processing OR unpacking a TCP/IP or UDP messages into electrical signals for transmission. No funny or hidden functionality should be implemented.

13. Register Interface Manager

The signal XYZReportingTask with data D1, D2, D3, …, Dn described in section 4 and diagram 1 will be delivered to its dormant process upon receiving by system thorough Internet communications. With this implementation, developers could avoid creating a forever loop to catch Internet data at an Internet port. The dormant process would be wakened up to handle data as needed, i.e. avoid wasting CPU processing power and RAM.

There would be several ways to implement and enforce dormant process to save processing power of a system, e.g.

·        For time slot applications such as collecting data in a predefined interval, e.g. 12 pm-1 pm, the processing application would be wakened up by an Internet signal via Register Interface Manager. It processes data between the allocated interval, and complete required tasks received until 1 pm. After processing the last signal, it would be back in sleep mode as coded by application developers.
·        Intermittent Internet process, i.e. processing Internet messages randomly, the application would be wakened up and processed data via a signal from Register Interface Manager. After processing data, it would be back to sleep mode, if idle for a couple of minutes as coded by application developers.
·        OS could determine any applications in idle mode for a predefined time interval; it will put those applications or processes in sleep mode.

Those suggested methods would help performance of a computer system.

Tuesday, January 17, 2017

Software System Development Strategies v1.1

1.     System Design

A good protocol or system design document is IS-41 Rev B standard used for TDMA wireless telecom network.
System designers should use diagrams to show flow of data between internal components/subsystems similar to HLR, MSC, RBS, MS, etc. in the IS-41 standard.

Each sub-system must be well defined for functionality and offered access to its functionality via an interface module.

- The interface should be well declared as in the API section if traditional programming languages were used in software development.

- If a protocol like programming language were used such as Ericsson’s PLEX-C, the interface would be like a dispatcher, which also offers the access to its functionality. See the dispatching interface for details.

2.     IS-41 Rev B standard

IS-41 protocol spec should be a reference to design interface and data exchanges. The specification has a good way to present how elements have passed data as well as details of each data in a message in another chapter. A diagram also offers an overview of a feature.
If a designer understood how the IS-41 spec organized, they would be able to present their system in a similar way, i.e. similar structure of documentation.

3.     API for traditional programming languages

API for traditional programming languages should describe function interface as well as meaning of parameters, variable types, and input and output data types
- An interface module would be the gateway to sub functions. All requests to access its functionality would go through the interface module. This is like a sub-system. A library of function would include only those interface modules for other developers to use.
- Instead of modifying an existing function to add more data/parameters, it would be easier to add a new function in the interface. It would be less work to retest existing functionality involving the "modified" functions.

4.     Dispatching interface for protocol programming languages
The entry would be a signal receiving interface with specified/required signal data, where the request would be dispatched to a specific block or module within its sub-system. However the returning data or result would be done by another signal or returning function via the same interface. How data would be communicated, users must refer to the protocol specification. Of course, the interface would also be documented with the meaning of each data in a signal. Developers may attempt to specify the possible returning signal(s) as there may be many possible outcomes for a request. Protocol specification would be a better option.


If an interface module is large, developer could introduce another interface module to access its sub-system.
The interface module should be updated to add more entries or returning signals for other sub-systems. However modifying an existing entry or returning signals to incorporate more data/results would require regression testing or updating of all existing functionality, i.e. lots of work and time as well as potential bugs.

This interface module could be used to be an interface to common functions. This is similar to building a library of functionality.
For PLEX-C, developers could add more new signal data at the end of a signal to carry new information; however modifying an existing signal data would require more investigation to exiting functionality or programming codes. The corresponding protocol specification and dispatching interface’s documentation must be updated for new message data.

5.     TEST CASE DOCUMENT
Test case document could be used with Microsoft Word to detail actions and results for each test step. The document also includes diagram(s) to illustrate the system or subsystem under tests.


- The first page of this document would briefly describe the test object as well as its revision. The main reference of the test object such as requirement specification, function specification, function description, technical specification, or protocol specification should be stated.
- Basically a test case has heading to include: requirement/specification section title including its document revision; brief description of a test case's goal - this could be described by a good test case heading. This is to ensure that all requirement or functionality have been covered in a test specification.

- The body of a test case would be 2 columns, i.e. a column for action steps and a column for corresponding results to verify.


- The requirement or technical specification’s section title would help in tracking of specification down to testing phases. If developers didn’t change a section title in future revisions, the test case’s reference would be valid for a long time.

6.     TEST CASE HEADING
A test case heading must be carefully written, thus tester would know what to do by reading the test case heading in the Test Report. Basically it should describe the action end-to-end. Tester would need to read the test case for detailed of data to be checked, btw.

7.     End to End Test

- Based on feature specification, we could come up with test cases to run end to end test to make sure that all key paths were corrected.
- Smaller test cases would cover "error handling" or logging in cases of unexpected errors happened.

8.     Sampling of check points at key interfaces
- During an end-to-end test, if an error happened, testers should quickly check all key interfaces to ensure that data had been passed correctly at that point or not. This is a method to narrow down issues.
- This could be done by checking message data based on a protocol specification or data logged in database at each stage.

9.     TEST REPORT
Test Report must be created for a test case document above. This document would include a column for the headings of the test case document, i.e. all of test cases to be executed. Another column would be the result P/F (passed/failed). Another column would be comments or notes about a test case such as failure reasons, special environment setups, etc. This document could be documented in an Excel format.


- Each test report would have a main page, e.g. use Excel, to include the number of test cases to be tested as well as number of test cases passed/failed.
- The main page would also give a brief description of the document.
- This main page could also summarize issues and remaining tasks.
- All failed test case should include the ID of a defect report in the comments/notes section.
- After a defect fixed, the tester should retest the corresponding failed case and update the Test Report document accordingly, e.g. changing F indicator to P indicator.
- All failed cases above would be logged in a defect tracking tool with test case ID or section heading and revision of a test case document.

10.                       DOCUMENT VERSION CONTROL
All documents must be stored using a revision tracking tool such as SVN. On the description upon checking in users should specify the revision as well as a brief description of the document. This would help in browsing of documents in the SVN to search/download a correct revision for reference quickly.

11.                       TEST PLAN

Test Plan document should base on a requirement specification to specify all new test case documents as well as re-test, regression test case documents. It should briefly specify what should be tested based on requirement. Testers would need to read specification to come up with detailed test cases. Discussion with developers to come up with regression test cases could be required.

12.                       SIMPLE FEATURE TEST
For a simple feature, we could create a master Test Report document in Excel. There should be only 1 new test case document and a few regression test case documents. The main tab of the Test Report would be headings of the new test case document, and subsequent tabs include headings of regression test case documents.


- Basically all new test cases must be executed. The remaining tab should mark the regression test cases. Thus this would show the total number of new and regression test cases in the main page for time allocation.
- Usually a new test case would require more time to figure out of environment settings and trouble shooting. Regression tests should go smoothly with all execution steps validated earlier.
- If we could estimate time required per test case (new or regression), the total time required for a test object could be obtained from a Test Report.
- The main page of the Test Report should include statistics of the test case progress of results.

13.                       SELECTION OF REGRESSION TEST CASES
Selection of regression test cases could be done by both testers and developers of related functions impacted by the new feature. Based on experience, testers and developers would select key test cases to be regression tested. Some companies asked for 30% of test cases to be regression tested.


- Basically testers should look into the flow charts or sequence diagrams corresponding to the regression test object, the test case covered "end-to-end" of the flow chart must be selected. The cases covered other branches could be randomly selected unless developers suspected something may happen to a specific case.

14.                       HANDLING OF A DEFECT REPORT
In case of resolving a defect report, developer should state "related test case(s)" should be regression tested. Of course developer shouldn't say to retest the whole test object.

- Anyway a bug reported in the field would be bad for both developers and testers, thus both sides should work together. Having many test cases retested would require more test time to cause delay of a product deployment that would trigger complaints by managers too.

15.                       DEFECT TRACKING REPORT TOOL

Any defect must be logged in a defect tracking tool. Based on the system design diagram, many categories could be created to log a bug into a correct sub-system or a module.
If a system is large, and tester was not able to track down a defect into a module (or black box tests), tester should log their found defect against a subsystem. There should be a test tool to help tracking down a bug into a subsystem or a module such as dispatching interface or an API. With PLEX-C, tester could use test system to trace data passing between modules/blocks or sub-systems; thus it’s easier to log a bug against a target.
If a bug is logged against a sub-system, it is up to developers to move or redirect it to a correct programming module/block.
Of course, testers could log bugs found in documentation as well as a protocol specification.

16.                       DEFECT MANAGEMENT
As a manager, all open/close defects must be extracted into a report. With all defects categorized against sub-systems and modules, you should be able to tell which part of the entire system had been the most trouble part during development phase. With that information, action plan would be developed such as breaking up a sub-system into smaller sub-systems, better/detailed protocol specification, restructuring of a block or programing module, training necessary for a subsystem, etc.

At the end of a test cycle, manager could review all opening defect reports and evaluate those with developers and business analysts, if needed. All critical bugs that would make customers yelled or inconvenient must be fixed. Bugs that impact many subsystems must be fixed, so other subsystems could function as expected. Minor bugs or cosmetic bugs with work around could be delayed to next product development phase. Ideally all bugs must be fixed, but time is limited.

If system or products have been deployed to many customers, common bugs or complaints would have higher priority to be fixed. This is called Pareto Chart in 80-20 philosophy.

-         All field bugs must be logged by a technical support or a tester to an appropriate module or a sub system. It should include steps to reproduce.
-         To make a field bug different than an internal bug report, the subject field of the field (defect) report could include a tag such as [FR] before its usual heading.

-         If a special category used in a defect report tool, then a field bug must be registered with that category. It’s better to register a field bug along with regular bugs for ease of evaluating or estimate work to be done in a phase.

-         With the management or statistic function of a defect tracking tool, a manager could extract all bugs and figure out if a module or subsystem had many field bugs attached.

-         Usually field bugs or customer complaints are at higher priority to fix.