![]() |
Home | Documentation |
Test Messenger
updated Mon Feb 12 2018 by Robert van Engelen
|
It is important to test Web services and client applications as part of a quality assurance process. Furthermore, the functional and non-functional requirements of an application should be verified and tested before the application is deployed in the field. Testing Web services with randomized message exchange patterns can help to pinpoint problems early on, before applications are deployed.
The benefit of the Test Messenger is the ability to use a set of message templates for unit and regression testing, with specific randomization of XML attribute and element values in the templates as specified.
The Test Messenger is an application to test Web services by sending (randomized) XML test messages to services and client applications. The Test Messenger application is simply executed from the command line or from a shell script, such as a script intended to test services and clients. No coding is required to use the Test Messenger.
The testmsgr Test Messenger application is located in the gSOAP distribution package under gsoap/samples/testmsgr
and is built with:
cd gsoap/samples/testmsgr make
To generate randomized XML messages with the Test Messenger, a SOAP/XML message template is required. The Test Messenger populates the template with random values to send the randomly populated SOAP/XML request message to a service for Server-side testing . Or the Test Messenger can be started as a stand-alone server to return the randomly populated response message template to a client for Client-side testing .
You can create a SOAP/XML message template manually or auto-generate it with soapcpp2 option -g
. For example, to auto-generate XML sample message templates from an interface header file file.h
(e.g. generated with wsdl2h from WSDLs), use soapcpp2 with option -g
:
soapcpp2 -g file.h
The soapcpp2 tool generates sample messages for the service operations defined in file.h
. Option -g
changes the sample message to message templates that contain placeholders of the form %[...]%
. A leading ???
qualifier indicates optional elements and attributes. Repeated elements are indicated and enclosed as child elements in <__REPEAT>
... </__REPEAT>
. Element selections are indicated and enclosed in <__SELECT>
... </__SELECT>
.
A placeholder value %[...]%
specifies an XSD-related type, enumeration, or numeric range. These placeholders are populated by the Test Messenger using random values that conform to the placeholder's specified value type.
While the sample messages generated by soapcpp2 (without option -g
) can be used to test clients and services as explained in the gSOAP user guide, the problem is that the content of the sample messages is always fixed. This provides only very limited testing capabilities. By contrast, the Test Messenger application generates randomized messages based on SOAP/XML message templates for enhanced testing of client and service applications.
The Test Messenger is intended to test common messaging scenarios that are defined by SOAP/XML message templates. The Test Messenger does not add (random) XML attributes and elements that are not already in the message template. It can randomly remove attributes and elements with option -p
to test the robustness of an application to consuming partial and possibly invalid XML messages (invalid according to the schema of the XML message).
The Test Messenger has several options to further randomize the messages by randomly removing attributes and elements at a specific rate of chance, or to add random spacing to XML attribute and element values, and to continuously test a service until it fails.
To demonstrate the Test Messenger, assume we have the following services declared in the interface file.h
:
We run soapcpp2 on this interface file.h
with option -g
:
soapcpp2 -g file.h
This produces the following SOAP/XML template files among other source code files:
example.upload.req.xml
XML template upload request messageexample.download.req.xml
XML template download request messageexample.upload.res.xml
XML template upload response messageexample.download.res.xml
XML template download response messageThe example.upload.req.xml
file looks like this:
Note that XML values in this template are represented by value placeholders of the form %[...]%
. Also note that the person
element, the id
attribute and miles
are optional.
A leading ???
qualifier indicates optional elements and attributes, when the ???
qualifier appears after the opening tag of an element (i.e. in front of the element value or child elements) and in attribute values, respectively. For the full list of placeholders that can be used, see Placeholder values .
This message template is read, populated and then sent to the standard output by testmsgr as follows:
./testmsgr -r1 -i example.upload.req.xml
This produced a randomized SOAP/XML message to standard output from the example.upload.req.xml
template message. Option -r1
sets the random seed to 1
. Any seed value between 1 and 2147483647 can be specified with -r
. Option -i
displays the randomized XML with indentation.
To randomly retain or discard optional elements and attributes, use option -q
with a specified percentage to retain optional elements and attributes in the generated XML message:
./testmsgr -r1 -q50 -i example.upload.req.xml
This retains the optional element person
, the id
attribute, and the miles
element with a 50% chance each.
To send the randomized SOAP/XML message to a service endpoint, pass the endpoint URL as the last argument to the testmsgr command:
./testmsgr -r1 -i example.upload.req.xml http://www.example.com/service
For example, we can let soapcpp2 generate a simple echo test service to which we can send messages. Since we auto-generate a simple echo test service we have no server-side logic to return meaningful values for this example.
First we invoke soapcpp2 with option -T
to generate the server-side code for a simple test service:
soapcpp2 -g -SL -T file.h
This generates the SOAP/XML message templates and the following source code files:
example.wsdl
a WSDL to publish the serviceexample.nsmap
an XML namespace mapping table to include in your source codesoapStub.h
annotated copy of the source interface filesoapH.h
serialization functionssoapC.cpp
serialization functionssoapServer.cpp
service request dispatchersoapTester.cpp
auto-test echo serverNext, we compile these files together with the gSOAP library and the custom serializer for struct tm
to build a simple echo server:
c++ -I. -o echoserver soapTester.cpp soapServer.cpp soapC.cpp stdsoap2.cpp custom/struct_tm.c
Now run the server. For example with arguments 8080 as the port to bind to and 8192 to set the SOAP_XML_INDENT
engine context flag:
./echoserver 8192 8080
Finally, let's fire a randomized message to the simple echo service, using the auto-generated example.upload.req.xml
message template:
./testmsgr -r1 example.upload.req.xml http://localhost:8080
The testmsgr displays the response received from the server, which for the simple echo service upload request is the upload response with an id element.
To send a continuous stream of randomized message to the server, use option -c
:
./testmsgr -c -r1 example.upload.req.xml http://localhost:8080
This continous until the service fails. A test is reproduced by using its random seed. So if the service fails you can replicate the message with the random seed and option -r
with this seed value.
The Test Messenger application can also send pruned XML messages with option -p
. Pruning means that attributes and elements are randomly discarded, which allows for testing the robustness of the service against incomplete and malformed SOAP/XML content:
./testmsgr -c -p80 -r1 example.upload.req.xml http://localhost:8080
Option -p80
means that 80% is retained and 20% of attributes and elements are randomly discarded from the SOAP/XML template example.upload.req.xml
.
When testing the auto-test echo service, the echo service displays failed requests and is silent for successful requests.
For the full list of testmsgr options, see Options .
To build testmsgr, install gSOAP and build testmsgr as follows:
cd gsoap/samples/testmsgr make testmsgr
This builds the command-line tool testmsgr in gsoap/samples/testmsgr
from where you can use it and/or copy it for use with your projects.
If you do not have the samples built, you can use soapcpp2 (or soapcpp2.exe in gsoap/bin/win32
) from the command line to generate the C++ code required for testmsgr:
cd gsoap/samples/testmsgr soapcpp2 -CSL ../../import/dom.h c++ -I../.. -o testmsgr testmsgr.cpp soapC.cpp ../../dom.cpp ../../stdsoap2.cpp
This builds the testmsgr command-line tool.
The testmsgr command takes a set of options, a SOAP/XML message file or a message template file (i.e. to randomize with option -r
), and an optional service endpoint URL.
To test services, run testmsgr against the service endpoint URL, for example when the service runs on the localhost on port 8080:
./testmsgr -r99 example.upload.req.xml http://localhost:8080
When using https endpoints, you must build testmsgr with compile-time flag WITH_OPENSSL
and link OpenSSL as follows:
soapcpp2 -CSL ../../import/dom.h c++ -DWITH_OPENSSL -o testmsgr testmsgr.cpp soapC.cpp ../../dom.cpp ../../stdsoap2.cpp -lssl -lcrypto
For CGI-based services that read SOAP/XML requests from standard input you can redirect the (randomized) message to the CGI-based service. For example:
./testmsgr -r99 example.upload.req.xml | ./echoserver
where echoserver is a CGI-based service (the echoserver built in the Introduction uses CGI by default).
To test clients, run testmsgr as a stand-alone server by specifying a port number instead of a service endpoint URL:
./testmsgr -r1 example.download.res.xml 8080
This runs the Test Messenger on port 8080. The Test Messenger discards request messages sent by clients and returns randomized responses using the SOAP/XML example.download.res.xml
response message template.
You can also use file redirection to test clients when your client uses the "http://" endpoint URL without a domain. By using this URL your client writes HTTP messages to standard output and reads HTTP responses from standard input. For example:
./testmsgr -r1 example.download.res.xml > response.xml ./client < response.xml > /dev/null
testmsgr [-aact] [-b] [-c] [-h] [-i] [-j] [-lmax] [-mmax] [-nlen] [-ofile] [-pnum] [-qnum] [-rseed] [-tsec] [-v] [-A] [-C] [-H] [-M] [infile|-] [URL|port]
where the testmsgr command-line options are:
Option | Description |
---|---|
-a act | use HTTP SOAP Action header value "act" |
-b | add random spacing to randomized values when using -r |
-c | continue testing until the service endpoint fails |
-h | display help message |
-i | indent XML with spaces when using -r |
-j | indent XML with tabs when using -r |
-l max | max number of randomized element repeats (default is 3) |
-m max | max length of randomized text values generated (default is 255) |
-n len | display the server response up to len bytes |
-o file | save to file |
-p perc | percentage XML kept when using -r (default=100) |
-q perc | percentage XML kept of optional indicators when using -r (default=100) |
-r seed | randomize XML message templates (use soapcpp2 -g ) |
-t sec | socket idle timeout seconds (default=1) |
-v | verbose mode |
-A | use SOAP with MIME attachments (SwA) |
-C | use HTTP chunked transfers |
-H | add HTTP headers when no URL is specified |
-M | use MTOM (application/xop+xml) |
-R rep | set XML element sequence indicator tag (default=__REPEAT ) |
-S sel | set XML element selection indicator tag (default=__SELECT ) |
infile | XML message template |
- | read XML from standard input |
URL | endpoint URL of service to test |
port | stand-alone server port for clients to test |
More specifically, infile
is an SOAP/XML message file (or message template file), -
specifies that the message should be read from standard input, and URL
is the optional service endpoint to send (randomized) messages to. When a port
number is specified instead of a URL, testmsgr runs as a stand-alone application to test clients that connect to it. A client application receives the (randomized) message generated by testmsgr from the specified SOAP/XML message file (or message template file).
-a act
Use HTTP SOAP Action header value "act"
with the SOAP request messages sent. This option is required for services that require a SOAP Action header.
-b
Add random blank spacing to randomized values when using option -r
.
-c
Continue testing of a service until the service fails, i.e. until the server at the specified endpoint URL becomes unresposive.
-h
Display help message.
-i
Indent XML with spaces when using option -r
.
-j
Indent XML with tabs when using option -r
.
-l max
Specifies the max number of randomized element repetitions for __SEQ
and __SEL
with min and max indicators, when using option -r
. The default is 3. The maximum value that can be specified is 18446744073709551615.
-m max
Specifies the max length of randomized text values generated when using option -r
. The default is 255. The maximum value that can be specified is 18446744073709551615.
-n len
Display the server response up to len
bytes. -n0
removes the server responses. Omitting this option displays the full response.
-o file
Save output to file. This option is not applicable when URL
or port
are specified.
-p perc
Retain XML with the specified percentage chance, when using option -r
. XML attributes and elements are removed by chance as specified. For example, -p10
keeps only roughly 10% of the XML elements and attributes of the original message. The default is 100 (retain all XML).
-q perc
Retain XML qualified with ???
and elements repeated with the __REPEAT
indicator with the specified percentage chance, when using option -r
. For example, if a message has an element or attribute value ???%[[INT32]]
then -q10
keeps the element or attribute with a 10% chance. Likewise, optional XML elements enclosed in <__REPEAT>
as indicated with min
and max
bounds are kept with a 10% chance. The default is 100 (retain).
-r seed
Randomize the SOAP/XML message template given as the infile
argument. SOAP/XML message templates can be auto-generated with soapcpp2 option -g
. The seed
value is required and is used to reproduce test messages. Any positive seed value can be specified, from 1 up to 2147483647.
-t sec
Sets a socket idle timeout in seconds. The default is one second. A service is considered unresponsive when a socket timeout occurs while messages are exchanged.
-v
Enable verbose mode.
-A
Use SOAP with MIME attachments, also called SOAP with Attachments (SwA). This option generates a MIME message with SOAP Body and zero or more MIME attachments. When combined with option -r
, a MIME attachment is added for each element with binary content (base64Binary XML element) specified in the template with %[[BASE64]]%
. The template %[[BASE64]]%
value is replaced with an href
attribute referencing the MIME attachment with randomized content.
-C
Use HTTP chunked transfers.
-H
Add HTTP headers to the generated message even when no URL is specified on the command line.
-M
Use MTOM (application/xop+xml) messaging. This option generates a MIME message with SOAP Body and zero or more MTOM attachments in MIME format. When combined with option -r
, a MIME attachment is added for each element with binary content (base64Binary XML element) specified in the template with %[[BASE64]]%
. The template %[[BASE64]]%
value is replaced with an <xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include"/>
element with an href
attribute referencing the MIME attachment with randomized content.
-R rep
Sets the XML element repetition indicator tag. The default is __REPEAT
. See Element repetition and selection indicators .
-S sel
Sets the XML element selection indicator tag. The default is __SELECT
. See Element repetition and selection indicators .
A SOAP/XML message template file is required when sending randomized messages with testmsgr option -r
. The SOAP/XML message template file can be auto-generated with soapcpp2 option -g
on a interface file. This interface file is usually generated with wsdl2h from a (set of) WSDL and XSD file(s). Note that option -g
does not produce SOAP/XML message templates if the interface file does not declare any service operations.
Templates may contain placeholder values. Placeholder values may start with ???
meaning the value is optional, followed by the placeholder value type, an enumeration, or a numeric range. Values are inserted with testmsgr option -r
.
Element and attributes that are qualified as optional with ???
are randomly discared with options -r
and -q
, where -q
specifies the percentage of elements and attributes that are retained in the generated XML message. A zero percentage specified with -q0
removes all optional values. A value of 100 specified with -q100
retains all optional values (the default) in the generated XML message.
The ???
qualifier may appear after the opening tag of an element (i.e. in front of the element value or child elements) and in attribute values. For example:
where id="???%[[INT32]]"
is an optional attribute with a 32-bit integer value, <name>???%[[TEXT]]%</name>
is an optional element with text, <contact>???
is an optional element with subelements, and <data lang="%[[LANG]]%">???</data>
is an optional empty element with an attribute.
%[[A][B][C]]%
specifies a choice of values A
, B
or C
. The values can be anything but should not contain ][
or ]]%
. For example:
where completed="%[[false][true]]%"
enumerates the values false
and true
. Note that this enumeration is identical to the %[[BOOL]]%
placeholder type.
A numeric range is integer-valued when the lower bound N
and upper bound M
are integer. Otherwise, the numeric range values are floating point. For example, %[[0:10]]%
speficies a range of integer values whereas %[[0.0:10.0]]%
specifies a range of floating point values. The bounds of a numeric range can be made exlusive by using (
and )
instead of [
and ]
.
%[[N:M]]%
specifies a range of integer or float values between N
and M
.
%[[N:M)]%
specifies a range of integer or float values between N
and M
(exclusive).
%[(N:M]]%
specifies a range of integer or float values between N
(exclusive) and M
.
%[(N:M)]%
specifies a range of integer or float values between N
(exclusive) and M
(exclusive).
For example:
where 0.00 is excluded from the float value range and 100000.00
is the inclusive maximum.
%[[BASE64]]%
and length-restricted %[[BASE64[N:M]]]%
specify xsd:base64Binary XSD values. The length restriction N:M
specifies the range of base64-encoded bytes.
%[[BOOL]]%
specifies xsd:boolean XSD values.
%[[DATE]]%
specifies xsd:date XSD values.
%[[DATETIME]]%
specifies xsd:dateTime XSD values.
%[[DOUBLE]]%
specifies xsd:double XSD values.
%[[DURATION]]%
specifies xsd:duration XSD values.
%[[HEX]]%
and length-restricted %[[HEX[N:M]]]%
specify xsd:hexBinary XSD values. The length restriction N:M
specifies the range of encoded bytes, not the number of hexadecimal characters.
%[[ENTITY]]%
and length-restricted %[[ENTITY[N:M]]]%
specify xsd:ENTITY XSD values. The range N:M
specifies the number of characters.
%[[ID]]%
and length-restricted %[[ID[N:M]]]%
specify xsd:ID XSD values. The range N:M
specifies the number of characters.
%[[IDREF]]%
and length-restricted %[[IDREF[N:M]]]%
specify xsd:IDREF XSD values. The range N:M
specifies the number of characters.
%[[FLOAT]]%
specifies xsd:float XSD values.
%[[INT8]]%
and %[[BYTE]]%
specify xsd:byte XSD values.
%[[INT16]]%
and %[[SHORT]]%
specify xsd:short XSD values.
%[[INT32]]%
and %[[INT]]%
specify xsd:int XSD values.
%[[INT64]]%
and %[[LONG]]%
specify xsd:long XSD values.
%[[LANG]]%
specifies xsd:language XSD values.
%[[NAME]]%
and length-restricted %[[NAME[N:M]]]%
specify xsd:Name XSD values. The range N:M
specifies the number of characters.
%[[NCNAME]]%
and length-restricted %[[NCNAME[N:M]]]%
specify xsd:NCName XSD values. The range N:M
specifies the number of characters.
%[[NMTOKEN]]%
and length-restricted %[[NMTOKEN[N:M]]]%
specify xsd:NMTOKEN XSD values. The range N:M
specifies the number of characters.
%[[TEXT]]%
and length-restricted %[[TEXT[N:M]]]%
specify xsd:string, xsd:normalizedString, xsd:anyURI, xsd:NOTATION, and xsd:token XSD values. The range N:M
specifies the number of characters.
%[[TIME]]%
specifies xsd:time XSD values.
%[[QNAME]]%
specifies xsd:QName XSD values.
%[[UINT8]]%
and %[[UBYTE]]%
specify xsd:unsignedByte XSD values.
%[[UINT16]]%
and %[[USHORT]]%
specify xsd:unsignedShort XSD values.
%[[UINT32]]%
and %[[UINT]]%
specify xsd:unsignedInt XSD values.
%[[UINT64]]%
and %[[ULONG]]%
specify xsd:unsignedLong XSD values.
Two special XML elements __REPEAT
and __SELECT
in message templates indicate XML element repetitions and selections, respectively. These elements are processed by the Test Messenger to generate child elements, after which the __REPEAT
and __SELECT
XML elements are removed from the generated randomized XML messages.
Child elements of the __REPEAT
indicator element are repeated within a range specified by min
and max
attributes of the __REPEAT
element. The min
and max
attributes are optional and default to 1 when omitted. Option -q
controls the percentage of child elements that are generated within the range. A zero percentage specified with -q0
generates the minimum number of child elements. A value of 100 specified with -q100
generates the maximum number of child elements. The maximum number of child elements (randomly) generated is bounded by option -l
(default is 3). Therefore, the maximum number of child elements is the greater of attribute min
and the lesser of attribute max
and the value lmax specified with option -l
:
min <= n <= maximum(min, minimum(max, lmax))
where n is the number of child elements generated with a random probability percentage specified by option -q
.
For example, to repeat an item
element with an integer value two or more times use the __REPEAT
indicator element with item
as a child element:
Assuming that testmsgr options -r1
, -l10
and -q50
are used, the generated XML message has at least two item
elements and at most 10 elements, with on average 50% of the elements kept between 2 (the min
minimum) and 10 elements (the maximum specified by -l10
), i.e. 6 elements on average. Each item
element has a random integer value.
One child element of the __SELECT
indicator element is randomly and uniformly selected as an XML element for the generated randomized XML message.
For example, to select one element from the three choices of elements A
, B
or C
:
The generated XML message has either an A
element with an integer value, a B
element with a floating point value, or a C
element with a text value.
Note that the ???
qualifier can be used with child elements to make them optional.
Copyright (c) 2017, Robert van Engelen, Genivia Inc. All rights reserved.