The Multi-Generator (MGEN) Version 4.1

The Multi-Generator (MGEN) is open source software by the Naval Research Laboratory (NRL) PROTocol Engineering Advanced Networking (PROTEAN) group which provides the ability to perform IP network performance tests and measurements using UDP/IP traffic (TCP is currently being developed). The toolset generates real-time traffic patterns so that the network can be loaded in a variety of ways. The generated traffic can also be received and logged for analyses. Script files are used to drive the generated loading patterns over the course of time. These script files can be used to emulate the traffic patterns of unicast and/or multicast UDP/IP applications. The receive portion of this tool set can be scripted to dynamically join and leave IP multicast groups. MGEN log data can be used to calculate performance statistics on throughput, packet loss rates, communication delay, and more. MGEN currently runs on various Unix-based (including MacOS X) and WIN32 platforms.

The principal tool is the mgen program which can generate, receive, and log test traffic. This document provides information on mgen usage, message payload, and script and log file formats. Additional tools are available to facilitate automated script file creation and log file analyses.

Quick Links

Mgen Usage

Mgen Script File Format

Mgen Log File Format

Mgen Message Payload

TRPR Log File Analysis Program

gpsLogger GPS Utility

Mgen Usage

The mgen version 4.x program must currently be launched from a command-line. In the future, a simple graphical user interface similar to that of mgen version 3.x will be provided to simply management of multiple sender and receiver instances. To launch mgen, use the following command-line syntax:

mgen [ipv4][ipv6][input <scriptFile>][save <saveFile>]
[output <logFile>][log <logFile>][binary][txlog]
[event "<mgen event>"][port <recvPortList>]
[sink <sinkFile>][source <sourceFile>]
[interface <interfaceName>][ttl <timeToLive>]
[tos <typeOfService>][label <value>]
[txbuffer <txSocketBufferSize>]
[rxbuffer <rxSocketBufferSize>]
[start <hr:min:sec>[GMT]][offset <sec>]
[txcheck][rxcheck][check]
[debug <debugLevel>]

Command-line Options

ipv4	Forces mgen to open sockets for IPv4 operation (i.e. AF_INET domain sockets) only. The default behavior for mgen is to open sockets with the domain based on environment (e.g. RES_OPTIONS) variables and the type of IP addresses used in the script file used.
ipv6	Forces mgen to open sockets for IPv6 operation (i.e. AF_INET6 domain sockets) only. The default behavior for mgen is to open sockets with the domain based on environment (e.g. RES_OPTIONS) variables and the type of IP addresses used in the script file used.
input <scriptFile>	Causes mgen, to parse the given <scriptFile> at startup and schedule any transmission or reception events given in the script.
save <saveFile>	Causes mgen to save the sequence number state of any pending transmit flows and the current relative script "offset" time to <saveFile> in the form of an MGEN script. The <saveFile> may be used as an additional input script on a subsequent launch of mgen to return mgen to the same state as when previously exited. See the equivalent global SAVE command for further detail on usage.
output <logFile>	Cause mgen to output logged information to the indicated <logFile>. By default, mgen will log to stdout. With the output command, an existing <logFile> of the same name will be overwritten. Use the log command to append to an existing log file.
log <logFile>	This is the same as the output command except that if <logFile> already exists, it will be appended instead of replaced.
binary	Causes mgen to save output logging information in a smaller-sized binary file format. This option should come before the output or log command.
txlog	This enables transmission logging. This results in SEND events being added to the log file every time a packet is sent by mgen.
event "<mgen event>"	The event command allows the user to enter the equivalent of MGEN script lines into mgen via the command-line. Multiple event commands can be used to pass the equivalent of a multi-line script to MGEN. Note that MGEN script events generally contain spaces and thus must be encapsulated in quotes on the command line. Note that the <eventTime> may be omitted and the action indicated by the event will be taken by mgen immediately.
port <recvPortList>	Causes mgen to monitor the given port numbers for received UDP traffic. The format of the <recvPortList> is a comma-delimited list of individual or inclusive ranges of port values (No spaces allowed in the list). Note this is the equivalent of a scripted "0.0 LISTEN UDP <recvPortList>" reception event and can also be equivalently achieved with the event command using the syntax: mgen event "LISTEN UDP <portList>" Example: mgen port 5000,5002,5005-5009
sink <sinkFile>	Causes mgen to use the file or device (e.g. stdout) indicated as a "sink" or destination for transmitted message flows of protocol type "SINK". I.e., MGEN message flows of type "SINK" are written to the "sink" device instead of to a UDP (or eventually TCP) socket. Piping mgen output to stdout with allows MGEN messages use alternative transport provided by another process (e.g. ssh, norm, etc). The special <sinkFile> value "STDOUT" will direct MGEN SINK flows to the mgen process stdout.
source <sourceFile>	This is the complement to the sink command. This allows mgen to directly receive a binary stream of MGEN messaging from the <sourceFile> which may be the piped stdout from another process (e.g. ssh, norm, etc). The special <sourceFile> string "STDIN" causes mgen to get input from its stdin stream. Messages read from the <sourceFile> (or stream) are time-stamped and logged in the MGEN log file as usual.
start <hr:min:sec>[GMT]	Causes mgen to delay processing events in script file relative to the indicated absolute time. The optional "GMT" keyword indicates the time is Greenwich Mean Time instead of the default local time. This command establishes an absolute time for the relative script time of 0.0 seconds.
offset <sec>	Causes mgen to skip <sec> seconds of relative time into the execution of the script file used. Note that if an absolute start time is given using the start command, the offset into the script will correspond to that absolute time. The default offset for MGEN is 0.0 seconds.
convert <binaryLogFile>	Causes mgen to convert the indicated <binaryLogFile> to a text-based log file. The text-based log file information will be directed to stdout unless you specify a filename with the output or log command. Mgen will exit after the file conversion is complete.
interface <interfaceName>	Causes mgen to set the default network interface for IP multicast and/or root node flow transmission to <interfaceName>. <interfaceName> will override any default interface specified within an mgen script file. <interfaceName> is a "per socket" attribute, and in its absence, MGEN will behave according to the operating system's default behavior.
ttl <timeToLive>	Causes mgen to set the hop count for IP multicast traffic generated by MGEN. <timeToLive> will override any default ttl indicated within an mgen script file. <timeToLive> is a "per socket" attribute. If no ttl option is used, MGEN will behave according to the operating system's default behavior.
tos <typeOfService>	Causes mgen to set the IPv4 type-of-service field (within the packet header) to <typeOfService>. <typeOfService> will override any default tos indicated within an mgen script file. As with ttl and interface, tos is a "per socket" attribute. If no tos option is used, MGEN will behave according to the operating system's default behavior.
label <value>	Causes mgen to set <value> as the default flow label for IPv6 flows. The <value> corresponds to the 28-bit IPv6 flow label field and may be specified in decimal or hex.
txbuffer <bufferSize>	Causes mgen to set the socket transmit buffer size to a value “at least” as large as <bufferSize>. If <bufferSize> is larger that the maximum allowed by the system, <bufferSize> will be set to the system maximum.
rxbuffer <bufferSize>	Causes mgen to set the socket receive buffer size to a value “at least” as large as <bufferSize>. If <bufferSize> is larger that the maximum allowed by the system, <bufferSize> will be set to the system maximum.
txcheck	Causes mgen to include an optional 32-bit cyclic redundancy checksum (CRC) at the end of its messages. The CHECKSUM flag is set to indicate the presence of the checksum content.
rxcheck	Forces mgen receivers to validate the checksum portion (last 4 bytes) of MGEN messages whether or not the CHECKSUM flag is set in the MGEN flags message field. Use this option when it is _known_ that the MGEN sender is supplying checksums to cover the case when the flags field itself is possibly corrupted.
check	Sets mgen behavior as if both the txcheck _and_ rxcheck commands were applied. This is the recommended option when MGEN checksum operation is desired so that both senders and receivers are providing and validating checksums, respectively.

Example Usage

To run mgen with script file "script.mgn" and log to stdout (by default):

mgen input script.mgn

To monitor ports 5000,5004,5005, and 5006 for received UDP traffic and log to a specific file "log.drc":

mgen port 5000,5004-5006 output log.drc

The "event" command can be used to achieve equivalent operation with the command-line syntax:

mgen event "listen udp 5000,5004-5006" output log.drc

The "event" command allows for use of mgen without script files for "quick and dirty" runs. In the future, MGEN will be capable of being dynamically scripted during run time with "event" commands passed to MGEN via inter-process communication.

Note: In previous versions, two different programs (each with different scripts) were used to separately generate and receive test traffic. The traffic generation capability of the former mgen program and the receive-side functionality of the Dynamic-Receiver (drec) program have now been integrated into a single executable mgen program.

The file extensions ".mgn" for MGEN scripts and ".drc" for MGEN log files are suggested conventions that users might wish to use for consistency. The ".drc" naming is in honor of the deprecated drec program.

The "sink" and "source" commands can be used to stream MGEN messages via alternative transport processes (e.g. reliable multicast, ssh, peer-to-peer protocols, etc). Here is an example using ssh to set up a TCP connection to a remote machine, start an mgen receiver at the remote machine to log receive messages, and attempt to transmit a 2 Mbps stream of MGEN messages via the ssh connection:

mgen event "ON 1 SINK DST 127.0.0.1/5001 PERIODIC [200 1250]" sink STDOUT output /dev/null |
ssh <remoteHost> sh –c "cat | mgen source STDIN output mgenLog.drc"

Note that the mgen executable must be present on the remote machine. Also note the importance of directing the MGEN sender's log output to /dev/null so that it doesn't get piped to the ssh process, mixed with the binary "sink" message stream.

MGEN Script Format

MGEN scripts are text files containing a sequence of commands and scheduled events describing traffic generation patterns, ports and/or multicast groups to be monitored, and other options. Each line in the script corresponds to either a "Transmission Event", or "Reception Event", or "Global Command". Lengthy script lines can be continued to multiple text file lines by using a trailing backslash '\' character at the end of the line. Additionally, blank lines are permitted and comment lines can be included by providing a leading '#' character at the beginning of lines. (Note comment lines cannot be inserted in between "continued" script lines). Currently mgen is case-sensitive in parsing the script file format (commands, options, etc are all upper case), but will be modified to be case-insensitive in the future.

Scheduled transmission and reception events in the script use lines in the format of:

[<eventTime>] <eventType> <parameters ...> [<options ...>]

These "events" are scheduled to be executed by MGEN at the relative time given by the <eventTime> field. The value of this field is a floating point number which denotes the relative time (in seconds) of the associated event. The time is relative to the start of the MGEN program or the time dictated by the global START command. If the <eventTime> is omitted, an <eventTime> of 0.0 (or immediately if MGEN is already started) is assumed (This can used with MGEN's "event" command to directly control the operation of ns-2 Agent/MGEN instances within an ns-2 TCL (Tool Command Language) script without use of an external MGEN script.

Global commands are generally used to define default behaviors for MGEN operation or other options independent of event scheduling. The format for global command script lines is:

<commandType> [<command parameters ...>]

Transmission Events

MGEN "Transmission Event" script lines are used to schedule and characterize mgen traffic generation. An instance of mgen can simultaneously transmit traffic to multiple destinations with different patterns of transmission. The MGEN script format uses "flow identifiers" (<flowIds>) to tag specific "threads" of MGEN traffic generation. While the <flowIds> are placed in the payload of associated MGEN messages, the primary purpose of the <flowId> is to simply tie together a sequence of script "transmission events" as a single "flow" or "thread".

The sequence of events pertaining to a "flow" of MGEN traffic generation consist of ON, MOD, and OFF types. The script line syntax for these event types is:

<eventTime> {ON|MOD|OFF} <flowId> [<options ...>]

The first scripted event for a given flow identified by a <flowId> must be an ON event. Subsequently, MOD events can be used to modify characteristics of the given flow until it is terminated with an OFF event. After a flow has been terminated with the OFF command, a flow with the same <flowId> value may be initiated with another ON event. The <options> fields are used to describe the characteristics of flows initiated with ON events and modified with subsequence MOD events. The OFF event uses no options.

ON Event

Script syntax:

<eventTime> ON <flowId> <protocol> DST <addr>/<port> \
<pattern [params]> [<options ...>]

This transmission event type is used to initiate a new flow at the time given by the <eventTime>. The <flowId> is used to identify the flow within the script and can be used by subsequent MOD or OFF events to reference the flow initiated here.

The <protocol> field indicates the transport protocol to be used for the generated MGEN test messages. Current supported <protocol> types include "UDP" and "SINK". The "UDP" protocol type encapsulates generated MGEN messages for the flow into UDP/IP packets and transmits them over the network. Messages for the "SINK" protocol type are written to the file/device/stream indicated by the mgen "sink" command-line option. In the future, TCP and possibly other protocol types will be available for MGEN traffic flows. The flow destination address and port must be specified for the ON event using the DST option and the <pattern> of message generation must be given as well. Other flow <options> may be specified to further characterize the flow.

Example:

This script line will originate a "flow" of MGEN UDP destined for the loopback address (IP address 127.0.0.1) port number 5000 beginning immediately when the script is executed. The messages will consist of 1024 byte messages at a regular rate of 1.0 per second:

0.0 ON 1 UDP DST 127.0.0.1/5000 PERIODIC [1.0 1024]

MOD Event

Script syntax:

<eventTime> MOD <flowId> [<options ...>]

This transmission event type is used to modify the characteristics of an existing flow identified by the <flowId> field. The given transmission event <options> determine which specific characteristics of the flow (e.g. PATTERN, TOS, destination (DST), etc) will be affected. Multiple options may be specified in the script line. Note that the protocol type and source port number (SRC) cannot be changed with the MOD event type (the referenced flow should be terminated with an OFF event and re-initiated with an ON event to accomplish this goal). If no <options>are given, the flow will remain unaltered. A script parse error will result if the identified flow was not previously initiated with an ON event.

Example:

This script line will modify "flow 1" to change it packet transmission pattern 5.0 seconds after script execution. The changed "flow 1" will then generate messages 512 bytes in size at an average rate of 10.0 messages per second following a Poisson (exponentially-distributed interval)

5.0 MOD 1 POISSON [10.0 512]

OFF Event

Script syntax:

This transmission event type terminates message transmission for the flow identified by the <flowId> field at the time given in the <eventTime> field. There are no options applicable to this event type. A script parse error will result if the identified flow was not previously initiated with an ON event.

Example:

This script line will terminate generation of MGEN message traffic for "flow 1" at 10.0 seconds after script execution.

10.0 OFF 1

Transmission Event Options

This section describes options which may be applied to ON or MOD transmission events in MGEN script files. Note that ON event lines require specification of at least the <protocol>, <destination>, and <pattern> options, while only the options to be changed need to be specified as part of MOD event lines.

Protocol (UDP)

Option syntax:

... <protocolType> ...

The transport protocol for MGEN messages generated by a flow must be specified as part of any ON events. Currently, "UDP" is the only protocol type supported. In the future, additional transport options my be provided in MGEN.

Example:

0.0 ON 1 UDP DST 127.0.0.1/5000 PERIODIC [1.0 1024]

Destination (DST)

Option syntax:

... DST <addr>/<port> ...

The destination address for a flow must be specified for ON events and may be altered as part of MOD events. The <addr> field specifies the destination IP address (IPv4 or IPv6) and the <port> field specifies the destination host port number. The destination address may be a unicast (point-to-point) or multicast address.

Examples:

#Start a flow to loopback address port 5000
0.0 ON 1 UDP DST 127.0.0.1/5000 PERIODIC [1.0 1024]

#Modify flow 1 to a different destination port
0.0 MOD 1 DST 127.0.0.1/5001

Source Port (SRC)

Option syntax:

... SRC <port> ...

The source port number used for generated traffic may be optionally specified as part of an ON event. The <port> field identifies the host port number to be used. When the SRC option is not specified, the flow will use a free port number picked by the operating system. Note that MGEN UDP flows may share the same source port and the associated flow will "inherit" some attributes (e.g. TOS, TTL, etc) which may have been set for other flows which use that same source port. This is because some of these attributes tend to be maintained by operating systems on a "per socket" basis. Also, any such attributes set for this flow will affect other existing flows using the same source port. Thus, the SRC option is useful when it is desired to explicitly create different flows with distinct "per socket" attributes such as TOS or multicast TTL.

Example:

Here, two flows are created with the same destination address, but different source ports. Flow 1 is also assigned non-default type-of-service using the TOS option. The use of the SRC option ensures that two different sockets are used to support the two different types of service.

#Start flow 1 using source port 5001(TOS = 0x10) and flow 2 using port 5002
0.0 ON 1 UDP DST 127.0.0.1/5000 SRC 5001 PERIODIC [1.0 1024] TOS 0x10
0.0 ON 2 UDP DST 127.0.0.1/5000 SRC 5002 PERIODIC [10.0 512]

Pattern (PERIODIC, POISSON, BURST)

Option syntax:

... <patternType> [parameters ...] ...

(Note: The '[' and ']' characters are explicitly required at the beginning and end of the pattern parameter set. Different pattern types may use different parameter sets.)

Traffic generated by MGEN consists of a series of sequence-numbered messages. The messaging generated by MGEN may vary in size and frequency of transmission to stress the network in a controlled fashion or possibly emulate other network applications. The "Pattern" of message generation must be specified in ON events and may be altered as part of subsequent MOD events. Currently MGEN supports three pattern types, "PERIODIC", POISSON", and "BURST". Complex traffic patterns can be created by using a compound of multiple "flows" (with the same SRC/DST) with different pattern types and parameters. Other pattern types (e.g. MARKOV), including ones with statistically varying payload sizes, will be added eventually.

PERIODIC Pattern:

Option syntax:

... PERIODIC [<rate> <size>] ...

This pattern type generates messages of a fixed <size> (in bytes) at a very regular <rate> (in messages/second). The <size> field must be greater or equal to the minimum MGEN message size and less than or equal to the maximum UDP message size of 8192 bytes. When other transport support is added (e.g. TCP), the message <rate> and <size> parameters will have additional flexibility as appropriate. Note the <rate> must be greater than or equal to 0.0 messages/second.

Example:

#Start an MGEN flow sending 1024 byte messages
#at a rate of 10.0 per second
0.0 ON 1 UDP DST 127.0.0.1/5000 PERIODIC [10.0 1024]

#Reduce the flow rate to one 512 byte message
#every 2.0 seconds
0.0 MOD 1 PERIODIC [0.5 512]

POISSON Pattern:

Option syntax:

... POISSON [<aveRate (msg/sec)> <size (bytes)>] ...

This pattern type generates messages of a fixed <size> (in bytes) at statistically varying intervals at an average <rate> (in messages/second). The <size> field must be greater or equal to the minimum MGEN message size and less than or equal to the maximum UDP message size of 8192 bytes. When other transport support is added (e.g. TCP), the message <aveRate> and <size> parameters will have additional flexibility as appropriate. Note the <rate> must be greater than or equal to 0.0 messages/second.

Example:

#Start an MGEN flow sending 1024 byte messages
#at an average rate of 10.0 per second
0.0 ON 1 UDP DST 127.0.0.1/5000 POISSON [10.0 1024]

#Reduce the flow rate to an average of one
#512 byte message every 2.0 seconds
0.0 MOD 1 POISSON [0.5 512]

BURST Pattern:

Option syntax:

... BURST [REGULAR|RANDOM <aveInterval (sec)>
<patternType> [<patternParams>]
FIXED|EXPONENTIAL <aveDuration (sec)>] ...

The BURST pattern generates bursts of other MGEN pattern types at a specified average interval. The first parameter of the BURST pattern is either "REGULAR" resulting in periodic burst uniformly distributed in time by the <aveInterval> value, or "RANDOM" which exponentially distributes the traffic generation bursts in time with an average burst interval as specified by the <aveInterval> parameter value. The characteristics of the MGEN messages generated during a burst is given by the <patternType> and associated <patternParams> parameters. The <patternType> may any MGEN pattern type including PERIODIC, POISSON, or, yes, even BURST. The <patternParams> must be appropriate for the given <patternType>. When a traffic generation burst occurs, its duration is either of a FIXED value as given by the <aveDuration> or a randomly varying duration with EXPONENTIAL statistics and an average duration as given by the <aveDuration> parameter.

An example use of the BURST pattern would be to roughly emulate the "talk spurts" which might result from Voice Over IP (VOIP) applications. As a voice conversation commences, a user's burst of activity (talk spurts) might be RANDOM with some average interval and the duration of the talk spurts might approximate EXPONENTIAL statistics. When the talk spurt (burst) occurs, the voice compression codec might generate messages following something like a PERIODIC flow with packet rates and packet sizes dependent upon the voice codec in use.

Other uses of the BURST pattern might be to roughly model message/packet generation occurring with random use of a network such as web browsing, etc. The BURST model provided by MGEN does not presuppose any specific traffic model, but might be useful in approximating some models of regular or intermittent network activity.

The average traffic generation rate for this pattern should be approximately the average transmission rate of the core <patternType> and <patternParams> multiplied by the burst duty cycle (<aveDuration> / <aveInterval>). Note that when average burst duration tends to exceed the average burst interval, the flow will tend to follow the characteristics of the core pattern (i.e. 100% duty cycle).

Example:

#Start a bursty MGEN flow with bursts of 1024 byte messages
#with a periodic rate of 10.0 messages per second. The #bursts will occur at random intervals with an average
#interval from the start of one burst until the start of #the next of 10.0 seconds. The duration of each burst is
#of exponential statistics with an average burst duration
#of 5.0 seconds.

0.0 ON 1 UDP DST 127.0.0.1/5000 BURST [RANDOM 10.0 PERIODIC [10.0 1024] EXP 5.0]

Type-Of-Service (TOS)

Option syntax:

... TOS <value> ...

The IP TOS (type-of-service) field can be controlled for IP packets associated with MGEN traffic generation. The <value> field specifies the value of the 8-bit TOS field in IPv4 packets. (IPv6 packets do not have a TOS field. MGEN will soon support control of the similar FLOW_ID field for IPv6 operation.) The <value> field must be in the range of 0-255 in decimal or hexadecimal notation. The interpretation of the TOS value by different computer operating systems and network devices may vary. In some cases, computer hosts will not allow all possible values to be used, and in others "super user" (root) privileges may be required to set the IP TOS field to certain values. Below are some notes on suggested interpretation by the Internet Engineering Task Force (IETF). Note that TOS is maintained on a "per socket" basis and that setting the TOS for a flow will affect other flows sharing the same network socket. See the SRC option to make sure different flows use different sockets.

Example:

#Start flow 1 with default TOS
0.0 ON 1 UDP DST 127.0.0.1/5000 PERIODIC [1.0 1024]
#Modify flow 1 to TOS = 0x10 (low delay)
5.0 MOD 1 TOS 0x10

Notes on the value of the IP TOS field:

0 1 2 3 4 5 6 7
+-----+-----+-----+-----+-----+-----+-----+-----+
| PRECEDENCE | TOS | MBZ |
+-----+-----+-----+-----+-----+-----+-----+-----+

The Type-of-Service byte in the IP header is divided into three sections: the Precedence field (high-order 3 bits), a field that is called Type of Service or TOS (next 4 bits), and a reserved bit (the low order bit). The TOS bits can be set to 5 different settings including the default setting of 0000, while the PRECEDENCE can be set to 8 different setting including default 000.

TOS definitions:

	IPTOS_TOS_MASK IPTOS_TOS(tos)	0x1E = ((tos) & IPTOS_TOS_MASK)
1000 --	IP_TOS_LOWDELAY	0x10	TOS = 16
0100 --	IP_TOS_THROUGHPUT	0x08	TOS = 8
0010 --	IPTOS_RELIABILITY	0x04	TOS = 4
0001 --	IPTOS_LOWCOST	0x02	TOS = 2
0000 --	normal service	0x00	TOS = 0

Precedence definitions:

111 --	IPTOS_PREC_MASK IPTOS_PREC(tos)	0xe0 = ((tos) & IPTOS_PREC_MASK)
111 --	IPTOS_PREC_NETCONTROL	0xe0	TOS = 224
110 --	IPTOS_PREC_INTERNETCONTROL	0xc0	TOS = 192
101 --	IPTOS_PREC_CRITIC_ECP	0xa0	TOS = 160
100 --	IPTOS_PREC_FLASHOVERRIDE	0x80	TOS = 128
011 --	IPTOS_PREC_FLASH	0x60	TOS = 96
010 --	IPTOS_PREC_IMMEDIATE	0x40	TOS = 64
001 --	IPTOS_PREC_PRIORITY	0x20	TOS = 32
000 --	IPTOS_PREC_ROUTINE	0x00	TOS = 0

If TOS = 164 (or 0xa4), the Precedence would be IPTOS_PREC_CRITIC_ECP and the TOS would be IPTOS_RELIABILITY. The IP TOS field bits would be set as 10100100.

Multicast Time-To-Live (TTL)

Option syntax:

... TTL <value> ...

The time-to-live (TTL) hop count can be controlled for IP multicast traffic generated by MGEN. As with TOS, this is generally a "per socket" attribute and care should be taken if it is desired to specify different TTL values for different MGEN flows. This can be accomplished by using different SRC (source ports) for different MGEN flows. The <value> field must be in the range of 1-255. The default multicast TTL assumed by MGEN is 3.

Example:

#Start an IP multicast flow with a maximum hop count ttl = 2
0.0 ON 1 UDP DST 224.1.2.3/5000 PERIODIC [1.0 256] TTL 2

Socket Transmit Buffer Size (TXBUFFER)

Option syntax:

... TXBUFFER <txBufferSize> ...

This option allows users to set the socket transmit buffer size to a value “at least” as large as <txBufferSize>. If <txBufferSize> is larger that the maximum allowed by the system, <txBufferSize> will be set to the system maximum. To date, this option has only been tested on linux systems.

Socket Receive Buffer Size (RXBUFFER)

Option syntax:

... RXBUFFER <rxBufferSize> ...

This option allows users to set the socket receive buffer size to a value “at least” as large as <rxBufferSize>. If <rxBufferSize> is larger that the maximum allowed by the system, <rxBufferSize> will be set to the system maximum. To date, this option has only been tested on linux systems.

IPv6 Flow Label (LABEL)

Option syntax:

... label <value> ...

This option allows users to specify the value applied to the IPv6 packet header "flow label" field. Although this field is 28 bits, different operating systems may restrict which portions of the field may be set. For example, the current Linux kernel (circa Jan 2003) only allows bits in the first octet of the flow label to be set. Other values are invalid on Linux, and generate an error message. Thus, using hexadecimal format for the <value> specified, legal values for Linux are restricted to <value> = 0x0??00000 where "??" specifies the first octet of the flow label. Other operating systems may behave differently.

Example:

# Start an IPv6 flow with flow label = 0x03d00000
0.0 ON 1 UDP LABEL 0x03d00000 SRC 5000 DST 5f1b:df00:ce3e:e200:0800:2078:e3e3/5001 PERIODIC [1 1024]

Multicast Interface (INTERFACE)

Option syntax:

... INTERFACE <interfaceName> ...

The network interface to use for IP multicast flow transmission can be controlled with this option. The <interfaceName> is the network interface device name to be used for IP multicast transmission for the associated flow. Again, as with TOS and TTL, this is generally a "per socket" attribute and care should be taken if it is desired to specify different multicast interfaces for different MGEN flows. This can be accomplished by using different SRC (source ports) for different MGEN flows. If no INTERFACE option is used, MGEN will behave according to the operating system's default behavior.

Example:

#Start an IP multicast flow on Ethernet interface named "eth1"
0.0 ON 1 UDP DST 224.1.2.3/5000 PERIODIC [1.0 256] INTERFACE eth1 SRC 5001

Sequence Number Initialization (SEQUENCE)

Option syntax:

... SEQUENCE <sequenceNumber> ...

This option sets the sequence number of the next message transmitted for the flow. MGEN flows are normally initialized to a sequence number of zero upon the first "ON" event for the flow. The sequence number is incremented by one with each message transmitted. The SEQUENCE option allows the user to override this behavior. It (along with the OFFSET command) is used by the SAVE command with MOD events for pending flows when it is desired that mgen return to a particular point in a script after being stopped and restarted.

Example:

#Modify the sequence number of an existing flow such that
#the next message is transmitted with sequence number 452.
12.0 MOD 1 SEQUENCE 452

Reception Events

For simple reception and logging of unicast traffic, it is generally sufficient just to launch mgen with the port command line option specifying the port numbers to monitor. However, for IP multicast operation or more complex behavior, an MGEN script with "Reception Events" is required. "Reception Events" in the MGEN script file format include LISTEN and IGNORE types to control which ports are being monitored when; and JOIN and LEAVE types to dynamically control IP group membership. The MGEN script syntax of "Reception Events" is:

<eventTime> <eventType> <parameters ...> [<options ...>]

LISTEN

Script syntax:

<eventTime> LISTEN <protocol> <portList>

The LISTEN event is used to prompt mgen to begin monitoring one or more ports for received traffic. The <eventTime> denotes the time (in seconds) relative to script execution. The <protocol> field specifies the transport protocol type. Currently, only "UDP" transport is supported. The <portList> field is a comma-delimited list of individual or inclusive ranges of the port numbers (no spaces allowed) to begin monitoring. Port ranges within the list are specified in the format "<lowValue>-<hiValue>".

Example:

#Monitor port numbers 5000,5003,5004,5005,5009
#beginning at time 0.0
0.0 LISTEN UDP 5000,5003-5005,5009

IGNORE

Script syntax:

<eventTime> IGNORE <protocol> <portList>

The IGNORE event type is the converse to the LISTEN event type. An IGNORE event causes mgen to stop monitoring (and logging) received traffic on the specified <portList>. The <eventTime> denotes the time (in seconds) relative to script execution. The <protocol> field specifies the transport protocol type. Currently, only "UDP" transport is supported. The <portList> field is a comma-delimited list of individual or inclusive ranges of the port numbers (no spaces allowed) to begin monitoring. Port ranges within the list are specified in the format "<lowValue>-<hiValue>".

Example:

#Stop monitoring port numbers 5003,5004,5005,5009
#beginning at time 10.0
10.0 IGNORE UDP 5003-5005,5009

JOIN

Script syntax:

<eventTime> JOIN <groupAddress> [INTERFACE <interfaceName>] [PORT <portNumber>]

The JOIN event is used to prompt mgen to "join" the specific IP multicast group indicated by the <groupAddress> field. The INTERFACE option forces the membership join request on the network interface identified by the <interfaceName> field. If no INTERFACE option is given, the operating system's default behavior is observed. Note it is possible to join the same group on multiple, different interfaces.

The PORT option is provided principally for operation on WIN32 systems where the IP multicast join must be performed on the same socket bound to a specific <portNumber>. Note that a corresponding LISTEN event for the indicated <portNumber> is required in order to receive traffic. Unix-based operating systems generally allow for IP multicast group membership to be independent of specific socket port bindings.

As many IP group memberships as the operating system will support is permitted by mgen. This is generally a limit of the maximum number of open sockets per process or in the system at large if multiple mgen instances are used. Note that WIN32 imposes a limitation of one IP multicast group membership per socket while Unix-based systems can allow for many memberships (often 20, but OS-specific) per socket.

Examples:

#JOIN group 224.1.2.3 at time 0.0
0.0 JOIN 224.1.2.3

#JOIN group 224.1.2.4 on interface "eth1"
0.0 JOIN 224.1.2.4 INTERFACE eth1

#JOIN group 224.1.2.5 using socket bound to port 5000
0.0 JOIN 224.1.2.5 PORT 5000

LEAVE

Script syntax:

<eventTime> LEAVE <groupAddress> [INTERFACE <interfaceName>] [PORT <portNumber]

The LEAVE event is used to prompt mgen to "leave" the specific IP multicast group indicated by the <groupAddress> field. The <groupAddress> must have been joined with a prior JOIN event. The INTERFACE and/or PORT options must be used if they were used with the corresponding JOIN event.

Examples:

#LEAVE group 224.1.2.3 at time 10.0
10.0 LEAVE 224.1.2.3

#LEAVE group 224.1.2.4 on interface "eth1" at time 10.0
10.0 LEAVE 224.1.2.4 INTERFACE eth1

#LEAVE group 224.1.2.4 on interface "eth1"and
#port 5000 at time 10.0
10.0 LEAVE 224.1.2.4 INTERFACE eth1 PORT 5000

Global Commands

The MGEN script file format supports a subset of commands which are independent of normal transmission and reception event scheduling. These are referred to as "Global Commands". This subset includes commands to specify an absolute script execution start time (START command) and to specify default traffic generation characteristics (e.g. multicast INTERFACE, multicast TTL, IP TOS, etc).

The format of script lines containing global

commands is:

In general, a script file should contain only one occurrence of each global command type. If there are multiple occurrences of a command type, the last occurrence will determine mgen's behavior. These commands can also be given on the mgen command-line. Global commands given on the mgen command-line will override any corresponding global commands in the script file(s).

The MGEN "Global Command" set includes:

START	Specifies an absolute start time for script processing.
OFFSET	Specifies an offset time into script for MGEN activity.
TOS	Specifies a default IPv4 TOS value for IPv4 flows.
LABEL	Specifies a default IPv6 Flow Label for IPv6 flows.
TTL	Specifies a default TTL (time-to-live) hop count for transmitted multicast packets.
INTERFACE	Specifies the name of the default interface to use for IP multicast.
INPUT	Specifies the name of a script file to be loaded and parsed.
OUTPUT	Specifies the name of the log file to record logged events. If the named log file pre-exists, it is overwritten.
LOG	Same as OUTPUT, except that pr-existing log files are appended instead of overwritten.
SAVE	Specifies a file to which MGEN should store sequence number state for any pending or active flows as well as the current relative script offset time when mgen is terminated.
TXBUFFER	Specifies a default socket transmit buffer size.
RXBUFFER	Specifies a default socket receive buffer size.

START

Script syntax:

START <hour:min:sec>[GMT]

The START command designates an absolute time as indicated by the <hour:min:sec> field to correspond to the relative script time of 0.0 seconds. All transmission and reception events will be scheduled relative to this absolute start time. The optional GMT suffix (no white space after the time) indicates that the clock time given is Greenwich Mean Time (GMT) rather than the operating systems local time zone. If no START command is given, mgen schedules transmission and reception events relative to program startup.

Example:

#Start MGEN exactly at 1:30PM local time
START 13:30:00

#Start MGEN at 30 seconds past 8:30 GMT
START 8:30:30GMT

OFFSET

Script syntax:

OFFSET <seconds>

The OFFSET global command specifies a relative time offset (in seconds)into script processing where MGEN should begin its activity. This allows multiple instances of MGEN using the same script to be dithered as desired. Additionally, this command may be used to immediately restore MGEN to a specific scripted state other than the beginning of the script upon launch.

Example:

#Skip the first 10 seconds of an MGEN script
OFFSET 10.0

TOS

Script syntax:

TOS <value>

The TOS command specifies ...

Example:

#Specify default ttl = 0x10 (low delay)
TOS 0x10

LABEL

Script syntax:

LABEL <value>

The LABEL command specifies a default value to be used as the "flow label" for IPv6 flows. The "flow label" is the corresponding 28-bit field in the IPv6 packet header. Refer to the Transmission Event LABEL option for further details..

Example:

#Specify default IPv6 flow label = 0x02500000
LABEL 0x02500000

TTL

Script syntax:

TTL <value>

The TTL command specifies the default time-to-live (TTL) hop count for generated IP multicast traffic according to the <value> field. The <value> must be in the range of 1-255. If the global TTL command is not used, mgen assumes a default multicast TTL value of 3. Note that the transmission event TTL option will override the default specification given by this global command.

Example:

#Specify default multicast flow ttl = 32
TTL 32

TXBUFFER

Script syntax:

TXBUFFER <txBufferSize> ...

This option allows users to set the default socket transmit buffer size to a value “at least” as large as <txBufferSize>. The exact behavior of this option may be operating system dependent. The TXBUFFER option given on transmission event script lines will override this default for the socket used by the corresponding flow.

RXBUFFER

Script syntax:

RXBUFFER <rxBufferSize> ...

This option allows users to set the default socket receive buffer size to a value “at least” as large as <rxBufferSize>. The exact behavior of this option may be operating system dependent. The RXBUFFER option given on transmission event script lines will override this default for the socket used by the corresponding flow.

INTERFACE

Script syntax:

INTERFACE <interfaceName>

The INTERFACE command specifies a default IP network interface to use for multicast traffic generation and group membership. If no INTERFACE command is given, the default operating system behavior is observed. Note that the transmission event INTERFACE option or the JOIN reception event INTERFACE option will override the default specification given by this global command.

Example:

#Specify "eth1" as the default network interface
#for multicast transmission and group joins
INTERFACE eth1

INPUT

Script syntax:

INPUT <scriptFile>

The INPUT command cause MGEN to load and parse the given <scriptFile>. (Circular references are not detected by mgen and should be avoided_). This allows scripts to "include" other scripts. The parsing occurs in the order that the INPUT commands are encountered on the command-line and within the script files themselves.

Example:

#Load and parse the MGEN script file "script2.mgn"
INPUT script2.mgn

OUTPUT

Script syntax:

OUTPUT <logFile>

The OUTPUT command cause mgen to direct its log output to the indicated <logFile>. The last occurring OUTPUT command determines the log file to be used and the command-line takes precedence over any scripts provided as input to mgen. The file named by <logFile> will be overwritten if it already exists.

Example:

#Use the file "logFile.drc" for logging
OUTPUT logFile.drc

LOG

Script syntax:

LOG <logFile>

The LOG command cause mgen to direct its log output to the indicated <logFile>. The last occurring LOG command determines the log file to be used and the command-line takes precedence over any scripts provided as input to mgen. The file named by <logFile> will be appended if it already exists.

Example:

#Append the file "logFile.drc"
LOG logFile.drc

SAVE

Script syntax:

SAVE <saveFile>

The SAVE command causes mgen to write a short MGEN script upon exit which includes current sequence number state for pending and active transmission flows as well as the current relative script offset time. If the <saveFile> created is given as an additional input script (with the same input script(s) given for the mgen instance which created <saveFile>), on a subsequent launch of mgen, mgen will return to the same state as it was when it previously exited.

The SAVE command can be used when it is desired to conduct separate runs of mgen, but preserve a continuous sequence number space across the multiple runs. An example script performing this function is given below. Note the same behavior can also be achieved via the mgen command-line using:

"mgen input <scriptFile> input <saveFile> save <saveFile> log <logFile>"

if an empty <saveFile> is provided on the first launch of mgen.

Example:

#This script executes another MGEN script, using
# "saveFile.mgn" for state recovery upon subsequent
# restarts after mgen is exited.
# (If "saveFile.mgn" is empty or non-existent upon
# initial startup, the "scriptFile.mgn" is run
# from the beginning.)
# (The "log" command is used to repeatedly append
# the "logFile.drc" file upon stop and restart)
INPUT scriptFile.mgn
INPUT saveFile.mgn
SAVE saveFile.mgn
LOG logFile.drc

Example Script

Below is an example MGEN script which generates two "flows" of UDP traffic. In this example, flow 1 is sent to the loopback interface address (127.0.0.1) port 5001 and flow 2 is sent to an IP multicast group on port number 5002. A LISTEN command is used to monitor these (and other) ports so that mgen can receive its own traffic for demonstration purposes. This script illustrates the usage of a number of MGEN script commands.

# MGEN script begins here

# These are some "Transmission Event" script lines

# Originate two UDP flows

0.0 ON 1 UDP SRC 5001 DST 127.0.0.1/5001 PERIODIC [1 1024]

0.0 ON 2 UDP SRC 5002 DST 224.225.1.2/5002 PERIODIC [1 512]

# Modify the pattern/rate of flow 2

4.0 MOD 2 POISSON [10 1024]

# These are some "Reception Event" script lines

# Monitor some ports for UDP traffic

0.0 LISTEN UDP 5000-5002,6000,6003

# Join an IP multicast group

0.0 JOIN 224.225.1.2 INTERFACE eth0

# For WIN32, use the "PORT" option

0.0 JOIN 224.225.1.2 PORT 5002

# Later, leave the group

5.0 LEAVE 224.225.1.2 INTERFACE eth0

# Incrementally ignore some receive traffic

6.0 IGNORE UDP 5000-5001

8.0 IGNORE UDP 5001,6000,6003

# More MGEN lines (terminate the flows)

10.0 OFF 1

10.0 OFF 2

# MGEN script ends here

MGEN Log File Format

The MGEN message format contains information to facilitate network performance measurements through post-analysis of MGEN log files. Some of the types of performance statistics which can be determined include:

Message Throughput	The time or arrival and size of received messages are logged by MGEN receivers. Network throughput can be assessed with this information.
Message Delivery Latency	MGEN messages contain a timestamp reflecting when they were sent and the time of reception is logged by MGEN receivers. Delivery latency statistics (jitter or absolute if clock synchronization (e.g. NTP) is possible) can be determined with this information.
Message Loss Rate	MGEN messages are sequence numbered. Message loss can be accounted for via analysis of logged sequence number information.
Message Re-ordering	The logged MGEN sequence number information can also be used to determine message re-ordering statistics.
Multicast JOIN/LEAVE Latency	The occurrence and time of JOIN/LEAVE events are logged by MGEN receivers. The JOIN latency can be determined by comparing the arrival time of the first message associated with a particular multicast group to the time the group was joined. The LEAVE latency can be determined by comparing the time of the _last_ packet arrival time to the time the receiver left that multicast group (Note that you need a packet sniffing program like tcpdump to see packets after you leave the group)

Many of the above performance measures and statistics can be measured and optionally graphed using the NRL trpr (trace plot real-time) program. This program can parse the MGEN log file format and tcpdump traces.

General Log Format

Each line of the MGEN text log file format corresponds to a unique event and follows the convention:

The <eventTime> field is in the form hrs:min:sec and represents the computer's system Greenwich Mean Time (GMT) at the time of the event.

The <eventType> field is one of the following:

RECV	Denotes the arrival of a received MGEN message.
RERR	Indicates an invalid MGEN message was received.
SEND	Denotes the transmission of an MGEN message.
JOIN	Marks a join to an IP multicast group.
LEAVE	Marks the departure from an IP multicast group.
LISTEN	Indicates when mgen began monitoring a specific port
IGNORE	Indicates when mgen ended monitoring of a specific port
START	Indicates when mgen started processing Transmission and Reception events.
STOP	Indicates when mgen stopped processing Transmission and Reception events.

Different event types will have different event attribute sets. The <event attribute> fields are explicitly labeled so that log file parsing programs can seek specific attributes of interest for given event types.

Log File RECV Events

The format of the RECV event log file line is:

<eventTime> RECV flow><flowId> seq><sequenceNumber> src><addr>/<port> dst><addr>/<port> sent><txTime> size><bytes> [host><addr>/<port>] [gps><status>,<lat>,<long>,<alt>] [data><len>:<data>]

The <eventTime> corresponds to when the message was received.

The <flowId>, <sequenceNumber>, and <txTime>, are from the payload of the MGEN message. The <txTime> is in the same format as the <eventTime> (i.e. <hr:min:sec> GMT)

The "dst" <addr>/<port> is from the message payload and corresponds to the destination address to which the source addressed the MGEN message.

The "src" <addr>/<port> is the source address determined from the corresponding recvfrom() call for UDP transport or the address to which the TCP connection was made. (An optional "host" address will be embedded in the payload by the MGEN message source and made available as an attribute of the logged RECV event in the future).

The message "size" in <bytes> is also from the payload, but for UDP transport, should also correspond to the UDP packet payload size.

The "host" <addr>/<port> corresponds to the MGEN message source's "perceived" default local address. Note that this may be different from the source address contained in the MGEN log file due to firewalls, Network Address Translation (NAT) devices, multi-homed sources, etc. The accuracy of this information depends upon the source host's configuration with regards to domain name service (DNS), etc. Note this field is optional and may not be present if this information is not valid (The current initial MGEN release does not yet support this option).

The "src", "dst", and "host" <addr> fields are dotted decimal IPv4 addresses or colon-delimited IPv6 addresses.

The global positioning system (GPS) information is available when the MGEN message source is used in conjunction with the NRL gpsLogger program. This program monitors an attached GPS receiver for position information and "publishes" it in shared memory. When mgen is run and detects that it can "subscribe" to GPS position information, it places it in the MGEN message payload. Note that gpsLogger can also be used with a pulse-per-second (PPS) capable GPS receiver to provide accurate time synchronization for hosts running the MGEN toolset. This may be useful for mobile network test environments. The MGEN log file "gps" attribute has the following comma-delimited fields:

<status>	This indicates the validity of the GPS information and may be either "INVALID", "CURRENT", or "STALE".
<lat>	This is the GPS latitude in degrees. A negative value denotes South while a positive value denotes North.
<long>	This is the GPS longitude in degrees. A negative value denotes West while a positive value denotes East.
<alt>	This is the GPS altitude in meters.

The optional "data" attribute is present only if the received MGEN message contains optional user-defined payload. If present, the <len> indicates the length (in bytes) of the user-defined payload and the <data> following the colon character':' is a hexadecimal representation of the user data where each pair of characters corresponds to one byte of user data. Thus, the number of characters in the <data> field will be 2 * <len>. (The "data" option was supported in MGEN 3.x via the MGEN Payload Manager (mpmgr) tool and is not yet supported in MGEN 4.x. The documentation will be updated when this option is supported).

Example RECV event log lines:

22:59:52.312721 RECV flow>1 seq>1 src>132.250.68.21/5000 dst>132.259.43.96/5002 sent>22:59:52.310324 size>1024

23:59:53.312721 RECV flow>1 seq>2 src>132.250.68.21/5000 dst>132.259.43.96/5002 sent>22:59:52.310324 size>1024 host>132.250.68.21/5000 gps>CURRENT,35.123,79.234,57

23:59:53.312721 RECV flow>1 seq>2 src>132.250.68.21/5000 dst>132.259.43.96/5002 sent>22:59:52.310324 size>1024 host>132.250.68.21/5000 gps>CURRENT,35.123,79.234,57 data>10:01a97b34458cff0021e8

Log File RERR Events

The format of the RERR (Receive Error) event log file line is:

<eventTime> RERR type><errorType> src><addr>/<port>

The <eventTime> corresponds to when the message in error was received. The <errorType> is one of "none", "version", "checksum", or "dstaddr". An receive error of type "version" indicates the MGEN sender is using an mgen executable with an incompatible version number. The "checksum" error indicates the received message failed checksum validation, and the "dstaddr" error indicates an invalid or unsupported destination address type in the MGEN message received. The <src> attribute indicates the source address of the message in error.

Log File SEND Events

The format of the SEND event log file line is:

<eventTime> SEND flow><flowId> seq><sequenceNumber> dst><addr>/<port> size><bytes> [host><addr>/<port>]

The <eventTime> corresponds to when the message was sent, and it should precisely match the <txTime> logged by the machine the packet is sent to, if the packet is received correctly.

All the data items are the same as those used in the Log File RECV Events.

Log File JOIN Events

The format of the JOIN log file event line is:

<eventTime> JOIN group><groupAddress> [interface><interfaceName>]

The <groupAddress> is the IP multicast group address which was joined. The format of this field is either a dotted decimal IPv4 address or a colon-delimited IPv6 address. The <interfaceName> is given only when the executed MGEN script used the INTERFACE option in the corresponding JOIN script event.

Example JOIN event log lines:

22:59:50:234757 JOIN group>224.1.2.3
22:59:51:129574 JOIN group>224.1.2.4 interface>eth1

Log File LEAVE Events

The format of log file LEAVE event lines is:

<eventTime> LEAVE group><groupAddress> [interface><interfaceName>]

The <groupAddress> is the IP multicast group address which was left. The format of this field is either a dotted decimal IPv4 address or a colon-delimited IPv6 address. The <interfaceName> is given only when the executed MGEN script used the INTERFACE option in the corresponding LEAVE script event.

Example LEAVE event log lines:

22:59:59:234757 LEAVE group>224.1.2.3
22:59:59:753683 LEAVE group>224.1.2.4 interface>eth1

Log File LISTEN Events

The format of the LISTEN event log file line is:

<eventTime> LISTEN proto><protocol> port><portNumber>

The <protocol> field corresponds to the transport protocol type being used. Currently, the only type supported is "UDP". The <portNumber> field is the host port number to be monitored.

Example LISTEN event log lines:

22:59:48:834205 LISTEN proto>UDP port>5000
22:59:49:328039 LISTEN proto>UDP port>5001

Log File IGNORE Events

The format of the IGNORE event log file line is:

<eventTime> IGNORE proto><protocol> port><portNumber>

The <protocol> field corresponds to the transport protocol type which was being used. Currently, the only type supported is "UDP". The <portNumber> field is the host port number to be no longer monitored.

Example IGNORE event log lines:

23:00:00:723467 IGNORE proto>UDP port>5000
23:01:00:235629 IGNORE proto>UDP port>5001

Log File START and STOP Events

The format of the START and STOP event log file line is:

<eventTime> START

<eventTime> STOP

These log file lines indicate the time at which MGEN began and ended its operation. The "START" time corresponds to the relative time zero for any executed scripts. This "START" time is when the mgen program was executed unless the global START command was invoked. The "STOP" command corresponds to when the mgen program was halted.

Binary Log File Format

At the beginning of binary log files, there is a plain text line to make it easy to tell what kind of file it is. It has the mgen version number, as well as the type of file ("binary_log"). This line is terminated with a line feed and a NULL ('\0') character. Following the NULL, the file contains a series of binary formatted records. There are several different types of records in the binary log file format. Each record consists of a number of fields. The first single-byte field indicates the record type. A record type of 0 is considered invalid. All multiple-byte fields are in standard network byte order (i.e. most significant byte first). Each record in the binary log file corresponds to a single unique Mgen event, just as each line in the text-based log file does. Each binary log file record contains the same information that every line of the text-format log file has. The text-format log file can actually be recreated from a binary log file using the "convert" command of mgen.

Binary Log File RECV Events

The format of the RECV event binary log file record is:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 1 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| srcPort | srcAddrType | srcAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| srcAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| messageSize | version | flags |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| mgenFlowId |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| sequenceNumber |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| txTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| txTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| dstPort | dstAddrType | dstAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| dstAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| hostPort | hostAddrType | hostAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| hostAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| latitude |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| longitude |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| altitude |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| gpsStatus | reserved | payloadLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| payload ... |

The <type> field contains the record type. The record type for RECV events is 1. For backwards compatibility, unknown record types are skipped by the binary to text log file conversion function. The <reserved> field is unused at this time and is set to 0.

The <eventRecordLength> field contains the length of this record, starting with the next byte. Thus, it contains the length of the entire record, less what will have already been read when this two-byte number is obtained.

The <srcPort> field contains the port number that the message was sent from. The <srcAddrType> field indicates the type of the source. Possible types and values include:

INVALID_ADDRESS	0
IPv4	1
IPv6	2

The <srcAddrLen> field indicates the length in bytes of source address field <srcAddr> to follow. The length should be 0 (zero) for the INVALID_ADDRESS type, 4 for IPv4 addresses, and 16 for IPv6 addresses.

The <srcAddr> contains the address to which the MGEN message was sent from. The address is in network byte order.

The rest of the packet is just a copy of the message payload from the original packet. For details on these fields, please look here.

Binary Log File RERR Events

The format of the RERR (receive error) event binary log file record is:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 1 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| srcPort | srcAddrType | srcAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| srcAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| errorType| |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The <srcPort> field contains the port number that the message was sent from. The <srcAddrType> field indicates the type of the source. Possible types and values include:

INVALID_ADDRESS	0
IPv4	1
IPv6	2

The <srcAddr> contains the address to which the MGEN message was sent from. The address is in network byte order.

The <errorType> indicates the type of message error detected and is one of the following possible error type values:

No error	0
Version number mismatch	1
Checksum validation failure	2
Message length error	3
Destination address invalid	4

Binary Log File SEND Events

SEND events are only logged if transmit logging has been turned on with the “txlog” option. The format of the SEND event binary log file record is:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 2 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| messageSize | version | flags |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| mgenFlowId |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| sequenceNumber |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| txTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| txTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| dstPort | dstAddrType | dstAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| dstAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| hostPort | hostAddrType | hostAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| hostAddr ... |

These fields are the same ones that are used in the RECV events. The <type> for SEND events is 2. Immediately following the <eventRecordLength> is a copy of the original message payload, without the GPS data and payload. The event time is left out, since it is the same as the transmit time.

Binary Log File LISTEN/IGNORE Events

LISTEN and IGNORE events both have the same format:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 3/4 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| protocol | reserved | portNumber |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

LISTEN events have a <type> of 3, while IGNORE events have a <type> of 4. The <protocol> field contains the transport protocol type to LISTEN for, while the <portNumber> field tells us what port number to LISTEN on. Possible types and values for the <protocol> include:

INVALID_PROTOCOL	0
UDP	1
TCP (unsupported)	2

Binary Log File JOIN/LEAVE Events

JOIN and LEAVE events also have an identical format:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 5/6 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| groupPort | groupAddrType | groupAddrLen |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| groupAddr ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| ifaceNameLen | asciiInterfaceName ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| asciiInterfaceName ... |

JOIN events are of <type> 5, and LEAVE events have a <type> of 6. The <groupPort> is the port which will be used to JOIN/LEAVE the group. The <groupAddrType>, <groupAddrLen>, and <groupAddr> are the type, length, and raw value of the group address, similar to those of addresses in other record types. The ifaceNameLen field contains the length in bytes of the <asciiInterfaceName> that follows it.

Binary Log File START/STOP Events

START and STOP events both have the same format:

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| type = 7/8 | reserved | eventRecordLength |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeSeconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| eventTimeMicroseconds |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

START events have a <type> 7, while STOP events are of <type> 8. They contain only the time the event occurred.

MGEN Message Payload

The MGEN message payload format is given below. Note that the Version 4.1 message format is slightly different than that of MGEN Version 4.0. The MGEN message payload is in the following format:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          messageSize          |    version    |    flags      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          mgenFlowId                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        sequenceNumber                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         txTimeSeconds                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       txTimeMicroseconds                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            dstPort            |  dstAddrType  |  dstAddrLen   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          dstAddr ...                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           hostPort            | hostAddrType  | hostAddrLen   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         hostAddr ...                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            latitude                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           longitude                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            altitude                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   gpsStatus   |  reserved   |           payloadLen            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          payload ...                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          padding ...                          |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          checksum                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

All multiple-byte fields are in standard network byte order (i.e. most significant byte first).

The <messageSize> field indicates the total size (including the <messageSize>, <version>, <flags>, etc fields) of the MGEN message in bytes. The current UDP-only transport limits this to a maximum of 8192 bytes. In the future, larger message sizes will be supported and in conjunction with the <flags> field, very large messages will be supported as a concatenation of MGEN messages to support emulation of large file transfers, etc.

The <version> field is the MGEN message protocol version number. This will enable future versions of MGEN to b backwards compatible and prevent older versions of MGEN from attempting to parse packets in unknown format.

Currently a single <flags> value (CHECKSUM = 0x01) is defined. When this flag is set, it indicates the presence of the <checksum> field at the end of the MGEN message. It is expected that additional flags will be useful as MGEN adds support for transport types besides UDP.

The <mgenFlowId> contains the flow/thread identification value associated with the MGEN flow in the corresponding script which created the flow. Note that each flow identified from an MGEN source has its own sequence number space.

The <sequenceNumber> contains the 32-bit sequence number which is incremented with each message generated for an MGEN flow. This will wrap to zero when the maximum is reached.

The <txTimeSeconds> and <txTimeMicroseconds> fields are used to mark the time of transmission of the MGEN message. The time is the source computer's system time in Greenwich Mean Time (GMT).

The <dstPort> is the destination port number to which the MGEN message addressed by the source.

The <dstAddrType> field indicates the type of destination address encapsulated in following fields. Possible types and values include:

INVALID_ADDRESS	0
IPv4	1
IPv6	2

The <dstAddrLen> field indicates the length in bytes of the destination address field <dstAddr> to follow. The length should be 0 (zero) for the INVALID_ADDRESS type, 4 for IPv4 addresses, and 16 for IPv6 addresses.

The <dstAddr> contains the destination address to which the source addressed the MGEN message. The address is in network byte order.

Note that the following fields are optional and the MGEN message length my be truncated at any point after here. Any incomplete optional fields are considered invalid.

The <hostPort> and <hostAddr> (if present and valid) contain the MGEN message source's default local address. Note that this may be different from the source address contained in the MGEN log file due to firewalls, Network Address Translation (NAT) devices, multi-homed sources, etc.

The <hostPort> is the destination port number to which the MGEN message was addressed by the source.

The <hostAddrType> field indicates the type of destination address encapsulated in following fields. The possible values are the same as for the <dstAddrType> described above.

The <hostAddrLen> field indicates the length in bytes of the destination address field <hostAddr> to follow.

The <hostAddr> contains the source's perception of its local default network address. In mgen, this is determined by a system call to gethostname(), followed by a call to name resolution. This address may be incorrect if the host is not correctly configured or domain name service (DNS) is unavailable.

The <latitude>, <longitude>, and <altitude> fields contain values corresponding to GPS information for the MGEN source if it was available. The <latitude> and <longitude> fields are encoded as follows:

<fieldValue> = (unsigned long)((<actualValue>+180.0)*60000.0)

The <altitude> field is the direct representation of the altitude value available from the source's GPS system.

The <gpsStatus> indicates the validity of the GPS information which was encoded. Possible status types and values currently include:

INVALID_GPS	0
STALE	1
CURRENT	2

In addition to the <gpsStatus> field, actual values of 999.0 for latitude and longitude, and –999 for altitude also correspond to invalid values.

The <payloadLen> field, when of non-zero value, indicates the presence of optional user-defined content in the MGEN message. The <payloadLen> value indicates the quantity (in bytes) of user-defined content which follows.

The <payload> field contains the user-defined content and is of length <payloadLen> bytes. Note that a short MGEN <messageSize> could truncate this field. If the MGEN user provides the optional user-defined content, it is up to the user to ensure that the generated MGEN messages are of sufficient size as not to truncate the <payload> content.

The <padding> portion of MGEN messages contain undefined data content.

The <checksum> field is optional and is present when the CHECKSUM (0x01) flag is set in the <flags> field. Note that corrupted messages may result in MGEN messages with the <flags> field itself corrupted, so it may be useful for MGEN implementations to have an option to validate checksums even when the CHECKSUM flag is not set if it is known that the sender is providing checksum content.

Note: The total size of the MGEN message is defined by the <messageSize> field. The optional fields may be truncated if the <messageSize> is small. The minimum MGEN message size will depend upon the IP address types being used. For example, the minimum allowed MGEN message size using IPv4 addresses with no optional fields is 28 bytes (i.e. for UDP, the UDP payload size would be 28 bytes). If GPS information is to be included without truncation, the minimum message size becomes 52 bytes with the inclusion of the <hostAddr> and GPS information. For IPv6 destination addresses, the minimum allowed MGEN message size is 40 bytes with no optional fields. If GPS information is included the minimum message size with truncating information is 76 bytes.