Data Monitor Project Specifications

1. Introduction
   The data monitor is an engineering tool for the display of real time engineering data. It is designed to be configurable: the messages and message fields to be received, the display of data on screen, the data to be recorded to file or other data sinks are settings that are stored in the configuration file. It is also designed to be extendable: new message protocols, display widgets and recording data sink can be added using a plugin system.
    
2. Deliverables
   1. Design [D1]: The following design descriptions shall be provided: Application/Display Configuration file format, Message Configuration file format, Export Data Sink Configuration file format, Class diagrams with key virtual functions for plugin system
   2. Display feature [D2]: The application's display widgets, associated page and widget property dialogs, on-line addition, modification and deletion of widgets, saving of widget settings into configuration file
   3. Receiving data feature [D3]: The application's support of one message protocol via plugin, factory for creation of message objects, and integration to display widget to display real time data
   4. Recording and playback [D4]: Recording of data to data sink and playback of data at various speeds. Bookmark system
       
3. General Requirements
   1.  All deliverables shall include full source code.
   2. The source code shall be in Standard ANSI C++ (C++98).
      Unless otherwise stated, the source code shall compile without warnings on two reference platforms: Microsoft Visual C++ 9.0 (Windows) and gcc 3.4 (Linux).
   3. The application shall use the Qt 4.x library.
   4. For additional widgets, such as plots, the Qwt library shall be used.
   5. The application shall work in both Windows and Linux.
   6. Build management shall be handled using Qt pro files.
   7. The customer will provide a library to facilitate the handling of data. However, use of the library is not mandatory (but preferred) if a similar features can be provided. This library will be referenced to describe subsequent requirements in this document and will be referred as the Library. This library is further described in the annex.
       
4. Application Look-and-Feel
   1. The application shall have no menu bar, tool bars or status bar. The display widgets shall fill the entire application window.
   2. The application shall have pages. Each page will display a unique set of display widgets, each showing its respective data in real time. The user shall be able to toggle between pages.
   3. Each page will be divided into columns. Each page can have its own number of columns. By default, the columns will be uniformly spaced. However, the user shall be able to designate a fixed width for certain columns. Columns without a fixed width will be uniformly sized using the remaining space.
   4. By default, a page will have a least one column. Display widgets will be placed in columns, stacking from the top of the window to the bottom. If widgets exceed the height of the window, they will be clipped.
   5. A page property dialog shall allow configuration of pages. The page properties are: page name, background colour, keyboard hot-key, number of columns and optionally the width of specific columns.
   6. All functions of the application are accessible from keyboard hot-keys. Hot-keys that can be assigned to pages include all alpha-numeric keys on the keyboard, and their Alt, Ctrl and Shift variants. Function keys are reserved for application use. An incomplete list follows:
      1. F1 - Help dialog: Simple dialog listing all the function keys and their functions. It will also show the hot-keys for all available pages. No help system is required.
      1. Shift F1 - User help page: display the contents of a user defined text help file.
      2. F2 - Page Properties: Call up the page property dialog for the current page.
      3. F3 - Add a new page
      4. F4 - Add a new display widget to the current page
      5. F5 - Edit display widget properties: A selector to select a display widget on the page then the property dialog for the widget
      6. F6 - Open a playback file, or toggle back to realtime mode
      7. F7 - Manage bookmark triggers and recording settings
      8. F12 - Exit application
          
5. Display Widgets
   1. Common Properties: All widgets have a text label, text foreground colour. The implementation of a display widget will involve:
      1. A widget property dialog to allow the entry of widget settings, and persistence of widget settings into a configuration file.
      2. Implementation of the display widget itself
      3. A customised observer (based on the Observer Pattern) that will attach to the designated data fields and update the display widget. The base class for the observer is provided in the Library.
   2. Static Widgets - this widgets are used to beautify the page, they do not display any real values.
      1. Header + Spacer - Optional widget height in pixels
   3. Dynamic Widgets
      1. Integer LED indicator - LED colour rules based on integer- comparisons (equal, less-than, equal or less-than). note that more than one rules shall be supported for this and subsequent widgets.
      2. Integer Field - field colour rules based on integer- comparisons (equal, less-than, equal or less-than) and choice of display in either decimal or hexidecimal. Optional scaling factor and offset to be applied before display.
      2. List Field - Configurable display text for each integer index. Undefined index default to numerical value of index.
      3. String Field - configurable field colour
      4. Float Field -  field colour rules based on double- comparisons (less-than, equal or less-than) and choice of number of decimal places. Optional scaling factor and offset to be applied before display.
      5. Float Plot - Up to ten concurrent plots of doubles. Configurable colour for each plot. Optional scaling factor and offset to be applied before plotting for each plot. The plot (y-axis) will be against time (x-axis).
      6. Binary Plot - Up to ten plots stacked vertically with common x/y-axes. Configurable colour for each plot. Pairs of dynamic data (one for x axis and one for y axis) shall be used.
   4. Control Widgets
      1. Push Button - Configurable to assign an integer value to a field when pushed
      2. Toggle Button - Configurable to assign an integer value to a field when toggled on, and another value when toggled off.
      2. List Control - Configurable display text and assigned integer value to a field.
   5. The instantiation and placement of the widgets in a page will be handled via an Abstract Factory pattern. The application will read from a configuration file and call concrete factories provided by a plugin to instantiate the widgets with the required properties. A plugin shall be developed for all widgets described in this section.

 



 

6. The application shall support many to many relation between data field in payload to the display widget. An example is the same time tag which is packed in more than one message and displayed in more than one page. In this example, the last arriving time tag from any message that is received will be displayed.
7. Dynamic widgets should be updated at a frequency of not less than 3Hz.

 

6. Data Types
   1. Packing and Unpacking of messages are facilitated by the Library. The Library provides a base class for all data values: value_var. This class defines the interfaces to convert to integer values (to_int(), to_unsigned_int()), convert to floating values (to_double()), convert to string (to_const_char_ptr()). These conversion methods are used by the respective display widgets.
   2. The application shall be designed to work with all subclasses of value_var (or equivalent) as long as the respective conversion methods are implemented. If an alternative library is used following basic data types shall be supported:
      1. Integer
      2. Unsigned Integer
      3. Short
      4. Unsigned Short
      5. Char
      6. Unsigned Char
      7. Float
      8. Double
      9. String
      10. Bitfields (supports bit width of 1 to 32) 
      11. Variable length arrays of basic data types
   3. Nested structures in messages are supported by the Library using the class: complex_var. This class implements the Composite Design Pattern. It calls each child component in turn for various functions, such as message extraction and packing. Each child component are assessable via the operator[], providing a basic form of reflection capability. Support for variable length arrays are facilitated by a sub-class of complex_var: flex_var. This is a template class that can be specialized to any type. Messages are also sub-classes of complex_var. If an alternative implementation is used, support for nested structures shall be implemented.
   4. A XML compliant configuration file shall describe all messages and fields and nested structures within each message. The format for this file will be reviewed by the customer in D1. The data monitor application shall parse the configuration file and instantiate the messages as required using an Abstract Factory Pattern. Concrete factories from plugins will instantiate the various data types. A concrete factory shall be developed to instantiate the following data types (or equivalent):
      1. generic_var<char> : 8 bit signed integer
      2. generic_var<unsigned char> : 8 bit unsigned integer
      3. generic_var<short>: 16 bit signed integer
      4. generic_var<unsigned short>: 16 bit unsigned integer
      5. generic_var<int>: 32 bit signed integer
      6. generic_var<unsigned int>: 32 bit unsigned integer
      7. generic_var<long long>: 64 bit signed integer
      8. generic_var<unsigned long long>: 64 bit unsigned integer
      9. generic_var<float>: 32 bit float
      10. generic_var<double>: 64 bit float
      11. vstring_var: string (currently not part of the library, will be provided by the customer if required)*
      12. vbitfield_var: bitfield
      13. array_var: variable length arrays of basic data types
      14. complex_var: the composite class
           
          *currently not part of the library. The library has a template class for strings (string_var) as the message structure is automatically generated in applications and there is no provision for arguments to be passed during object construction
           
   5. The factory shall also provide special handling of the flex_var class, possibly by providing a special sub-class of complex_var that is capable of interfacing with the factory to produce nested structures of fields. The treatment of flex_vars shall be like arrays. 

7. Message Protocols
   1. The purpose of the message protocol is to adapt to the different communication mediums, message protocols to channel the correct payload to the data type object for extraction and display.

 



 

2. The message protocol object is a subclass of the  data type composite object (i.e. the Library's complex_var). It "has-a" data type objects that describe, pack and extract the fields in the payload of a message.
3. The message protocol object contains additional properties that describe the message, for example, the message ID, flag to indicate whether a message needs to be sent out in the next cycle, etc.
4. Like data type objects, message protocol objects are also instantiated by a concrete factory provided by a message type plugin. A message protocol plugin will typically provide a concrete factory for a protocol or a family of related protocols.
5. The message protocol factory "is-a" data type abstract factory. It uses parameters provided to it by the application to instantiate the correct message protocol object, initialized with the appropriate properties. These parameters shall come from the same XML configuration file using the process as described in item 6.4.
6. Message protocol plugins are also data type plugins. However, they export three additional methods: initialize, input and output.
7. Initialize is called for each message protocol plugin once all message protocol objects that the application needs have been instantiated. This is to allow the message protocol plugin to take stock of all objects and perform any housekeeping if required. Initialize can also be called midway during the execution of the application if new message protocol objects are added or removed. Depending on the specific implementation, this may cause the "connection" with the remote to be disconnected.
8. Input and Output are called for each message protocol plugin periodically at an interval of 50msec. These will check for messages from a remote party or send messages to the remote party respectively. Each plugin will be call consecutively. There will be no requirement for initialize, input and output to be thread-safe.
   
8. Selection of Message Fields
   1. Dynamic widgets (5.3) are associated with message fields to display. Control widgets (5.4) are associated with message fields and messages to send out. The Data Explorer shall provide a standard dialog for browsing and selection of message fields.

 



 

2. Static arrays ("street_name" in the figure above) shall have a spin box to allow the selection of the element of the array to display. The spin box shall limit to all selectable indices only.
3. Dynamic arrays which include flex_var and array_var ("timecharge" (flex_var), "daily_breakdown" (array_var) )  shall have a spin box that allow selection of an element to display. The spin box shall limit up to the 16384-th element.
4. The selection of a complex_var or its subclass ("employee_record", "address", "timecharge" (flex_var) ) shall display the number of children. This also applies for array_var.
5. array_vars shall have two special sub-fields for selection. The <<string>> field shall display the entire contents of the array_var as a string. The <<value>> field shall display the value of the element specified by the index in the spin box.
6. The design shall separate the functionality to implement this selection into:
   1. Widget plugin: Provides the widget property page and a button for the selection of message fields, calls the Data Explorer application to display a selection dialog for message fields. Retrieves a reference to the data type object that it should attach to for display updates when there are changes.
   2. Data Explorer: Provides the selection dialog for message fields that can be called by widget plugins. It will query all the message protocol plugins to build a message tree for selection. It shall call the Message Protocol plugin with a selection and retrieve the reference to a data type object that it will pass to the widget plugin.
   3. Message Protocol plugin: Provides a message tree when queried by the Data Explorer application. Receives a selection of an object from the tree and translates this to a reference to a data type object.

 



 

 

9. Recording
   1. All received data shall be recorded into a data sink. A data sink can be a log file or an export file.
   2. For export file data sink:
      1. This is a tab or comma delimited text file that can be opened with Microsoft Excel, gnuplot and Matlab.
      2. The export file shall contain an optional header line that starts with the '#' character. The header line contains the name of the fields with all delimiter characters in the name replaced with a configurable character.
      3. The application shall supports the logging of integer, floating point and string value. Delimiter character in the string shall be replaced with a configurable character. Integer and floating point data shall have an optional scaling factor and offset before export.
      4. A dialog will be provided to allow the user to configure the fields that are to be exported. The application shall support unlimited columns (only limited by heap). For this work, up to 50 columns will be tested. The configuration will be automatically saved when the user clicks "OK" or "Apply".
      5. A XML compliant configuration file shall describe all fields exported to the export file, the delimiter character, the presence of header line, delimiter replacement character and whether the export file is enabled. The format for this file will be reviewed by the customer in D1.
      6. The export function can be implemented as a observer of the message protocol object which contain the data that is being exported.
      7. The observer objects are instantiated by a concrete factory parsing the XML configuration file.
      8. The data monitor application shall support more than one export file data sinks. Each export file will have a filename that is specified in the configuration dialog. A special escape sequence can be used to denote a unique increasing integer number to create a unique name at every run of the application. (This increasing integer shall be automatically derived by scanning the directory for matching sequences.)
      9. The configuration dialog shall allow the management of all export file sinks, including a feature to enable or disable them. This will be saved in the global application configuration.
   3. For log file data sink:
      1. All data received shall be recorded, including data that are not displayed and data that arrives too quickly to be displayed (note: requirement for data received is  20Hz, while requirement for display is 3Hz)
      2. All log data shall be time tagged relative to the time of start of logging.
      3. The log file shall have a fixed filename appended with a unique increasing integer number to create a unique name at every run of a session. The export filename, when the escape sequence is used, shall use the same number.

 

10. Playback
    1. Only the log file needs to have playback capability. Export files cannot be played back.
    2. Data monitor application shall be able to playback from log file data sink in speed interval of 0x, 0.1x (100ms:1s), 0.25x, 0.5x, 1x (real-time), 2x, 5x, 10x, 20x, 30x and 60x (1 minute:1s) the actual speed, in both forward and reverse directions. In 0x mode, the user will press a key to step to the next message on a frame by frame basis. (note: 0x = pause)
    3. A "find" feature will be implemented in playback mode. This functions like a "find" dialog; it allows searching forward or backward from the current position. Instead of a text search, the features shall allow the selection of a field and a satisfying condition. The rules will have a fix format of checking data object value (equal, not equal, less than, equal or less-than, etc) against entered values. The rules can be linked together by either 'and' or 'or' conditions.
    4. Data monitor application shall be able to skip to the next or previous bookmark position.
    5. By default, there shall be no recording to export files in playback mode. The user shall be able to toggle recording to export files during forward playback.  There shall be no missed or skipped data in recording to export files in fast forward playback from the log file. There is no need for any form of recording to export files in reverse playback. The export file configuration used in playback mode is the same as in realtime mode.
    6. Possible keys definitions: 
       1. space - pause/play(previous speed) toggle
       2. [ ] - speed increment/ decrement
       3. . - playback in pause/1x mode toggle
       4. Enter /  ' - one step forward / backward in pause mode
       5. backspace - reverse direction of playback
       6. F8 - "Find" dialog box to define find rules
       7. Shift-F9 / F9 - search backward / forward based on previous find rules
       8. Shift-F10 / F10 - skip backward / forward to bookmark
       9. Shift-F6 - open another playback file
    7. The application shall be able to playback files as long as: the appropriate message plugin is available, and the XML configuration file for all the messages and fields is the same as during the recording. All other configuration (eg. display widget or export file) changes shall not affect the ability to playback and display data correctly.

 

11. Bookmark system
    1. Each bookmark has a name and identifies a position in the log file. The bookmarks for the start and end position of the file shall be predefined.
    2. Bookmarks can be created during recording when bookmark rules are matched and these are embedded into the log file.
    3. Bookmarks can also be created by the user by pressing a keystroke (e.g. F11).
    4. A bookmark rule will trigger only once in a session. Once triggered, it will become dormant. There shall be two ways to reset all bookmark rule triggers: by entering a new session, or by user keystroke (eg. Shift-F11).
    5. The application shall have a configuration dialog for the entry of bookmark rules. Each rule shall consist of a data field, a comparison operation (equal, not equal, less than, equal or less than, etc) and a entered constant.
    6. Up to three rules can be chained via either AND or OR conditions.
    7. Unlimited bookmark rule chains (subject to heap limitations) shall be supported. When a rule chain evaluates to true, a bookmark shall be embedded.

 

12. Session
    1. The application shall have two modes of operation: realtime mode and playback mode.
    2. The application shall start in realtime mode, unless the command-line arguments indicate otherwise.
    3. A new session begins when the application starts, or when switching to another mode. When switching to another mode, the previous session is also ended.
    4. When the application exits, the current session is ended.
    5. When a session is ended:
       1. Log file is closed (only if session is in realtime mode)- if no data has been recorded, the log file is deleted
       2. Export(s) file are closed (if any) - if no data has been recorded, the export file is deleted
       3. Bookmark triggers are reset if session is in realtime mode
    6. When toggling between playback mode and realtime mode, or when loading a playback file, the last playback location is discarded. The playback can only resume from the beginning.
    7. When a realtime session is started:
       1. New log file is opened.
       2. Export file(s) are opened (according to export file(s) configuration)

 

13. Application Files
    1. One application/display configuration file storing the page and display widget configuration, book mark rules and enabled export sinks.
    2. One or more message configuration files - to store message and field structure
    3. One or more message protocol plugin
    4. Zero or more export configuration files - to store export file settings

 

14. Application robustness
    1. If messages are received by a message protocol plugin and an unrecognised message ID is detected, the message shall still be recorded in the log file. It shall not be displayed, nor export, nor trigger a bookmark.
    2. If messages are received by a message protocol plugin and the length is unexpected (either longer or shorter), the message shall still be recorded in the log file. If the message is long enough for the displayed or bookmarked field, the message shall be handled according to the rules (export, bookmark, display).
    3. If a message is read during playback from a log file and an unrecognised message ID is detected, there is no need for display, or export.
    4. If  a message is read during playback from a log file and  the length is unexpected (either longer or shorter), the message shall be handled according to the rules (export, bookmark, display) if there is sufficient length for the monitored field.
    5. If a dynamic display widget rule contains fields or messages that cannot be found in the message configuration, the widget shall be disabled (i.e. visibly grayed-out). The application shall continue to function otherwise.
    6. If a bookmark rule contains fields or messages that cannot be found in the message configuration, the rule shall be silently ignored. The application shall continue to function otherwise.
    7. If an export sink rule contains fields or messages that cannot be found in the message configuration, the column of data shall be omitted.  The application shall continue to function otherwise.
    8. If there are changes to the message configuration file(s), where new messages are inserted or appended, or new fields appended to existing messages, or previously unused messages (i.e. not referenced by any rule) removed, all application features should not be affected.
    9. The application shall permit the playback of log files that were recorded by a session with unrecognized message ID and incorrect field descriptions, to playback correctly when the appropriate message IDs and field descriptions are added / amended to the message configuration files.
    10. The application shall allow playback to display data that were not previously displayed or wrongly display during recording as appropriate display widgets or display widget rules was then unavailable (eg. scaling factor was wrong) .
    11. The application may receive data packets at rates higher than 40Hz, for example at 100Hz. All data packets should be recorded.