TimeBase Replicator Market Data Connector
A generic TimeBase-to-TimeBase stream replication connector. Unlike connectors that ingest external market data into a fixed set of message types, TimeBase Replicator copies one or more streams from a source TimeBase server into a target TimeBase server, preserving whatever custom message classes the source stream contains, without requiring the connector to know about those classes in advance.
Features
timebase_replicatorConnector KeyConfiguration
Data Connector Parameters
| Parameter | Default Value | Description |
|---|---|---|
| Source TimeBase URL | dxtick://127.0.0.1:8011 | Source TimeBase URL. |
| User Name | N/A | Source TimeBase user name. |
| Password | N/A | Source TimeBase user password. |
| Use Dedicated Connection | false | Use dedicated connection for source TimeBase. |
| Compatible With 4.3 | false | Use the option if you want to connect to 4.3 version of source TimeBase. |
| Target TimeBase URL | dxtick://127.0.0.1:8011 | Target TimeBase URL. |
| User Name | N/A | Target TimeBase user name. |
| Password | N/A | Target TimeBase user password. |
| Use Dedicated Connection | false | Use dedicated connection for target TimeBase. |
| Source Stream | N/A | TimeBase stream to replicate. |
| Target Stream | N/A | Replication Target TimeBase stream name that will be created during replication. When used inside an Aggregator process, the Replication Target Stream must differ from the Aggregator's own Target Stream (see Behavior Notes). |
| Replicate All Streams | false | Replicate all streams from the source TimeBase except system streams. Source and target stream keys are the same for every replicated stream in this mode. Cannot be combined with Specify Streams. |
| Exclude Streams List | N/A | Coma-separated streams list. For example: stream_key1,stream_key2 Only used when Replicate All Streams is enabled. |
| Specify Streams | false | Enables an explicit list of source-to-target stream pairs (see Streams List) instead of a single stream or all streams. Unlike Replicate All Streams, this mode allows the target stream name to differ from the source stream name. Cannot be combined with Replicate All Streams. |
| Streams List | N/A | Coma-separated streams list. For example: stream_key:new_stream_key,stream_key2:new_stream_key2 Only used when Specify Streams is enabled. |
| Mode | Live | Replication mode: Live (continuous replication) or Batch (bounded time-range runs driven by the Aggregator's batch schedule). |
| Reload Mode | allow | Controls what happens when the target stream's schema no longer matches the source stream's current schema (for example, after a new message type or field was added to the source). One of allow, truncate, prohibit.allow automatically migrates the target stream's schema to match the source. This requires exclusive write access to the target stream; if another process holds a write lock on it, the schema migration fails and replication of that stream stops with an error.truncate allows the source's truncation events to also truncate the target.prohibit never changes the target schema automatically; replication stops with an error if the schemas diverge, and manual intervention is required. |
| Low Latency Mode | false | Prefer to minimize latency at the expense of higher CPU usage (one CPU core per process). |
| Start From Current Time | false | Starts replication from current time. |
| Use Start Interval | false | Use start interval. |
| Interval Start | 1970-01-01 00:00:00.000 | Start time of the replication in the 24-hours date time format (YYYY-MM-DD HH:mm:SS.sss). Default value is 1970-01-01 00:00:00.000. |
| Use End Interval | false | Use end interval. |
| Interval End | 1970-01-01 00:00:00.000 | End time of the replication in the 24-hours date time format (YYYY-MM-DD HH:mm:SS.sss). Default value is 1970-01-01 00:00:00.000. |
| Replicate All Instruments | true | Replicate all instruments. |
| Instruments List | N/A | Coma-separated instruments list. For example: EQUITY:AAPL,EQUITY:EPAM,FX:EUR/USD |
| Instruments List File | N/A | An instruments list file. Each line of the file must contain an instrument. For example, file instruments.txt: EQUITY:AAPL EQUITY:EPAM FX:EUR/USD |
| Type Mapping File | N/A | File that contains additional mapping between types and fields in CSV format. Each line must contain type of field mapping in format: [SourceStream,]Class,SourceClass,TargetClass [SourceStream,]Field,SourceClass.SourceField,TargetClass.TargetField For example, file mapping.txt: Class,deltix.qsrv.hf.pub.BarMessage,deltix.timebase.api.messages.CustomBarMessage Field,deltix.qsrv.hf.pub.BarMessage.volume,deltix.timebase.api.messages.BarMessage.totalVolume SourceStream,Class,deltix.qsrv.hf.pub.BarMessage,deltix.timebase.api.messages.CustomBarMessage |
Behavior Notes
Target stream schema follows the source stream, not the Aggregator model configuration. When a target stream does not yet exist, it is auto-created with a schema copied directly from the live source stream at that moment. On every subsequent connection, the connector re-checks that the source and target schemas are still compatible, and applies the Reload Mode behavior described above if they have diverged.
The Aggregator's generic per-field/per-class data model selection (as configured for most other data connectors) does not apply to TimeBase Replicator and has no effect on which message types or fields are replicated. All replicated content is determined entirely by the source stream's own schema.
Replication Target Stream and Aggregator Target Stream must be different. When this connector is configured inside an Aggregator process, the Replication Target Stream and the Aggregator's own Target Stream serve different purposes and must point to different streams. The Aggregator's Target Stream is the stream used for Aggregator process statuses; the Replication Target Stream is the stream where replicated data is written.
Stream pairs are independent. Each source-to-target stream pair configured via Replicate All Streams or Specify Streams runs as its own independent connection with its own connection status and error reporting, so a failure in one stream pair does not affect replication of the others.
Symbology
TBD
Logging
To enable detailed logging, use these lines in the <QSHOME>/config/gflog.xml file:
<logger name="deltix.qsrv.hf.plugins.data.timebase_replicator" level="TRACE">
<appender-ref ref="safeAppender"/>
</logger>