OMQ::DbRemoteRawReceive Class Reference

A class for inbound/reading data from the remote instance from a raw SQL query, I/O is handled in a background thread and is made available in the getData() and getAllData() methods. More...

Inheritance diagram for OMQ::DbRemoteRawReceive:

Collaboration diagram for OMQ::DbRemoteRawReceive:

Public Member Methods
	constructor (DbRemoteBase remote, string sql, *softlist< auto > args, *hash< auto > options)
	constructor taking an OMQ::DbRemoteBase object for the remote connection
	constructor (string remote, string datasource, string sql, *softlist< auto > args, *hash< auto > options)
	constructor taking a string giving the name of the remote connection for the remote server
	constructor (QorusSystemRestHelper remote, string datasource, string sql, *softlist< auto > args, *hash< auto > options)
	constructor taking an OMQ::QorusSystemRestHelper object for the remote connection and assuming the "select" stream
Public Member Methods inherited from OMQ::DbRemoteReceive
	constructor (DbRemoteBase remote, string stream, string table_name, *hash< auto > options)
	constructor taking an OMQ::DbRemoteBase object for the remote connection
	constructor (DbRemoteBase remote, string table_name, *hash< auto > options)
	constructor taking an OMQ::DbRemoteBase object for the remote connection and assuming the "select" stream
	constructor (string remote, string datasource, string stream, string table_name, *hash< auto > options)
	constructor taking a string giving the name of the remote connection for the remote server
	constructor (string remote, string datasource, string table_name, *hash< auto > options)
	constructor taking a string giving the name of the remote connection for the remote server and assuming the "select" stream
	constructor (QorusSystemRestHelper remote, string datasource, string stream, string table_name, *hash< auto > options)
	constructor taking an OMQ::QorusSystemRestHelper object for the remote connection
	constructor (QorusSystemRestHelper remote, string datasource, string table_name, *hash< auto > options)
	constructor taking an OMQ::QorusSystemRestHelper object for the remote connection and assuming the "select" stream
	openStream (string stream, string table_name, *hash< auto > options)
	flushes the current stream and reopens a new stream with the same remote connection and in the same datasource
	commit ()
	Commit remote transaction.
	rollback (bool action=DO_DISCONNECT)
	Rollback remote transaction.
	disconnect ()
	Disconnects the connection.
*hash< auto >	getData (*timeout timeout_ms)
	returns queued data as soon as it is available, if the timeout is omitted or equal or less than 0, no timeout is used and the call will block until data is available on the queue; once this method return NOTHING it signifies that the end of stream data condition has been reached; do not call this method again after it returns NOTHING
*hash< auto >	getAllData (*timeout timeout_ms)
	returns all data recevied by the object in a single call, if the timeout is omitted or equal or less than 0, no timeout is used and the call will block until data is available on the queue
*list< auto >	getDataRows (*timeout timeout_ms)
	returns queued data as a list<auto> of rows as soon as it is available, if the timeout is omitted or equal or less than 0, no timeout is used and the call will block until data is available on the queue; once this method return NOTHING it signifies that the end of stream data condition has been reached; do not call this method again after it returns NOTHING
*list< auto >	getAllDataRows (*timeout timeout_ms)
	returns all data recevied by the object in a single call as a list<auto> of rows, if the timeout is omitted or equal or less than 0, no timeout is used and the call will block until data is available on the queue
OMQ::StreamConfig	config ()
	returns the configuration object
Public Member Methods inherited from OMQ::AbstractParallelReceiveStream
	destructor ()
	disconnects and aborts the I/O thread if it's still running
nothing	cleanup ()
	called when the thread resource is still allocated and the thread exits or one of Qore::throw_thread_resource_exceptions() or Qore::throw_thread_resource_exceptions_to_mark() is called
Public Member Methods inherited from OMQ::AbstractParallelStream
	destructor ()
	wait for any I/O in progress to complete
nothing	cleanupBase ()
	called by child classes when the thread resource is still allocated and the thread exits or one of Qore::throw_thread_resource_exceptions() or Qore::throw_thread_resource_exceptions_to_mark() is called
bool	inTransaction ()
	Returns True if a remote transaction is in progress.
OMQ::StreamInfo	getInfo ()
	returns performance information about the stream

Additional Inherited Members
Static Public Member Methods inherited from OMQ::AbstractParallelStream
static	staticCleanup ()
	performs static thread resource cleanup on I/O thread resources
static	checkStream (QorusSystemRestHelper helper, string op)
	checks if there is a stream operation running in the background, if so, a user-friendly exception is thrown
Public Attributes inherited from OMQ::AbstractParallelStream
const	DO_DISCONNECT = True
	A constant for rollback() methods to perform real disconnect.
const	DONT_DISCONNECT = False
	A constant for rollback() methods to keep the network connection alive.
Private Member Methods inherited from OMQ::DbRemoteReceive
	constructor (*hash< auto > options, DbRemoteBase remote, string sql, *list< auto > args)
	Private constructor.
	constructor (*hash< auto > options, string remote, string datasource, string sql, *list< auto > args)
	Private constructor.
	constructor (*hash< auto > options, QorusSystemRestHelper remote, string datasource, string sql, *list< auto > args)
	Private constructor.
	startStreamImpl ()
	opens the remote transaction if necessary
	socketThreadImpl ()
	receive reimplementation
nothing	recvDataImpl (auto rdata)
	An abstract method to handle incoming data.
nothing	recvDataDoneImpl (*string err)
	called when all data has been received or the background I/O operation terminates due to an error
	checkComplete (string meth)
	checks if data is requested after the transfer is complete
Private Member Methods inherited from OMQ::AbstractParallelReceiveStream
	abortInternMethod (string meth)
	if the I/O thread is running, then abort it, disconnect the connection, and throw a user-friendly exception what happened and how to avoid such situations in the future
	abortIntern (string meth, string fmt)
	if the I/O thread is running, then abort it, disconnect the connection, and throw a user-friendly exception what happened and how to avoid such situations in the future
Private Member Methods inherited from OMQ::AbstractParallelStream
Qore::Thread::Counter	cnt ()
	confirmation counter for the socket I/O thread
OMQ::StreamInfo	m_info ()
	stream information
	waitIoComplete ()
	waits for the I/O thread to complete and returns
	markIoInProgress ()
	marks the current connection as in progress; must be already checked beforehand that it's not in progress
	markIoComplete ()
	performs thread cleanup on the background I/O thread if any is registered and running
	checkError (string cmd)
	checks for an error in the background send thread; if there is one, then rethrows the exception in the main application / data thread
	startStream ()
	opens the remote transaction by calling startStreamImpl() and starts the background I/O socket thread
	socketThread (*ThreadLocalData my_tld)
	sets up thread-local data if possible and calls socketThreadImpl() that implements the background socket I/O for the object
	abortIo (string reason)
	signals the end of queue operations
bool	abortIoWait (string reason)
	signals the end of queue operations and waits until the socket thread terminates
	beginTransaction (OMQ::DbStreamConfig m_config)
	starts or continues a remote transaction
	commit (OMQ::DbStreamConfig m_config)
	commits a remote transaction
	rollback (OMQ::DbStreamConfig m_config)
	rolls back a remote transaction
Private Attributes inherited from OMQ::DbRemoteReceive
DbSelectStreamConfig	m_config
	stream configuration
bool	complete
	flag indicating that all data has been taken from the Queue
Private Attributes inherited from OMQ::AbstractParallelStream
Qore::Thread::Queue	dataq
	Queue for the socket I/O thread.
*hash< auto >	thread_ex
	exception info for send thread to communicate errors to the primary data thread
string	objid
	stream object identifier
const	StreamTerminatedError = "STREAM-TERMINATED"
	stream terminated error
const	ThreadKey = "_STREAM_"
	thread-local data key

Detailed Description

A class for inbound/reading data from the remote instance from a raw SQL query, I/O is handled in a background thread and is made available in the getData() and getAllData() methods.

See also

OMQ::DbRemoteReceive for information about usage

Example:

DbRemoteRawReceive recv(qrest, "omquser", sql, args, ("block": 2000));
while (*hash<auto> h = recv.getData()) {
    processData(h);
}

Note

a remote transaction is only started in this object's constructor if the transaction option is set, otherwise the remote transaction status is left unchanged

Member Function Documentation

constructor() [1/3]

OMQ::DbRemoteRawReceive::constructor	(	DbRemoteBase	remote,
		string	sql,
		*softlist< auto >	args,
		*hash< auto >	options
	)

constructor taking an OMQ::DbRemoteBase object for the remote connection

Example:

DbRemoteRawReceive recv(db, sql);
while (*hash<auto> h = recv.getData()) {
    processData(h);
}

Parameters

remote	an OMQ::DbRemoteBase object
sql	the SQL for the select statement in the remote DB
args	optional bind arguments for the SQL statement
options	optional Streaming API Constructor Options as follows: "block": data block size; the number of rows to retrieve from the remote database and send in each block "block_queue_size": the number of blocks to queue for receiving before the I/O thread will block (default: 2); the total number of rows that can be queued = block_queue_size * block "timeout": an HTTP socket timeout value in milliseconds; used locally and in the remote for socket I/O and queue operations; default value: 5m "loglevel": a default log level option for logging; see Log Levels for valid value "no_remote_timeout": if True the "timeout" option will not be sent to the remote "transaction": start a remote transaction; if this option is not given, the remote transaction status is left unchanged

Note

• this class always uses the system.sqlutil service stream: "select_raw"
• a remote transaction is only started in this object's constructor if the transaction option is set, otherwise the remote transaction status is left unchanged
• the explicit or default timeout value here overrides any socket I/O timeout set for the remote connection object

constructor() [2/3]

OMQ::DbRemoteRawReceive::constructor	(	QorusSystemRestHelper	remote,
		string	datasource,
		string	sql,
		*softlist< auto >	args,
		*hash< auto >	options
	)

constructor taking an OMQ::QorusSystemRestHelper object for the remote connection and assuming the "select" stream

Example:

DbRemoteReceive recv(qrest, datasource_name, table_name);
while (*hash<auto> h = recv.getData()) {
    processData(h);
}

Parameters

remote	an OMQ::QorusSystemRestHelper object
datasource	a string with name of the remote datasource to use
sql	the SQL for the select statement in the remote DB
args	optional bind arguments for the SQL statement
options	optional Streaming API Constructor Options as follows: "block": data block size; the number of rows to retrieve from the remote database and send in each block "block_queue_size": the number of blocks to queue for receiving before the I/O thread will block (default: 2); the total number of rows that can be queued = block_queue_size * block "timeout": an HTTP socket timeout value in milliseconds; used locally and in the remote for socket I/O and queue operations; default value: 5m "loglevel": a default log level option for logging; see Log Levels for valid value "no_remote_timeout": if True the "timeout" option will not be sent to the remote "transaction": start a remote transaction; if this option is not given, the remote transaction status is left unchanged

Note

• this class always uses the system.sqlutil service stream: "select_raw"
• a remote transaction is only started in this object's constructor if the transaction option is set, otherwise the remote transaction status is left unchanged
• the explicit or default timeout value here overrides any socket I/O timeout set for the remote connection object

constructor() [3/3]

OMQ::DbRemoteRawReceive::constructor	(	string	remote,
		string	datasource,
		string	sql,
		*softlist< auto >	args,
		*hash< auto >	options
	)

constructor taking a string giving the name of the remote connection for the remote server

Example:

DbRemoteRawReceive recv(remote_name, datasource_name, sql);
while (*hash<auto> h = recv.getData()) {
    processData(h);
}

Parameters

remote	a string giving the name of the remote connection for the remote server
datasource	a string with name of the remote datasource to use
sql	the SQL for the select statement in the remote DB
args	optional bind arguments for the SQL statement
options	optional Streaming API Constructor Options as follows: "block": data block size; the number of rows to retrieve from the remote database and send in each block "block_queue_size": the number of blocks to queue for receiving before the I/O thread will block (default: 2); the total number of rows that can be queued = block_queue_size * block "timeout": an HTTP socket timeout value in milliseconds; used locally and in the remote for socket I/O and queue operations; default value: 5m "loglevel": a default log level option for logging; see Log Levels for valid value "no_remote_timeout": if True the "timeout" option will not be sent to the remote "transaction": start a remote transaction; if this option is not given, the remote transaction status is left unchanged

Note

• this class always uses the system.sqlutil service stream: "select_raw"
• a remote transaction is only started in this object's constructor if the transaction option is set, otherwise the remote transaction status is left unchanged
• the explicit or default timeout value here overrides any socket I/O timeout set for the remote connection object

---

The documentation for this class was generated from the following file:

• Streams.qc