Class for managing native timeseries database files. Extremely high speed data access. More...

#include <TSADatabase.h>

Inheritance diagram for tsa::fast::database:

Public Member Functions
virtual std::string	type (void) const override
	Returns the 'type' of the database as a string, such as "sqlite", "postgresql" or the native "fast". More...

Public Member Functions inherited from tsa::data_base
	data_base (void)
	Constructor.

virtual	~data_base (void)
	Destructor.

void	copy_table_from (data_base &source_db, const std::string &source_table_name, const std::string &target_table_name, const table_copy_params &copy_prop=table_copy_params())
	Copies a table, potentially from one database to another, according to the given table_copy_properties object, which by default copies the entire table. See class table_copy_properties for more details. More...

void	import (const file_import_rules__obsolete &import_properties)
	Imports records from a text file according to the given import properties. See class file_import_properties for more information. More...

void	print_table_info (const std::string &table_name, std::ostream &stream)
	Prints basic information about the table. More...

bool	series_columns_match (const std::string &table_name, const tsa::mem_table &m_table)
	Returns true if a database table has an identical column structure to a given mem_table. More...

Public Member Functions inherited from tsa::object
	object (void)
	Constructor.

virtual	~object (void)
	Destructor.

const char *	class_name (void) const
	Returns the class name as returned by the 'Real Time Type Information' (RTTI) mechanism. More...

bool	has_same_class_as (const object &other) const
	Returns true if self has the same type as other object. More...

Constructors / Destructor
	database (const os::path &db_path, open_db_flag_type flag=create_db)
	Constructor. Opens the database at given db_path, with given flag. More...

virtual	~database (void)
	Destructor.

Database Creation / Deletion
static bool	exists (const os::path &db_path)
	Static member that returns true if a valid database directory exists at the given db_path. More...

static void	remove (const os::path &db_path)
	Static member that removes a given database directory and the files it contains. Only use the function with a db_path argument that represents a valid database, else an exception is thrown. More...

Table Operations
table_imp_ref	table (const std::string &table_name)
	Opens a table. This operations is required before the table can be read or new records can be appended. Use the return value to initialize a fast::table instance. More...

void	create_table (const std::string &table_name, const tsa::column_defs &columns)
	Creates a new 'timeseries' table with given table_name and a given set of columns. More...

virtual void	create_series_table (const std::string &name, const tsa::column_defs &columns) override
	Creates series table. More...

void	create_table (const tsa::mem_table &mem_table, const std::string &db_table_name, replace_flag_type replace=no_replace_existing)
	Creates a new table called db_table_name by copying the structure and data from mem_table. More...

void	copy_table (const std::string &source_table_name, const std::string &copy_table_name)
	Creates a table named copy_table_name by copying the table named source_table_name. No table named copy_table_name must exist. More...

void	drop_table (const std::string &table_name, missing_table_flag_type missing_table_flag=silent_on_missing)
	Drops the table with given table_name. If missing_table_flag is set to missing_table_flag_type::throw_on_missing, then the function throws an exception if the table does not exist or is in use (opened). More...

virtual bool	drop_series_table (const std::string &table_name) override
	??????? drop_table

virtual bool	table_exists (const std::string &name) const override
	Returns true if a table with given table_name exists. More...

bool	table_exists_with_data (const std::string &table_name)
	Returns true if table with given table_name exists and contains one or more records. More...

bool	table_is_open (const std::string &table_name)
	Returns true if the table with given table_name is currently in 'opened' state. More...

bool	table_is_writeable (const std::string &name) const
	Returns true if table name can be opened for writing (by a TableWriter object). Returns false if the table is currently opened for reading or writing. (See TSA User Guide Chapter 7 - 'Table Locking') More...

bool	table_is_readable (const std::string &name)
	Returns true if table name can be opened for reading (by a TableReader object). Returns false if the table is currently opened for writing. (See TSA User Guide Chapter 7 - 'Table Locking') More...

tsa::column_defs	table_column_defs (const std::string &table_name)
	Returns a column_defs object describing the columns of the table with given table_name. More...

size_t	table_count (void) const
	Returns the number of tables currently stored in the database. More...

void	rename_table (const std::string &table_name, const std::string &new_table_name)
	Renames the table named table_name as new_table_name. No table with new_table_name must exists and table name must not currently in 'opened' state. More...

void	table_names (std::vector< std::string > &table_name_arr) const
	Populates table_name_arr with all names of tables in the database. More...

Virtual Members (from parent class data_base)
virtual date_time	series_table_first_timestamp (const std::string &table_name) override
	Returns the timestamp of the first record in the table. More...

virtual date_time	series_table_last_timestamp (const std::string &table_name) override
	Returns the timestamp of the last records in the table. More...

virtual bool	table_has_data (const std::string &table_name) override
	Returns true if the table has records, false if empty. More...

virtual tsa::column_defs	series_table_column_defs (const std::string &table_name) override
	Returns the column definition structure. More...

virtual date_time	series_table_closest_timestamp_LOE (const date_time &ts, const std::string &table_name) override
	Returns the record timestamp that is closest to, but lower than, the given target timestamp. More...

virtual void	series_load (const std::string &table_name, tsa::mem_table &data, const date_time &from_timestamp, size_t num_records) override
	Loads a number of series records from a given database table into a mem_table (memory based table) object. More...

virtual void	series_append (const std::string &table_name, const tsa::mem_table &data) override
	Appends the records from the given m_table to a database table. This operation will fail if there is overal in the range of timestamps, or if the column structure does not match. More...

virtual void	truncate_series_table (const std::string &table_name, const date_time &new_max_timestamp=date_time::min) override
	Truncates a database table by deleting all records higher than a given new_max_timestamp. More...

virtual bool	table_record_count_is_known (void) const
	Returns true if the database system knows the size (record count) of a table at all times, or whether it needs to iterate through the table to calculate the size. In the native fast::database knows the table size at all times, whereas SQL databases need to launch the 'select count(*)' command. More...

virtual int64_t	table_record_count (const std::string &table_name)
	Returns the number of records in the table. More...

Import / Export
void	export_table (const std::string &name, const os::path &dir_path, bool replace=true)
	Exports table name as a platform independent text file. The table can be imported back into a series_base via the ImportTable() member. More...

bool	import_table (const os::path &dir_path, const std::string &file_name, bool replace=false, bool error_as_exception=true)
	Imports a table from given file representing a table that was previously exported via ExportTable(). More...

Printing
void	print_table_report (std::ostream &stream)
	Printing (now performed by class table) More...

void	print_table_names (std::ostream &)
	Prints all table names to given stream. More...

void	print_table (const std::string &name, std::ostream &stream, size_t FPPrecision=8, size_t maxNumRec=tsa::print_no_limit, bool bPrintTimestamp=true)
	Prints the table name to stream as a formated table with borders and column names. The number of records written will be limited to maxNumRec. Floating point numbers are written with FPPrecision. More...

void	save_table_html (const std::string &name, const os::path &path, size_t FPPrecision=8, size_t maxNumRec=tsa::print_no_limit, bool bPrintTimestamp=true)
	Saves a table HTML. More...

void	print_table_csv (const std::string &name, std::ostream &stream, size_t maxNumRec=tsa::print_no_limit)
	Prints the table name to stream as records with comma separted values (CSV). The number of records written will be limited to maxNumRec. More...

void	print_table_csv (const std::string &name, const date_time &start, const date_time &end, std::ostream &stream)
	Prints the table name to stream as records with comma separted values (CSV). Only records with timestamps between start and end (inclusive) will be written! More...

void	print_table_vertical (const std::string &name, std::ostream &stream, bool inclTimestamp=true, size_t maxNumRecToPrint=tsa::print_no_limit)
	Prints table name in vertical format (field:value pairs) to stream. More...

void	print_graph (const std::string &table, std::ostream &stream, const std::string &column, size_t width=70, size_t height=20, const date_time &start=date_time::min, const date_time &end=date_time::max)
	Print graph. More...

void	print_hlc_graph (const std::string &table, std::ostream &stream, size_t width=70, size_t height=20, const date_time &start=date_time::min, const date_time &end=date_time::max)

void	print_table (const std::string &tableName, std::ostream &S, TablePrinter &tablePrinter, bool bPrintTimestamp, size_t maxNumRec=tsa::print_no_limit)

Detailed Description

Class for managing native timeseries database files. Extremely high speed data access.

Trading strategies are able to read and write data from multiple sources including some SQL databases like PostgreSQL. SQL databases are convenient as a central datastore since data can be manage easily and there are many tools for backing up or replicating the database.

However, for trading strategy development and optimisation a local embedded high speed data store is preferable. Class tsa::fast::database provides exactly this. Instances of this class create and manage dedicated timeseries database files. There is no 'server' involved and data is read and written directly to and from the local file system. Both read and write operations are extremely fast (millions of records per second) and at least a magnitude faster than their SQL conterparts.

Timeseries Operations The database allows the storage of timeseries data only. The: speeds derives from the fact that tables have no index as records are always stored sequentially in ascending order. There is also no 'fill-factor'. Records are always stored adjacent to each other and there are no empty spaces. This has the advantage that the range and number of records is always knows. The first and last records timestamps as well as the number of records can be retrieved without the overhead that the SQL's count(*) statement would have. The drawback of this approach is that insert operations are not supported. Records are to be appended only. In practice, when working with timeseries data, insert operations are however only rarely needed. Error correction usually involves update operations only. Alternatively, a table can be truncated, where records after a certain timestamp are erased, and the range of missing data can be rewritten with the necessary corrections and additions.

However fast::database should not be used as only and permanent store for your data. It is recommended to use PostgreSQL for such purposes, and move database to a fast::database only when required for trading system development.

Database Files The fast::database constructor creates a directory with a number of files: that are not to be modified directly by users. Access is exclusive, and any attempt to open the same database files by multiple database files will cause an exception. A database directory can be created anywhere in the file system as long as the process has the required permissions.

Backup These databases are backed up easily simply by copying the database directory and: all its content.

Constructor & Destructor Documentation

tsa::fast::database::database	(	const os::path &	db_path,
		open_db_flag_type	flag = `create_db`
	)

Constructor. Opens the database at given db_path, with given flag.

When opening an existing database, the db_path must refer to an existing database directory. This directory contains a number of files which should not be modified directly. If a directory with the given name does not exist, then this constructor will create one if the open_db_flag_type::create_db flag is passed as second argument, which it is be default. To prevent creation, pass the open_db_flag_type::no_create_db flag. This will only open an existing database. This constructor will fail if the given db_path already exists but is a file rather than a directory, or, if the directory does not contain the required database files.

os::path db_path = "c:/temp/my_db";

database db(db_path, no_create_db);

Opens Exclusively

A given database can only be opened once. Attempts to call multiple constructors on the same database directory will cause an exception.

Check Database Existence

To check if a valid database exists at the given directory use the static database::exists() member:

bool exists = database::exists(db_path);

Truncate Database

Sometime, when working with temporary databases, it is possible to truncate an existing database when opening it with this constructor by passing the open_db_flag_type::truncate_db flag. This will open the database and erase all content. This has the same effect as deleting the database and recreating one with the same name.

os::path db_path = "c:/temp/my_db";

database db(db_path, no_create_db | truncate_db);

Note how multiple constructor flags can be joined via operator|.

Remove Database

To remove an existing database, you can simply delete its directory and all its content manuall, or you can call the static database::remove() member:

database::remove(db_path);

Backup Database

To backup a database, simply copy the database directory and all its content.

Database db("myDB", "c:/db/");

Note

Database db("myDB", "c:/db/");

Flags

create_db Creates a database if one does not exist already.
no_create_db Throws an exception if the series_base does not already exist
truncate_db Clears all data in the series_base (tables, schemas, holiday regions, securities etc.
Parameters

db_path The name.

flag The flag.

Member Function Documentation

void tsa::fast::database::copy_table	(	const std::string &	source_table_name,
		const std::string &	copy_table_name
	)

Creates a table named copy_table_name by copying the table named source_table_name. No table named copy_table_name must exist.

Parameters

source_table_name	Name of the source table.
copy_table_name	Name of the new table.

void tsa::fast::database::create_series_table	(	const std::string &	name,
		const tsa::column_defs &	columns
	)

overridevirtual

Creates series table.

Parameters

name	The name of the new table.
columns	The column definition.

Implements tsa::data_base.

void tsa::fast::database::create_table	(	const std::string &	table_name,
		const tsa::column_defs &	columns
	)

Creates a new 'timeseries' table with given table_name and a given set of columns.

Parameters

table_name	The new table name.
columns	The column definition.

void tsa::fast::database::create_table	(	const tsa::mem_table &	mem_table,
		const std::string &	db_table_name,
		replace_flag_type	replace = `no_replace_existing`
	)

Creates a new table called db_table_name by copying the structure and data from mem_table.

Parameters

mem_table	A memtable object, representing a memory based table.
db_table_name	The name of the new series_base table.
replace	If false, an exception is thrown if a table with given name already exists.

void tsa::fast::database::drop_table	(	const std::string &	table_name,
		missing_table_flag_type	missing_table_flag = `silent_on_missing`
	)

Drops the table with given table_name. If missing_table_flag is set to missing_table_flag_type::throw_on_missing, then the function throws an exception if the table does not exist or is in use (opened).

Parameters

table_name	The name.
missing_table_flag	The missing table flag.

bool tsa::fast::database::exists ( const os::path & db_path )

static

Static member that returns true if a valid database directory exists at the given db_path.

Parameters

db_path The database directory.

Returns: bool.

void tsa::fast::database::export_table	(	const std::string &	name,
		const os::path &	dir_path,
		bool	replace = `true`
	)

Exports table name as a platform independent text file. The table can be imported back into a series_base via the ImportTable() member.

Parameters

name	The name of the table to export.
dir_path	The name of the directory to which table should be exported.
replace	flag - if true an existing file will be silently replaced.

bool tsa::fast::database::import_table	(	const os::path &	dir_path,
		const std::string &	file_name,
		bool	replace = `false`,
		bool	error_as_exception = `true`
	)

Imports a table from given file representing a table that was previously exported via ExportTable().

Parameters

dir_path	The directory where the file is located.
file_name	The file name, which will also be used as name for the newly created table.
replace	If true, an existing table is silently replaced, else an exception is thrown.
error_as_exception	If true, any internal error while importing the table is raised as an exception.

Returns: true if it succeeds, false if it fails.

Parameters

replace The error as exception.

void tsa::fast::database::print_graph	(	const std::string &	table,
		std::ostream &	stream,
		const std::string &	column,
		size_t	width = `70`,
		size_t	height = `20`,
		const date_time &	start = `date_time::min`,
		const date_time &	end = `date_time::max`
	)

Print graph.

Parameters

	table	The table.
[in,out]	stream	The stream.
	column	The column.
	width	The width.
	height	The height.
	start	The start.
	end	The end.

void tsa::fast::database::print_table	(	const std::string &	name,
		std::ostream &	stream,
		size_t	FPPrecision = `8`,
		size_t	maxNumRec = `tsa::print_no_limit`,
		bool	bPrintTimestamp = `true`
	)

Prints the table name to stream as a formated table with borders and column names. The number of records written will be limited to maxNumRec. Floating point numbers are written with FPPrecision.

Parameters

	name	The name of the table to be printed.
[in,out]	stream	The stream to which the table should be written.
	FPPrecision	The precision of floating point numbers.
	maxNumRec	The maximum number of records to be printed.
	bPrintTimestamp	true to print timestamp.

void tsa::fast::database::print_table_csv	(	const std::string &	name,
		std::ostream &	stream,
		size_t	maxNumRec = `tsa::print_no_limit`
	)

Prints the table name to stream as records with comma separted values (CSV). The number of records written will be limited to maxNumRec.

Parameters

	name	The name of the table to print.
[in,out]	stream	The stream to which the table is to be printed.
	maxNumRec	The maximum number of records to be printed.

void tsa::fast::database::print_table_csv	(	const std::string &	name,
		const date_time &	start,
		const date_time &	end,
		std::ostream &	stream
	)

Prints the table name to stream as records with comma separted values (CSV). Only records with timestamps between start and end (inclusive) will be written!

Parameters

	name	The name of the table to print.
	start	no records with a date_time lower than start will be written.
	end	no records with a date_time higher than end will be written.
[in,out]	stream	The stream to which the table will be written.

void tsa::fast::database::print_table_names ( std::ostream & _S )

Prints all table names to given stream.

Parameters

[in,out] parameter1 The first parameter.

void tsa::fast::database::print_table_report ( std::ostream & stream )

Printing (now performed by class table)

Prints a report, which lists all tables in the series_base, to stream.

Parameters

[in,out] stream The stream.

void tsa::fast::database::print_table_vertical	(	const std::string &	name,
		std::ostream &	stream,
		bool	inclTimestamp = `true`,
		size_t	maxNumRecToPrint = `tsa::print_no_limit`
	)

Prints table name in vertical format (field:value pairs) to stream.

Parameters

	name	The name of the table to print.
[in,out]	stream	The stream to which the table will be written.
	inclTimestamp	flag which determines whether each record's timestamp will also be written to stream.
	maxNumRecToPrint	The maximum number of records to be printed.

void tsa::fast::database::remove ( const os::path & db_path )

static

Static member that removes a given database directory and the files it contains. Only use the function with a db_path argument that represents a valid database, else an exception is thrown.

Parameters

db_path The database path.

void tsa::fast::database::rename_table	(	const std::string &	table_name,
		const std::string &	new_table_name
	)

Renames the table named table_name as new_table_name. No table with new_table_name must exists and table name must not currently in 'opened' state.

Parameters

table_name	The table name.
new_table_name	The new table name.

void tsa::fast::database::save_table_html	(	const std::string &	name,
		const os::path &	path,
		size_t	FPPrecision = `8`,
		size_t	maxNumRec = `tsa::print_no_limit`,
		bool	bPrintTimestamp = `true`
	)

Saves a table HTML.

Parameters

name	The name of the table to be printed.
path	Full pathname of the file.
FPPrecision	The precision of floating point numbers.
maxNumRec	The maximum number of records to be printed.
bPrintTimestamp	true to print timestamp.

void tsa::fast::database::series_append	(	const std::string &	table_name,
		const tsa::mem_table &	m_table
	)

overridevirtual

Appends the records from the given m_table to a database table. This operation will fail if there is overal in the range of timestamps, or if the column structure does not match.

Parameters

table_name	Name of the target databse table.
m_table	The memory based table.

Implements tsa::data_base.

void tsa::fast::database::series_load	(	const std::string &	table_name,
		tsa::mem_table &	mtable,
		const date_time &	from_timestamp,
		size_t	num_records
	)

overridevirtual

Loads a number of series records from a given database table into a mem_table (memory based table) object.

Parameters

	table_name	Name of the database table.
[in,out]	mtable	The mem_table.
	from_timestamp	The timestamp of the first record to load.
	num_records	The number of records to load.

Implements tsa::data_base.

date_time tsa::fast::database::series_table_closest_timestamp_LOE	(	const date_time &	target,
		const std::string &	table_name
	)

overridevirtual

Returns the record timestamp that is closest to, but lower than, the given target timestamp.

Parameters

target	The target timestamp.
table_name	Name of the table.

Returns: A date_time.

Implements tsa::data_base.

tsa::column_defs tsa::fast::database::series_table_column_defs ( const std::string & table_name )

overridevirtual

Returns the column definition structure.

Parameters

table_name Name of the table.

Returns: A tsa::column_defs.

Implements tsa::data_base.

date_time tsa::fast::database::series_table_first_timestamp ( const std::string & table_name )

overridevirtual

Returns the timestamp of the first record in the table.

Parameters

table_name Name of the table.

Returns: A date_time.

Implements tsa::data_base.

date_time tsa::fast::database::series_table_last_timestamp ( const std::string & table_name )

overridevirtual

Returns the timestamp of the last records in the table.

Parameters

table_name Name of the table.

Returns: A date_time.

Implements tsa::data_base.

table_imp_ref tsa::fast::database::table ( const std::string & table_name )

Opens a table. This operations is required before the table can be read or new records can be appended. Use the return value to initialize a fast::table instance.

fast::database db(...)

fast::table t = db.table("my_table");

.

Parameters

table_name Name of the database table.

Returns: A table_imp_ref which is intended to be used to initialize a table object.

tsa::column_defs tsa::fast::database::table_column_defs ( const std::string & table_name )

Returns a column_defs object describing the columns of the table with given table_name.

Parameters

table_name Name of the table.

Returns: A tsa::column_defs object.

size_t tsa::fast::database::table_count ( void ) const

Returns the number of tables currently stored in the database.

Returns: A size_t.

bool tsa::fast::database::table_exists ( const std::string & name ) const

overridevirtual

Returns true if a table with given table_name exists.

Parameters

name	The table name.

Returns: bool

Implements tsa::data_base.

bool tsa::fast::database::table_exists_with_data ( const std::string & table_name )

Returns true if table with given table_name exists and contains one or more records.

Parameters

table_name The table name.

Returns: bool.

bool tsa::fast::database::table_has_data ( const std::string & table_name )

overridevirtual

Returns true if the table has records, false if empty.

Parameters

table_name Name of the table.

Returns: true if it succeeds, false if it fails.

Implements tsa::data_base.

bool tsa::fast::database::table_is_open ( const std::string & table_name )

Returns true if the table with given table_name is currently in 'opened' state.

Parameters

table_name Name of the table.

Returns: bool

bool tsa::fast::database::table_is_readable ( const std::string & name )

Returns true if table name can be opened for reading (by a TableReader object). Returns false if the table is currently opened for writing. (See TSA User Guide Chapter 7 - 'Table Locking')

Parameters

name The name.

Returns: true if it succeeds, false if it fails.

bool tsa::fast::database::table_is_writeable ( const std::string & name ) const

Returns true if table name can be opened for writing (by a TableWriter object). Returns false if the table is currently opened for reading or writing. (See TSA User Guide Chapter 7 - 'Table Locking')

Parameters

name The name.

Returns: true if it succeeds, false if it fails.

void tsa::fast::database::table_names ( std::vector< std::string > & table_name_arr ) const

Populates table_name_arr with all names of tables in the database.

Parameters

[in,out] table_name_arr Array of table names.

int64_t tsa::fast::database::table_record_count ( const std::string & table_name )

virtual

Returns the number of records in the table.

Parameters

table_name Name of the table.

Returns: An int64_t.

Implements tsa::data_base.

bool tsa::fast::database::table_record_count_is_known ( void ) const

virtual

Returns true if the database system knows the size (record count) of a table at all times, or whether it needs to iterate through the table to calculate the size. In the native fast::database knows the table size at all times, whereas SQL databases need to launch the 'select count(*)' command.

Returns: bool.

Implements tsa::data_base.

void tsa::fast::database::truncate_series_table	(	const std::string &	table_name,
		const date_time &	new_max_timestamp = `date_time::min`
	)

overridevirtual

Truncates a database table by deleting all records higher than a given new_max_timestamp.

Parameters

table_name	Name of the table.
new_max_timestamp	The new maximum record timestamp.

Implements tsa::data_base.

std::string tsa::fast::database::type ( void ) const

overridevirtual

Returns the 'type' of the database as a string, such as "sqlite", "postgresql" or the native "fast".

Returns: A std::string.

Implements tsa::data_base.

Public Member Functions

Constructors / Destructor

Database Creation / Deletion

Table Operations

Virtual Members (from parent class data_base)

Import / Export

Printing

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation