abstract class, defines interface for data access implementations More...

#include <data_accessor.hpp>

Inheritance diagram for FST::Data_Accessor< DATATYPE, IDXTYPE >:

Classes
struct	Block
	one consecutive block of patterns as yielded by Block_Range/blocks() More...
class	Block_Range
	RAII range over consecutive data blocks of one DataPart in the current split. More...

Public Types
typedef const DATATYPE *	PPattern
	one data sample: a raw pointer to its (read-only) feature vector, getNoOfFeatures() entries long

Public Member Functions
virtual unsigned int	getNoOfClasses () const =0
	returns number of classes
virtual unsigned int	getNoOfFeatures () const =0
	returns data dimensionality
virtual IDXTYPE	getClassSize (const unsigned int c) const =0
	returns size (number of samples in) of class c
virtual IDXTYPE	getClassSizeSum () const =0
	returns summed size (number of samples in) of all classes, i.e., no. of all patterns in data
virtual void	setClass (const int c)=0
	sets active class from which get*Block() will fetch data
virtual int	getClass () const =0
	returns active class
virtual unsigned int	getNoOfSplits () const =0
	data access iteration (to support, e.g., loops in cross-validation)
virtual bool	getFirstSplit ()=0
	data access iteration (to support, e.g., loops in cross-validation)
virtual bool	getNextSplit ()=0
	data access iteration (to support, e.g., loops in cross-validation)
virtual unsigned int	getSplitIndex () const =0
	data access iteration (to support, e.g., loops in cross-validation)
virtual IDXTYPE	getNoOfBlocks (const DataPart ofwhat) const =0
	returns the number of consecutive blocks the requested DataPart is made of in the current split
virtual bool	getFirstBlock (const DataPart ofwhat, PPattern &firstpattern, IDXTYPE &patterns, const unsigned int loopdepth=0)=0
	returns pointer to first consecutive block of data of requested DataPart type in the current split (access iteration)
virtual bool	getNextBlock (const DataPart ofwhat, PPattern &firstpattern, IDXTYPE &patterns, const unsigned int loopdepth=0)=0
	returns pointer to next consecutive block of data of requested DataPart type in the current split (access iteration) Must be preceded by getFirstBlock() on the same `loopdepth`; returns false once the blocks are exhausted.
Block_Range	blocks (const DataPart ofwhat)
	RAII block range over the current split - prefer over the deprecated getFirstBlock()/getNextBlock() cursor.
virtual IDXTYPE	getBlockIndex (const unsigned int loopdepth=0) const =0
	returns index of the current consecutive block of data of requested DataPart type in the current split (access iteration)
virtual IDXTYPE	getNoOfPatterns (const DataPart ofwhat) const =0
	returns number of patterns in all consecutive blocks of data of requested DataPart type in the current split (access iteration)
virtual void	substitute (const DataPart source, const DataPart target)=0
	enables change of meaning of DataPart types, for use in specialized data access scenarios like in bias predicting wrappers Installs a remapping so that every later request for `source` is served with `target's` data instead. The mapping is a property of the accessor, applies to all classes and splits, and stays in force until resubstitute() – so a component that substitutes must restore the mapping before handing the accessor on. Mappings do not chain: substituting TRAIN->TEST and then TEST->ALL makes TRAIN yield TEST's data, not ALL's.
virtual void	resubstitute ()=0
	resets standard DataPart types' meaning, i.e. undoes every substitute() call
Data_Accessor *	clone () const override=0
Data_Accessor *	sharing_clone () const override=0
Data_Accessor *	stateless_clone () const override=0
Public Member Functions inherited from FST::Clonable
bool	is_clone () const
bool	is_sharing () const
	true if this instance is a sharing_clone() result (referencing, not owning, the source's structures)
bool	is_sharing_clone () const
	true if this instance is BOTH a clone and sharing – i.e. specifically a sharing_clone() result, not a clone()/stateless_clone() one
virtual std::ostream &	print (std::ostream &os) const
	writes a human-readable identification of the object; base default writes nothing

Private Member Functions
unsigned int	acquire_block_loop_slot ()
	Reserves and returns a free loopdepth (reusing a released one if available, else growing the pool by one).
void	release_block_loop_slot (const unsigned int depth)
	Returns the loopdepth slot acquired by acquire_block_loop_slot() so a future Block_Range can reuse it.

Private Attributes
std::vector< bool >	_block_slot_used

Static Private Attributes
static const unsigned int	_first_acquired_loopdepth = 4

Additional Inherited Members
Protected Member Functions inherited from FST::Clonable
void	set_cloned (const bool cloned=true)
	marks/unmarks this instance as a clone; call from clone()/stateless_clone() overrides (Clonable_Impl does this automatically)
void	set_sharing (const bool sharing=true)
	marks/unmarks this instance as sharing (see is_sharing())
void	set_sharing_cloned (const bool cloned=true, const bool sharing=true)
	sets both flags at once; call from sharing_clone() overrides (Clonable_Impl does this automatically)

Detailed Description

template<typename DATATYPE, typename IDXTYPE>
class FST::Data_Accessor< DATATYPE, IDXTYPE >

abstract class, defines interface for data access implementations

A Data_Accessor presents the data set as a set of classes, each split into a training and a test part, and hands the data out in consecutive blocks of patterns.

Usage protocol. Reading always follows the same three nested levels, in this order:

select the class with setClass(c), c in 0..getNoOfClasses()-1;
iterate over splits with getFirstSplit() / getNextSplit() (one iteration for a plain hold-out, k for k-fold cross-validation, and so on) – both return false once the splits are exhausted;
for the current class and split, read the data in blocks: either for(const auto &b : da->blocks(FST::TRAIN)) {...}, or the deprecated getFirstBlock()/getNextBlock() cursor pair.

The split iteration is global to the accessor, not per class: setClass() must be re-issued for each class INSIDE the split loop, and a split loop must not be restarted while blocks of that split are still being read.

Memory layout. A block is a pointer to the first pattern plus a pattern count. Patterns are stored consecutively, and within a pattern the getNoOfFeatures() feature values are consecutive too. So for a block (p, n): pattern i starts at p + i*getNoOfFeatures(), feature j of that pattern is p[i*getNoOfFeatures()+j], and walking one feature across the block means striding by getNoOfFeatures(). The data is valid only until the next split change or the next block call on the same loop slot, and the accessor keeps ownership of it – never free it.

Thread safety. A single instance must not be driven from several threads; each thread takes its own sharing_clone() (see Clonable), which shares the read-only sample storage but keeps a private iteration state.

Member Function Documentation

◆ clone()

template<typename DATATYPE, typename IDXTYPE>

Data_Accessor * FST::Data_Accessor< DATATYPE, IDXTYPE >::clone ( ) const

overridepure virtual

create 1:1 independent clone of the current object. The caller owns the returned object. Families that cannot support a given clone kind implement it to throw fst_error; Clonable_Impl's CLONE_INHERIT slot defers to such a body.

Implements FST::Clonable.

◆ getBlockIndex()

template<typename DATATYPE, typename IDXTYPE>

virtual IDXTYPE FST::Data_Accessor< DATATYPE, IDXTYPE >::getBlockIndex ( const unsigned int loopdepth = 0 ) const

pure virtual

returns index of the current consecutive block of data of requested DataPart type in the current split (access iteration)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getClass()

template<typename DATATYPE, typename IDXTYPE>

virtual int FST::Data_Accessor< DATATYPE, IDXTYPE >::getClass ( ) const

pure virtual

returns active class

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getClassSize()

template<typename DATATYPE, typename IDXTYPE>

virtual IDXTYPE FST::Data_Accessor< DATATYPE, IDXTYPE >::getClassSize ( const unsigned int c ) const

pure virtual

returns size (number of samples in) of class c

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getClassSizeSum()

template<typename DATATYPE, typename IDXTYPE>

virtual IDXTYPE FST::Data_Accessor< DATATYPE, IDXTYPE >::getClassSizeSum ( ) const

pure virtual

returns summed size (number of samples in) of all classes, i.e., no. of all patterns in data

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getFirstBlock()

template<typename DATATYPE, typename IDXTYPE>

virtual bool FST::Data_Accessor< DATATYPE, IDXTYPE >::getFirstBlock	(	const DataPart	ofwhat,
		PPattern &	firstpattern,
		IDXTYPE &	patterns,
		const unsigned int	loopdepth = 0 )

pure virtual

returns pointer to first consecutive block of data of requested DataPart type in the current split (access iteration)

Parameters

ofwhat	which part of the split to read
firstpattern	receives a pointer to the block's first pattern (owned by the accessor)
patterns	receives the number of patterns in the block
loopdepth	the cursor slot to use; two block loops that overlap in time MUST use different slots or their state collides and the behaviour is undefined. Slots 0..3 are the ones available here.

Returns: false if the requested part is empty in the current split

Deprecated: Deprecated since FST4 4.0; prefer the RAII blocks() range, which acquires a private cursor slot automatically instead of leaving the choice to the caller. Removal planned for the next major release.

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >.

◆ getFirstSplit()

template<typename DATATYPE, typename IDXTYPE>

virtual bool FST::Data_Accessor< DATATYPE, IDXTYPE >::getFirstSplit ( )

pure virtual

data access iteration (to support, e.g., loops in cross-validation)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNextBlock()

template<typename DATATYPE, typename IDXTYPE>

virtual bool FST::Data_Accessor< DATATYPE, IDXTYPE >::getNextBlock	(	const DataPart	ofwhat,
		PPattern &	firstpattern,
		IDXTYPE &	patterns,
		const unsigned int	loopdepth = 0 )

pure virtual

returns pointer to next consecutive block of data of requested DataPart type in the current split (access iteration) Must be preceded by getFirstBlock() on the same loopdepth; returns false once the blocks are exhausted.

Deprecated: Deprecated since FST4 4.0; prefer the RAII blocks() range. Removal planned for the next major release.

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >.

◆ getNextSplit()

template<typename DATATYPE, typename IDXTYPE>

virtual bool FST::Data_Accessor< DATATYPE, IDXTYPE >::getNextSplit ( )

pure virtual

data access iteration (to support, e.g., loops in cross-validation)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNoOfBlocks()

template<typename DATATYPE, typename IDXTYPE>

virtual IDXTYPE FST::Data_Accessor< DATATYPE, IDXTYPE >::getNoOfBlocks ( const DataPart ofwhat ) const

pure virtual

returns the number of consecutive blocks the requested DataPart is made of in the current split

Warning: getNoOfBlocks is not to be relied upon due to possible limitations in some implementations

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNoOfClasses()

template<typename DATATYPE, typename IDXTYPE>

virtual unsigned int FST::Data_Accessor< DATATYPE, IDXTYPE >::getNoOfClasses ( ) const

pure virtual

returns number of classes

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNoOfFeatures()

template<typename DATATYPE, typename IDXTYPE>

virtual unsigned int FST::Data_Accessor< DATATYPE, IDXTYPE >::getNoOfFeatures ( ) const

pure virtual

returns data dimensionality

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNoOfPatterns()

template<typename DATATYPE, typename IDXTYPE>

virtual IDXTYPE FST::Data_Accessor< DATATYPE, IDXTYPE >::getNoOfPatterns ( const DataPart ofwhat ) const

pure virtual

returns number of patterns in all consecutive blocks of data of requested DataPart type in the current split (access iteration)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getNoOfSplits()

template<typename DATATYPE, typename IDXTYPE>

virtual unsigned int FST::Data_Accessor< DATATYPE, IDXTYPE >::getNoOfSplits ( ) const

pure virtual

data access iteration (to support, e.g., loops in cross-validation)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ getSplitIndex()

template<typename DATATYPE, typename IDXTYPE>

virtual unsigned int FST::Data_Accessor< DATATYPE, IDXTYPE >::getSplitIndex ( ) const

pure virtual

data access iteration (to support, e.g., loops in cross-validation)

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ resubstitute()

template<typename DATATYPE, typename IDXTYPE>

virtual void FST::Data_Accessor< DATATYPE, IDXTYPE >::resubstitute ( )

pure virtual

resets standard DataPart types' meaning, i.e. undoes every substitute() call

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ setClass()

template<typename DATATYPE, typename IDXTYPE>

virtual void FST::Data_Accessor< DATATYPE, IDXTYPE >::setClass ( const int c )

pure virtual

sets active class from which get*Block() will fetch data

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

◆ sharing_clone()

template<typename DATATYPE, typename IDXTYPE>

Data_Accessor * FST::Data_Accessor< DATATYPE, IDXTYPE >::sharing_clone ( ) const

overridepure virtual

create equivalent clone of the current object, permitting read-only access to structures in the source object (allows referencing instead of copying of large memory structures). may be faster and save space but requires more caution with respect to concurrency Use example: Data_Accessor memory data representation cloning

Implements FST::Clonable.

◆ stateless_clone()

template<typename DATATYPE, typename IDXTYPE>

Data_Accessor * FST::Data_Accessor< DATATYPE, IDXTYPE >::stateless_clone ( ) const

overridepure virtual

create clone of the current object, ignoring internal temporary structures to save speed. Does not replicate exact object state. The clone must be used carefully in a way that ensures internal structure re-initialization Use example: Data_Splitter cloning or Classifier_LIBSVM cloning due to inability to clone external structures defined in LibSVM

Implements FST::Clonable.

◆ substitute()

template<typename DATATYPE, typename IDXTYPE>

virtual void FST::Data_Accessor< DATATYPE, IDXTYPE >::substitute	(	const DataPart	source,
		const DataPart	target )

pure virtual

enables change of meaning of DataPart types, for use in specialized data access scenarios like in bias predicting wrappers Installs a remapping so that every later request for source is served with target's data instead. The mapping is a property of the accessor, applies to all classes and splits, and stays in force until resubstitute() – so a component that substitutes must restore the mapping before handing the accessor on. Mappings do not chain: substituting TRAIN->TEST and then TEST->ALL makes TRAIN yield TEST's data, not ALL's.

Parameters

source	the DataPart the caller will keep asking for
target	the DataPart actually delivered

Implemented in FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >.

Member Data Documentation

◆ _block_slot_used

template<typename DATATYPE, typename IDXTYPE>

std::vector<bool> FST::Data_Accessor< DATATYPE, IDXTYPE >::_block_slot_used

private

one flag per acquired slot (index 0 = loopdepth _first_acquired_loopdepth); grows as concurrent Block_Range instances need more slots

◆ _first_acquired_loopdepth

template<typename DATATYPE, typename IDXTYPE>

const unsigned int FST::Data_Accessor< DATATYPE, IDXTYPE >::_first_acquired_loopdepth = 4

staticprivate

loopdepth slot pool for Block_Range; depths 0.._first_acquired_loopdepth-1 stay reserved for the deprecated hard-coded cursor loops. The pool is plain, unsynchronized bookkeeping: it serves overlapping ranges within ONE accessor instance, not concurrent threads, which use their own sharing_clone() and therefore their own pool.

The documentation for this class was generated from the following file:

/Users/psms/dev/devtest1/_src_dataio/data_accessor.hpp

Classes

Public Types

Public Member Functions

Private Member Functions

Private Attributes

Static Private Attributes

Additional Inherited Members

Detailed Description

Member Function Documentation

◆ clone()

◆ getBlockIndex()

◆ getClass()

◆ getClassSize()

◆ getClassSizeSum()

◆ getFirstBlock()

◆ getFirstSplit()

◆ getNextBlock()

◆ getNextSplit()

◆ getNoOfBlocks()

◆ getNoOfClasses()

◆ getNoOfFeatures()

◆ getNoOfPatterns()

◆ getNoOfSplits()

◆ getSplitIndex()

◆ resubstitute()

◆ setClass()

◆ sharing_clone()

◆ stateless_clone()

◆ substitute()

Member Data Documentation

◆ _block_slot_used

◆ _first_acquired_loopdepth