partly-abstract class, defines support for data splitting More...
#include <data_accessor_splitting.hpp>
Classes | |
| class | DataSplit |
| Data splitting support structure; holds one set of intervals (train, test) per each splitting depth and (data)class. More... | |
Public Types | |
| typedef std::shared_ptr< Data_Splitter< INTERVALCONTAINER, IDXTYPE > > | PSplitter |
| shared pointer to one Data_Splitter (one nested splitting level) | |
| typedef std::shared_ptr< std::vector< PSplitter > > | PSplitters |
| shared pointer to the ordered list of splitters, one per nested splitting depth (depth 0 = outermost) | |
| typedef const DATATYPE * | PPattern |
| Public Types inherited from FST::Data_Accessor< DATATYPE, IDXTYPE > | |
| typedef const DATATYPE * | PPattern |
| one data sample: a raw pointer to its (read-only) feature vector, getNoOfFeatures() entries long | |
Public Member Functions | |
| Data_Accessor_Splitting (const PSplitters _dsp) | |
Constructs an (uninitialized) accessor bound to the splitter stack _dsp; concrete subclasses must still call initialize() once dimensionality/class sizes are known. | |
| virtual unsigned int | getNoOfClasses () const override |
| returns number of classes | |
| virtual unsigned int | getNoOfFeatures () const override |
| returns data dimensionality | |
| virtual IDXTYPE | getClassSize (const unsigned int c) const override |
| returns size (number of samples in) of class c | |
| virtual IDXTYPE | getClassSizeSum () const override |
| returns summed size (number of samples in) of all classes, i.e., no. of all patterns in data | |
| virtual void | setClass (const int c) override |
| sets active class -> from now on only data from class c will be considered | |
| virtual int | getClass () const override |
| returns active class | |
| void | setSplittingDepth (const unsigned int depth) |
| Selects the nested splitting level (0 = outermost/depth-0 splitter, 1 = the next-nested level, etc.) that subsequent getFirstSplit()/getNextSplit()/getFirstBlock()/getNextBlock() calls operate on. | |
| unsigned int | getSplittingDepth () const |
| the nested splitting level last set via setSplittingDepth() | |
| virtual unsigned int | getNoOfSplits () const override |
| data access iteration (to support, e.g., loops in cross-validation) | |
| virtual bool | getFirstSplit () override |
| data access iteration (to support, e.g., loops in cross-validation) | |
| virtual bool | getNextSplit () override |
| data access iteration (to support, e.g., loops in cross-validation) | |
| virtual unsigned int | getSplitIndex () const override |
| data access iteration (to support, e.g., loops in cross-validation) | |
| virtual IDXTYPE | getNoOfBlocks (const DataPart ofwhat) const override |
| virtual bool | getFirstBlock (const DataPart ofwhat, PPattern &firstpattern, IDXTYPE &patterns, const unsigned int loopdepth=0) override=0 |
| returns pointer to first consecutive block of data of requested DataPart type in the current split (access iteration) | |
| virtual bool | getNextBlock (const DataPart ofwhat, PPattern &firstpattern, IDXTYPE &patterns, const unsigned int loopdepth=0) override=0 |
| returns pointer to next consecutive block of data of requested DataPart type in the current split (access iteration) | |
| virtual IDXTYPE | getBlockIndex (const unsigned int loopdepth=0) const override |
| returns index of the current consecutive block of data of requested DataPart type in the current split (access iteration) | |
| virtual IDXTYPE | getNoOfPatterns (const DataPart ofwhat) const override |
| returns number of patterns in all consecutive blocks of data of requested DataPart type in the current split (access iteration) | |
| virtual void | substitute (const DataPart source, const DataPart target) override |
| enables change of meaning of DataPart types, for use in specialized data access scenarios like in bias predicting wrappers | |
| virtual void | resubstitute () override |
| resets standard DataPart types' meaning | |
| virtual std::ostream & | print (std::ostream &os) const override |
writes the splitter stack's print() output (or "Data_Accessor_Splitting()" if the stack is empty) to os | |
| Public Member Functions inherited from FST::Data_Accessor< DATATYPE, IDXTYPE > | |
| Block_Range | blocks (const DataPart ofwhat) |
| RAII block range over the current split - prefer over the deprecated getFirstBlock()/getNextBlock() cursor. | |
| Data_Accessor * | clone () const override=0 |
| Data_Accessor * | sharing_clone () const override=0 |
| Data_Accessor * | stateless_clone () const override=0 |
| Public Member Functions inherited from FST::Clonable | |
| bool | is_clone () const |
| bool | is_sharing () const |
| true if this instance is a sharing_clone() result (referencing, not owning, the source's structures) | |
| bool | is_sharing_clone () const |
| true if this instance is BOTH a clone and sharing – i.e. specifically a sharing_clone() result, not a clone()/stateless_clone() one | |
Protected Types | |
| typedef std::vector< unsigned int > | CLASSSIZES |
| per-class pattern counts, indexed by class index; the format initialize() takes | |
| typedef const Data_Interval< IDXTYPE > * | DATAINTERVAL |
| typedef std::shared_ptr< INTERVALCONTAINER > | PIntervaller |
Protected Member Functions | |
| Data_Accessor_Splitting (const Data_Accessor_Splitting &da) | |
| "Strong" (1:1, deep) copy constructor: stateless_clone()s every splitter and re-assign()s it to this instance's own (deep-copied via splits) interval lists. | |
| void | initialize (const unsigned int _features, const CLASSSIZES &_classes) |
| DataPart | mappedDataPart (const DataPart ofwhat) const |
looks up ofwhat's current remapping target (mappedTRAIN/mappedTEST/mappedTRAINTEST/mappedALL); NONE maps to itself | |
| virtual bool | getFirstBlock (const DataPart ofwhat, DATAINTERVAL &tmp, const unsigned int loopdepth=0) |
| returns Data_Interval record representing the first consecutive block of data of requested DataPart type in the current split (access iteration) | |
| virtual bool | getNextBlock (const DataPart ofwhat, DATAINTERVAL &tmp, const unsigned int loopdepth=0) |
| returns Data_Interval record representing the next consecutive block of data of requested DataPart type in the current split (access iteration) | |
| bool | is_initialized () const |
| void | assert_splits (const int splitting_check=-1) const |
| DEBUG-only: asserts the internal structures (dsp, splits, enum_split, enum_block, tt_phase, classes) are non-empty, consistently sized, and every splits entry has its train/test/reduced interval lists allocated; a no-op in release builds. | |
| Protected Member Functions inherited from FST::Clonable | |
| void | set_cloned (const bool cloned=true) |
| marks/unmarks this instance as a clone; call from clone()/stateless_clone() overrides (Clonable_Impl does this automatically) | |
| void | set_sharing (const bool sharing=true) |
| marks/unmarks this instance as sharing (see is_sharing()) | |
| void | set_sharing_cloned (const bool cloned=true, const bool sharing=true) |
| sets both flags at once; call from sharing_clone() overrides (Clonable_Impl does this automatically) | |
Protected Attributes | |
| CLASSSIZES | classes |
| pattern count per class, as passed to initialize() | |
| unsigned int | features |
| data dimensionality, as passed to initialize() | |
| DataPart | mappedTRAIN |
| current remapping target of TRAIN (see substitute()/resubstitute()); defaults to TRAIN | |
| DataPart | mappedTEST |
| current remapping target of TEST (see substitute()/resubstitute()); defaults to TEST | |
| DataPart | mappedTRAINTEST |
| current remapping target of TRAINTEST (see substitute()/resubstitute()); defaults to TRAINTEST | |
| DataPart | mappedALL |
| current remapping target of ALL (see substitute()/resubstitute()); defaults to ALL | |
| PSplitters | dsp |
| the splitter stack this accessor was constructed with, one Data_Splitter per nested splitting depth | |
| std::vector< std::vector< DataSplit > > | splits |
| one set of splitters per each splitting depth and class | |
| std::vector< IDXTYPE > | enum_split |
| current split.. 0~none | |
| std::vector< std::vector< IDXTYPE > > | enum_block |
| current block loop.. 0~none | |
| std::vector< std::vector< DataPart > > | tt_phase |
| in current block loop.. for DataPart==TRAINTEST indicates which half is being delivered: TRAIN, then TEST once the train blocks run out, NONE when the loop is finished | |
| unsigned int | splitting_depth |
| the nested splitting level the get*Split()/get*Block() calls currently refer to; set via setSplittingDepth() | |
| int | active_class |
| denotes from which class the get*Train* get*Validate* get*Test* methods return patterns, to be set using setClass() | |
| bool | _initialize_called |
| set true by initialize(); read by is_initialized() | |
partly-abstract class, defines support for data splitting
Data structures in Data_Accessor_Splitting are directly used in the splitting mechanism that enables structured access to data. In order to keep the implementation of data splitters (specializations of Data_Splitter) as simple as possible for the user, we moved most of the technicalities here.
Splitting depth. The accessor is constructed with an ordered list of splitters, one per nested splitting level; the length of that list is the splitting depth of the data access. Depth 0 is the outermost level and splits the whole data set; depth 1 splits the TRAIN part of the current depth-0 split, depth 2 the TRAIN part of the current depth-1 split, and so on. Only one level is active at a time: setSplittingDepth() selects it, and getFirstSplit()/getNextSplit() and the block-access methods then all refer to that level. A two-level stack is the usual arrangement for selection experiments – the criterion works inside the depth-1 splits while the depth-0 test part is held back as an independent validation set.
Correct state of key data structures in Data_Accessor_Splitting is as follows:
The splitting mechanism needs not just one train-test data structure pair, but two - the second pair denoted _reduced_train and _reduced_test. The "reduced" pair is constructed from the "base" pair separately in each splitting level, so as to correctly represent the subset of data defined by the respective splitter (each deeper level further reduces access to data visible in the preceeding level). The actual access to data through getFirstBlock() and getNextBlock() is commanded by data intervals stored in the "reduced" pair of interval lists only.
The two pairs of interval lists exist separately for each splitting level and each data class. In Data_Accessor_Splitting they are collected in the DataSplit subclass, of which the required number of instances is kept in the "splits" container. In correct representation "splits" must contain [number of classes]*[number of splitting levels] DataSplit instances.
The DataSplit that represent top splitting level differ from the deeper level - the "reduced" pair of lists is actually just referencing the "base" pair. This is because data indexes as produced by splitters are valid indexes usable to access the data. In deeper splitting levels this is not so because splitter indexes must be treated are relative to the data possibly restricted in higher level. Transforming the relative indexes to absolute indexes is achieved through the "reduce" method implemented in Data_Intervaller. In non-top splitting levels before data can be accessed, the "base" indexes/intervals are first transformed using the "reduce" method with the result stored in "_reduced" train and test lists.
The "base" train and test lists allocated here in Data_Accessor_Splitting need to be interlinked with respective data splitters. The splitters do not hold any allocated structures, they re-direct their output to the "base" train and test lists kept in Data_Accessor_Splitting, to enable data accessing routines to transform the indexes by means of "reduce" whenever needed and subsequently to access the correct subset of data.
|
protected |
pointer to a read-only Data_Interval record (as yielded by the protected Data_Interval-returning getFirstBlock()/getNextBlock() overloads)
|
protected |
shared pointer to one class's list of train/test data intervals at one splitting depth
| typedef const DATATYPE* FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >::PPattern |
one data sample: a raw pointer to its (read-only) feature vector
|
protected |
"Strong" (1:1, deep) copy constructor: stateless_clone()s every splitter and re-assign()s it to this instance's own (deep-copied via splits) interval lists.
|
protected |
DEBUG-only: asserts the internal structures (dsp, splits, enum_split, enum_block, tt_phase, classes) are non-empty, consistently sized, and every splits entry has its train/test/reduced interval lists allocated; a no-op in release builds.
| splitting_check | -1 (default) skips the per-level sample-count check; a value >=0 additionally asserts every splitting level 0..splitting_check has a positive sample count set (i.e. set_n() was already called there) |
|
overridevirtual |
returns index of the current consecutive block of data of requested DataPart type in the current split (access iteration)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
inlineoverridevirtual |
returns active class
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
returns size (number of samples in) of class c
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
returns summed size (number of samples in) of all classes, i.e., no. of all patterns in data
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridepure virtual |
returns pointer to first consecutive block of data of requested DataPart type in the current split (access iteration)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
Implemented in FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >.
|
overridevirtual |
data access iteration (to support, e.g., loops in cross-validation)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
Reimplemented in FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.
|
overridepure virtual |
returns pointer to next consecutive block of data of requested DataPart type in the current split (access iteration)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
Implemented in FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >.
|
overridevirtual |
data access iteration (to support, e.g., loops in cross-validation)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
Reimplemented in FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.
|
overridevirtual |
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
returns number of classes
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
returns data dimensionality
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
returns number of patterns in all consecutive blocks of data of requested DataPart type in the current split (access iteration)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
overridevirtual |
data access iteration (to support, e.g., loops in cross-validation)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
Reimplemented in FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.
|
overridevirtual |
data access iteration (to support, e.g., loops in cross-validation)
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
protected |
sets-up memory structures needed in the splitting mechanism data access is not needed here, the structures work with indexes only - the only information needed is dimensionality and sizes of data classes
|
inlineprotected |
whether initialize() has been called on this instance
|
overridevirtual |
writes the splitter stack's print() output (or "Data_Accessor_Splitting()" if the stack is empty) to os
Reimplemented from FST::Clonable.
Reimplemented in FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_Mem< DATATYPE, IDXTYPE, INTERVALLER >, FST::Data_Accessor_Splitting_MemARFF< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_MemCSV< DATATYPE, IDXTYPE, INTERVALCONTAINER >, FST::Data_Accessor_Splitting_MemTRN< DATATYPE, IDXTYPE, INTERVALCONTAINER >, and FST::Data_Accessor_Universe< DATATYPE, IDXTYPE, INTERVALCONTAINER >.
|
overridevirtual |
resets standard DataPart types' meaning
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
|
inlineoverridevirtual |
sets active class -> from now on only data from class c will be considered
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.
| void FST::Data_Accessor_Splitting< DATATYPE, IDXTYPE, INTERVALCONTAINER >::setSplittingDepth | ( | const unsigned int | depth | ) |
Selects the nested splitting level (0 = outermost/depth-0 splitter, 1 = the next-nested level, etc.) that subsequent getFirstSplit()/getNextSplit()/getFirstBlock()/getNextBlock() calls operate on.
depth is within the splitter stack and that every shallower level already has an active split (i.e. getFirstSplit()/getNextSplit() was already called there).
|
overridevirtual |
enables change of meaning of DataPart types, for use in specialized data access scenarios like in bias predicting wrappers
Implements FST::Data_Accessor< DATATYPE, IDXTYPE >.