FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR > Class Template Reference

Wrapper Bias Evaluator - returns bias estimate (difference of classifier accuracy on training and validation data parts). More...

#include <criterion_wrapper_bias_estimate.hpp>

Inheritance diagram for FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >:

Public Types
typedef Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >	parent
	base class: plain accuracy wrapper criterion
typedef Clonable_Impl< Criterion_Wrapper_Bias_Estimate, parent >	parent_impl
	implements clone(); sharing/stateless_clone throw in Criterion
typedef std::shared_ptr< CLASSIFIER >	PClassifier
	shared pointer to the wrapped classifier
typedef std::shared_ptr< DATAACCESSOR >	PDataAccessor
	shared pointer to the data accessor supplying training/test splits
typedef std::shared_ptr< SUBSET >	PSubset
	shared pointer to the feature subset being evaluated
Public Types inherited from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >
typedef Clonable_Impl< Criterion_Wrapper, Criterion< RETURNTYPE, SUBSET > >	parent_impl
	implements clone(); sharing/stateless_clone throw in Criterion
typedef std::shared_ptr< CLASSIFIER >	PClassifier
	shared pointer to the wrapped classifier
typedef std::shared_ptr< DATAACCESSOR >	PDataAccessor
	shared pointer to the data accessor supplying training/test splits
typedef std::shared_ptr< SUBSET >	PSubset
	shared pointer to the feature subset being evaluated
Public Types inherited from FST::Criterion< RETURNTYPE, SUBSET >
typedef std::shared_ptr< SUBSET >	PSubset
	shared pointer to the feature subset being evaluated

Public Member Functions
virtual bool	evaluate (RETURNTYPE &result, const PSubset sub) override
	Trains on the TRAIN part of each split, tests on both parts, and returns the mean per-split bias: accuracy on the (dependent) train part minus accuracy on the test part. Higher values indicate more overfitting: unlike the other criteria this one is a diagnostic whose LOW values are the good ones, so a maximizing search must not be pointed at it directly (Criterion_Negative flips it if needed). Side effect: the per-split biases are also accumulated for getBiasEstimate(). Returns false when training or testing fails on any split.
RETURNTYPE	getBiasEstimate () const
	returns the mean per-split bias over all splits evaluated since the last reset (each evaluate() call contributes one value per split), or 0 when nothing has been accumulated
void	resetBiasEstimate ()
	clears the accumulated bias estimate and the split count
virtual std::ostream &	print (std::ostream &os) const override
	writes a human-readable identification of the object; base default writes nothing
Public Member Functions inherited from FST::Clonable_Impl< Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >, Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR > >
Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR > *	clone () const override
	dispatches to the DEEP_KIND mechanism (default CLONE_DEEP: Derived's copy-constructor)
Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR > *	sharing_clone () const override
	dispatches to the SHARING_KIND mechanism (default CLONE_INHERIT: defers to Base, which throws by family convention)
Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR > *	stateless_clone () const override
	dispatches to the STATELESS_KIND mechanism (default CLONE_INHERIT: defers to Base, which throws by family convention)
Public Member Functions inherited from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >
virtual void	initialize (const PClassifier classifier, const PDataAccessor da)
	Stores the classifier and the data accessor to be used by every later evaluate(). Must be called before the first evaluate(). Both are kept by shared_ptr for the whole life cycle of *this; the accessor must already be initialized and must carry the split setting the evaluation is to run over. Clones take a clone of the classifier and a sharing clone of the accessor, so a threaded search never shares either instance.
void	set_metric (const Metric_Spec< RETURNTYPE > &spec)
const Metric_Spec< RETURNTYPE > &	get_metric () const
	returns the currently configured wrapper metric (see set_metric())
Public Member Functions inherited from FST::Criterion< RETURNTYPE, SUBSET >
virtual bool	is_monotonic () const
Criterion *	sharing_clone () const override
Criterion *	stateless_clone () const override
Public Member Functions inherited from FST::Clonable
bool	is_clone () const
bool	is_sharing () const
	true if this instance is a sharing_clone() result (referencing, not owning, the source's structures)
bool	is_sharing_clone () const
	true if this instance is BOTH a clone and sharing – i.e. specifically a sharing_clone() result, not a clone()/stateless_clone() one

Protected Member Functions
	Criterion_Wrapper_Bias_Estimate (const Criterion_Wrapper_Bias_Estimate &cwbe)
	Copy-constructor; delegates to parent_impl (clones _classifier, sharing-clones _da) and copies the collected bias accumulators.
Protected Member Functions inherited from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >
	Criterion_Wrapper (const Criterion_Wrapper &cw)
	Copy-constructor; clones _classifier and sharing-clones _da, copies _metric_spec.
Protected Member Functions inherited from FST::Clonable
void	set_cloned (const bool cloned=true)
	marks/unmarks this instance as a clone; call from clone()/stateless_clone() overrides (Clonable_Impl does this automatically)
void	set_sharing (const bool sharing=true)
	marks/unmarks this instance as sharing (see is_sharing())
void	set_sharing_cloned (const bool cloned=true, const bool sharing=true)
	sets both flags at once; call from sharing_clone() overrides (Clonable_Impl does this automatically)

Protected Attributes
friend	parent_impl
RETURNTYPE	_collected_bias_estimate
	running sum of per-split bias (train-part minus test-part accuracy) since the last resetBiasEstimate()
RETURNTYPE	_collected_bias_count
	number of SPLITS accumulated into _collected_bias_estimate since the last reset (one evaluate() call adds as many as the accessor has splits)
Protected Attributes inherited from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >
friend	parent_impl
std::shared_ptr< CLASSIFIER >	_classifier
	the wrapped classifier trained then tested by evaluate()
std::shared_ptr< DATAACCESSOR >	_da
	data accessor supplying the train/test splits
Metric_Spec< RETURNTYPE >	_metric_spec
	configured wrapper metric; default = plain accuracy (see set_metric())

Detailed Description

template<class RETURNTYPE, class SUBSET, class CLASSIFIER, class DATAACCESSOR>
class FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >

Wrapper Bias Evaluator - returns bias estimate (difference of classifier accuracy on training and validation data parts).

Wraps Classifier objects to serve as secondary feature selection criterion. Trains classifier on training data, then evaluates the difference between estimated classification accuracy on training data and test data. Especially in small-sample scenarios a considerable difference (i.e., bias) can be expected. Large bias can be viewed as a sign of overfitting and can thus be used as indicator of feature selection process failure.

Note

Expect at least 2x slower operation than that of Criterion_Wrapper.

Member Function Documentation

evaluate()

template<class RETURNTYPE, class SUBSET, class CLASSIFIER, class DATAACCESSOR>

bool FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >::evaluate ( RETURNTYPE & result, const PSubset sub )

overridevirtual

Trains on the TRAIN part of each split, tests on both parts, and returns the mean per-split bias: accuracy on the (dependent) train part minus accuracy on the test part. Higher values indicate more overfitting: unlike the other criteria this one is a diagnostic whose LOW values are the good ones, so a maximizing search must not be pointed at it directly (Criterion_Negative flips it if needed). Side effect: the per-split biases are also accumulated for getBiasEstimate(). Returns false when training or testing fails on any split.

Note

The inherited Criterion_Wrapper::initialize() must be called before the first evaluate().

set_metric() is inherited but has NO effect here: the bias is always computed from plain classification accuracy, never from the configured wrapper metric.

Reimplemented from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >.

print()

template<class RETURNTYPE, class SUBSET, class CLASSIFIER, class DATAACCESSOR>

virtual std::ostream & FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >::print ( std::ostream & os ) const

inlineoverridevirtual

writes a human-readable identification of the object; base default writes nothing

Reimplemented from FST::Criterion_Wrapper< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >.

Member Data Documentation

parent_impl

template<class RETURNTYPE, class SUBSET, class CLASSIFIER, class DATAACCESSOR>

friend FST::Criterion_Wrapper_Bias_Estimate< RETURNTYPE, SUBSET, CLASSIFIER, DATAACCESSOR >::parent_impl

protected

grants Clonable_Impl access to the copy-constructor

---

The documentation for this class was generated from the following file:

• /Users/psms/dev/devtest1/_src_criteria/criterion_wrapper_bias_estimate.hpp