Bridge classifier delegating training and prediction to a user-supplied external command. More...

#include <classifier_extern.hpp>

Inheritance diagram for FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >:

Public Types
typedef Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >	parent
	the immediate abstract base
typedef Clonable_Impl< Classifier_Extern, parent >	parent_impl
	implements clone() via the copy-constructor below
typedef std::shared_ptr< DATAACCESSOR >	PDataAccessor
	shared pointer to the data accessor supplying train/test blocks
typedef std::shared_ptr< SUBSET >	PSubset
	shared pointer to the feature subset being trained/evaluated
typedef DATAACCESSOR::PPattern	PPattern
	one data sample as exposed by DATAACCESSOR
Public Types inherited from FST::Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >
typedef std::shared_ptr< DATAACCESSOR >	PDataAccessor
	shared pointer to the data accessor supplying train/test blocks
typedef std::shared_ptr< SUBSET > const	PSubset
	shared pointer to the (fixed, non-reassignable) feature subset being evaluated
typedef DATAACCESSOR::PPattern	PPattern
	one data sample as exposed by DATAACCESSOR (a raw pointer to its feature vector)

Public Member Functions
void	set_command (const std::string &cmd)
	sets the external command line (--extern-cmd); the bridge asserts it is not already running
const std::string &	get_command () const
	returns the configured command line
void	initialize (const PDataAccessor da)
	Records the data shape and allocates the index buffer. Must be called before train(). Throws fst_error when `da` has fewer than 2 classes, or when no command has been set yet (set_command()) – the child is not spawned here, but a bridge without a command could never spawn one.
virtual bool	classify (DIMTYPE &cls, const PPattern &pattern) override
virtual bool	train (const PDataAccessor da, const PSubset sub) override
	Streams the designated training part as one TRAIN block: buffers the subset index, counts the training patterns with a cheap block-size sweep, sends the "TRAIN <n> <d> <classes>" header followed by one labeled row per pattern (send_row()), then requires the single-line response "OK" – anything else throws fst_error (the external tool's own rejection reason is included verbatim). Returns false only when there are zero training patterns; a subset/data shape mismatch is an assertion failure, not a bool return.
virtual bool	test (RETURNTYPE &result, const PDataAccessor da) override
virtual bool	test_confusion (std::vector< RETURNTYPE > &confusion, DIMTYPE &classes, const PDataAccessor da) override
virtual std::ostream &	print (std::ostream &os) const override
	writes a human-readable identification of the object; base default writes nothing
Public Member Functions inherited from FST::Clonable_Impl< Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >, Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR > >
Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR > *	clone () const override
	dispatches to the DEEP_KIND mechanism (default CLONE_DEEP: Derived's copy-constructor)
Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR > *	sharing_clone () const override
	dispatches to the SHARING_KIND mechanism (default CLONE_INHERIT: defers to Base, which throws by family convention)
Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR > *	stateless_clone () const override
	dispatches to the STATELESS_KIND mechanism (default CLONE_INHERIT: defers to Base, which throws by family convention)
Public Member Functions inherited from FST::Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >
virtual bool	train (const PDataAccessor da, const PSubset sub)=0
	learns from designated training part of data
virtual bool	classify_scores (std::vector< RETURNTYPE > &scores, const PPattern &pattern)
virtual bool	test_scores (std::vector< std::pair< RETURNTYPE, DIMTYPE > > &scores, const PDataAccessor da)
virtual bool	test_scores_matrix (std::vector< RETURNTYPE > &scores, std::vector< DIMTYPE > &labels, DIMTYPE &classes, const PDataAccessor da)
virtual bool	test_metric (RETURNTYPE &result, const PDataAccessor da, const Metric_Spec< RETURNTYPE > &spec)
Public Member Functions inherited from FST::Clonable
bool	is_clone () const
bool	is_sharing () const
	true if this instance is a sharing_clone() result (referencing, not owning, the source's structures)
bool	is_sharing_clone () const
	true if this instance is BOTH a clone and sharing – i.e. specifically a sharing_clone() result, not a clone()/stateless_clone() one

Protected Member Functions
void	send_row (const DIMTYPE label_or_none, const bool with_label, const PPattern &pattern)
	Writes one %.17g-formatted data row over the selected subset to the child's stdin. When `with_label` is true, `label_or_none` is written first (space-separated from the feature values); otherwise it is ignored and the row starts directly with features. Throws (via Extern_Bridge::die_on_write()) if the child has stopped accepting input.
long	parse_prediction (const std::string &line)
	Strict full-line numeric parse of one predicted class index: the ENTIRE line must be consumed by strtol(), unlike a bare strtol() call which would silently accept any garbage line (a stray child diagnostic, an "ERROR ..." message) as a valid class-0 prediction. Throws fst_error (and shuts the bridge down) on a non-numeric or partial line, symmetric with train()'s TRAIN-rejection loudness.
bool	run_test_block (std::vector< DIMTYPE > &predictions, const PDataAccessor da)
	Runs one batched TEST round-trip over the designated test part, shared by test()/test_confusion(): sends the "TEST <m> <d>" header and one unlabeled row per pattern (send_row()), then collects `predictions` in the canonical class-major, block-order enumeration. Returns false (without throwing) when there is no test data, or when any returned prediction is out of [0,_classes) – in the latter case the remaining unread response lines are abandoned and the bridge is shut down so the next call respawns a clean child instead of reading a desynced stream.
Protected Member Functions inherited from FST::Clonable
void	set_cloned (const bool cloned=true)
	marks/unmarks this instance as a clone; call from clone()/stateless_clone() overrides (Clonable_Impl does this automatically)
void	set_sharing (const bool sharing=true)
	marks/unmarks this instance as sharing (see is_sharing())
void	set_sharing_cloned (const bool cloned=true, const bool sharing=true)
	sets both flags at once; call from sharing_clone() overrides (Clonable_Impl does this automatically)

Protected Attributes
Extern_Bridge	_bridge
	persistent child process + line I/O (per instance; never shared by clones)
DIMTYPE	_classes
	class count, set by initialize()
DIMTYPE	_features
	raw (full, unnarrowed) feature count, set by initialize()
DIMTYPE	_subfeatures
	d = current subset size
std::vector< DIMTYPE >	_index
	subset position -> raw feature index (first d entries valid)

Private Member Functions
	Classifier_Extern (const Classifier_Extern &ce)
	Copy-constructor backing clone(): copies shape/config (class and feature counts, the narrowing _index, the configured command line) but deliberately gives the clone a FRESH, not-yet-running Extern_Bridge – clones never share pipes with their source, each spawns its own child process lazily on first use (Extern_Bridge::ensure_process()), which is what makes threaded evaluation isolation-safe.

Private Attributes
friend	parent_impl

Detailed Description

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>
class FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >

Bridge classifier delegating training and prediction to a user-supplied external command.

One PERSISTENT child process per classifier instance (spawned lazily on first use; criterion clones spawn their own, so threaded evaluation is isolation-safe), speaking a line protocol on stdin/stdout:

-> TRAIN <N> <d> <K>\n  followed by N lines  <label> <v1> ... <vd>
<- OK\n                 (anything else, e.g. "ERROR ...", fails the evaluation)
-> TEST <M> <d>\n       followed by M lines  <v1> ... <vd>
<- M lines, each the predicted class index (0-based integer)

TRAIN/TEST pairs repeat for every candidate subset; closing stdin (fst4 exit or clone destruction) tells the child to terminate. Values are exchanged as %.17g decimal (bit round-trip for doubles). A reference implementation ships in tests/extern_centroid.awk.

What the external command MUST do:

FLUSH ITS STDOUT after each reply (the "OK" line, and the last prediction line of a TEST block). Both sides are blocking pipes: a child that leaves its reply in a stdio buffer deadlocks against a bridge already waiting to read it. Line-buffered output (or an explicit flush) is the usual fix - awk's print is line-buffered to a pipe on most implementations, but C and Python programs need fflush()/flush=True.
Read until stdin reports EOF and then exit. EOF is the only shutdown signal sent; no signal, no QUIT command.
Write diagnostics to STDERR, never to stdout. Stderr is not read by the bridge at all: it is inherited from fst4 and lands wherever fst4's own stderr goes. Anything unexpected on stdout desynchronizes the protocol and fails the evaluation loudly.
Answer every TRAIN with exactly one line and every TEST with exactly M lines, in the order the rows were sent. The row order is the canonical class-major, block-order enumeration used throughout FST4.

LOUD CAVEATS (by design, documented rather than hidden):

REPRODUCIBILITY GUARANTEES END AT THE BRIDGE. FST4's determinism story (a single controlled PRNG, results independent of the thread count) covers everything up to the protocol; whether equal TRAIN/TEST input yields equal output is entirely the external command's property. A stochastic external trainer makes results run- and thread-count-dependent.
THE COMMAND IS USER-SUPPLIED CODE EXECUTION through the system shell. Config files carrying extern-cmd must be treated like scripts.
No timeout machinery: a hung command hangs the search (kill fst4). In particular shutdown_process() (and thus the destructor) waits for the child to exit; a child that ignores stdin-EOF keeps fst4 waiting at teardown until the child is killed externally.
SIGPIPE handling: on macOS/BSD the pipe fd is marked F_SETNOSIGPIPE; on other POSIX platforms SIGPIPE is ignored PROCESS-WIDE from the first spawn (there is no per-fd control) so that a dying child surfaces as an FST error instead of silently killing fst4. Windows has no SIGPIPE.
classify_scores() is unsupported (the protocol exchanges labels only), so ranking metrics (AUC) are unavailable; the confusion-matrix metrics (balanced accuracy, F1, class weights) work via the batched test_confusion() override.

Member Function Documentation

◆ classify()

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>

bool FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >::classify	(	DIMTYPE &	cls,
		const PPattern &	pattern )

overridevirtual

a single-row TEST round-trip: one header line, one data line, one reply line per call. Correct but slow – prefer test()/test_confusion(), which batch the whole test part into a single round-trip. Returns false (and drops the child, so the next call respawns it) when the reply is not a valid class index; a non-numeric reply throws instead.

Implements FST::Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >.

◆ print()

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>

virtual std::ostream & FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >::print ( std::ostream & os ) const

inlineoverridevirtual

writes a human-readable identification of the object; base default writes nothing

Reimplemented from FST::Clonable.

◆ test()

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>

bool FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >::test	(	RETURNTYPE &	result,
		const PDataAccessor	da )

overridevirtual

accuracy over the designated test part, obtained in ONE batched TEST block. Returns false either when the test part is empty or when the child's answers were unusable (see run_test_block()) – the two cases are not distinguished at this interface.

Reimplemented from FST::Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >.

◆ test_confusion()

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>

bool FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >::test_confusion	(	std::vector< RETURNTYPE > &	confusion,
		DIMTYPE &	classes,
		const PDataAccessor	da )

overridevirtual

the confusion matrix over the designated test part, from the same single batched TEST block as test() (overridden so the metric palette does not fall back to the base class's per-pattern classify() loop, which would cost one round-trip per pattern). Same two false-return cases as test().

Reimplemented from FST::Classifier< RETURNTYPE, DIMTYPE, SUBSET, DATAACCESSOR >.

Member Data Documentation

◆ parent_impl

template<class RETURNTYPE, typename DATATYPE, typename REALTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET, class DATAACCESSOR>

friend FST::Classifier_Extern< RETURNTYPE, DATATYPE, REALTYPE, IDXTYPE, DIMTYPE, SUBSET, DATAACCESSOR >::parent_impl

private

grants Clonable_Impl access to the copy-constructor

The documentation for this class was generated from the following file:

/Users/psms/dev/devtest1/_src_criteria/classifier_extern.hpp

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Detailed Description

Member Function Documentation

◆ classify()

◆ print()

◆ test()

◆ test_confusion()

Member Data Documentation

◆ parent_impl