FST::Data_File_CSV Class Reference

CSV (delimited text) data format filter — the simplest spreadsheet-to-FST4 path. More...

#include <data_file_CSV.hpp>

Public Member Functions
	Data_File_CSV (std::string fileName, const Data_File_CSV_Options &_opts)
	Constructs an (unloaded) reader for fileName under options _opts; call prepareFile() to actually read it.
bool	prepareFile ()
	Reads and parses the whole CSV/TSV file: BOM-strip + blank-line skip, dialect/class-column/header resolution off line 1 (resolveDialect()/resolveClassAndHeader()), then every data row (splitLine(), missing-value handling, parseNumeric()).
void	sortRecordsByClass ()
	Regroups rows/classIds into class-id order (stable within each class), mirroring Data_File_ARFF::sortRecordsByClass().
unsigned int	getNoOfClasses ()
	number of classes (classNames's size); throws fst_error if not loaded
unsigned int	getNoOfFeatures ()
	data dimensionality (columns minus the class column); throws fst_error if not loaded
unsigned int	getTotalSize ()
	number of loaded records (rows's size); throws fst_error if not loaded
unsigned int	getClassSize (int clsId)
	number of records in class clsId; throws fst_error if not loaded or clsId is out of range (rather than returning -1 wrapped to a huge unsigned value; in-tree callers iterate over [0,getNoOfClasses()) so never trigger this)
std::string	getClassName (int clsId)
float	getFeatureValue (long record, int feature)
	the (already-parsed) value of feature feature in record record

Private Member Functions
bool	readLine (FILE *f, std::vector< char > &buff)
std::vector< std::string >	splitLine (const char *line) const
bool	isMissing (const std::string &tok) const
	Whether tok (case-insensitively) is one of the recognized missing-value markers: empty, "?", "NA", "N/A", "NaN", "NULL".
bool	parseNumeric (const std::string &tok, float &out) const
void	resolveDialect (const char *firstLine)
	Resolves delim from opts.delimiter, or auto-detects it from firstLine (candidates counted outside double-quoted spans).
void	resolveClassAndHeader (const std::vector< std::string > &toks)
	Resolves columns, classColumn and hasHeader from the first line's split tokens toks (also headerNames when a header is present). Class column: opts.class_column empty = last column, a parseable integer = that 0-based index, otherwise = a header column name (looked up case-insensitively once the header is confirmed present). Header presence: forced by opts.header ("yes"/"no") or by naming the class column by name; otherwise auto-detected as present iff any non-class field on line 1 fails a numeric-or-missing parse.

Static Private Member Functions
static std::string	trimTok (const std::string &s)
	Strips leading/trailing spaces, tabs and CRs from s (a single already-split field token).

Private Attributes
std::string	_fileName
	input .csv/.tsv file name, as passed to the constructor
Data_File_CSV_Options	opts
	dialect/class-column/missing-value options, as passed to the constructor
char	delim
	resolved delimiter character (comma/semicolon/tab), set by resolveDialect()
bool	hasHeader
	resolved header presence, set by resolveClassAndHeader()
int	classColumn
	resolved 0-based file column of the class label, set by resolveClassAndHeader()
unsigned int	columns
	total file columns (features + class), set by resolveClassAndHeader()
std::vector< std::string >	headerNames
	the header row's tokens, empty when hasHeader is false
std::vector< std::vector< float > >	rows
	loaded records; each row holds feature values only, in non-class file-column order
std::vector< int >	classIds
	each record's class id, parallel to rows
std::vector< std::string >	classNames
	class label text, indexed by class id (assigned in first-appearance order)
std::vector< unsigned int >	classSizes
	per-class record counts, indexed by class id, parallel to classNames
unsigned int	skippedRows
	number of data rows skipped due to a missing value (see the class description's missing-value paragraph)
unsigned int	mappedCells
	number of missing feature values replaced by Data_File_CSV_Options::missing_to
unsigned int	mappedRows
	data rows carrying at least one replaced value
bool	loaded
	whether prepareFile() completed without error; getters that need loaded data throw fst_error while false

Detailed Description

CSV (delimited text) data format filter — the simplest spreadsheet-to-FST4 path.

Dialect handling: delimiter auto-detected on the first line (comma/semicolon/tab, counted outside double quotes); decimal COMMAS accepted whenever the delimiter is not the comma, so a European-locale spreadsheet export loads without any options; a UTF-8 byte-order mark at the start of the file is stripped; double-quote quoting with "" as the escaped quote; header row auto-detected (any non-numeric non-class field in line 1 means a header). Class labels are arbitrary tokens mapped to ids in first-appearance order.

The file has no comment syntax: every line that is not blank (or pure whitespace) is either the header or a data row, and must have exactly the same number of columns as the first line.

Missing values. A field reading "", ?, NA, N/A, NaN or NULL (case-insensitively) is missing. By default the whole row is skipped, with a message naming the row and column, and the load summary reports how many rows went that way. With Data_File_CSV_Options::missing_to_active a missing FEATURE value is instead replaced by a numeric code and the row is kept; a missing CLASS label still skips the row. A malformed row – wrong column count, or a feature token that is neither a number nor a missing marker – is fatal and names the line number.

Member Function Documentation

getClassName()

std::string FST::Data_File_CSV::getClassName ( int clsId )

inline

class clsId's label text, or "-n/a-" if clsId is out of range; throws fst_error if not loaded

getFeatureValue()

float FST::Data_File_CSV::getFeatureValue ( long record, int feature )

inline

the (already-parsed) value of feature feature in record record

Warning

Unchecked: record must be in [0,getTotalSize()) and feature in [0,getNoOfFeatures()), and the file must be loaded. Out-of-range arguments are undefined behaviour.

parseNumeric()

bool FST::Data_File_CSV::parseNumeric ( const std::string & tok, float & out ) const

inlineprivate

strict full-token numeric parse: the whole token must be consumed (trailing spaces and tabs aside), so "1.5x" or "1 2" fail. Commas are read as the decimal point, but only when the delimiter is not the comma AND the token contains no '.' at all – a token that already has a decimal point keeps its commas, which then make the parse fail rather than silently reinterpreting a thousands separator.

prepareFile()

bool FST::Data_File_CSV::prepareFile ( )

inline

Reads and parses the whole CSV/TSV file: BOM-strip + blank-line skip, dialect/class-column/header resolution off line 1 (resolveDialect()/resolveClassAndHeader()), then every data row (splitLine(), missing-value handling, parseNumeric()).

Returns

true (a malformed row, wrong column count, or unreadable file all throw fst_error instead); sets loaded.

readLine()

bool FST::Data_File_CSV::readLine ( FILE * f, std::vector< char > & buff )

inlineprivate

reads one full line of arbitrary length, growing the buffer as needed (the ARFF readLine pattern — no line-length cap); false at EOF with nothing read

resolveClassAndHeader()

void FST::Data_File_CSV::resolveClassAndHeader ( const std::vector< std::string > & toks )

inlineprivate

Resolves columns, classColumn and hasHeader from the first line's split tokens toks (also headerNames when a header is present). Class column: opts.class_column empty = last column, a parseable integer = that 0-based index, otherwise = a header column name (looked up case-insensitively once the header is confirmed present). Header presence: forced by opts.header ("yes"/"no") or by naming the class column by name; otherwise auto-detected as present iff any non-class field on line 1 fails a numeric-or-missing parse.

Exceptions

fst_error

if toks has fewer than 2 columns, a numeric class-column index is out of range, or a by-name class column isn't found in the header.

resolveDialect()

void FST::Data_File_CSV::resolveDialect ( const char * firstLine )

inlineprivate

Resolves delim from opts.delimiter, or auto-detects it from firstLine (candidates counted outside double-quoted spans).

Note

Auto tie-break is deliberately comma-LAST (tab > semicolon > comma): in a European-locale export every data line carries about as many decimal commas as delimiters, and those inflate only the comma count, never the tab or semicolon count – so a candidate that merely ties the comma count should win.

sortRecordsByClass()

void FST::Data_File_CSV::sortRecordsByClass ( )

inline

Regroups rows/classIds into class-id order (stable within each class), mirroring Data_File_ARFF::sortRecordsByClass().

Exceptions

fst_error

if the file was not loaded or no classes were identified.

splitLine()

std::vector< std::string > FST::Data_File_CSV::splitLine ( const char * line ) const

inlineprivate

quote-aware field split: the delimiter inside "..." does not split, "" inside quotes is a literal quote (the Excel convention)

---

The documentation for this class was generated from the following file:

• /Users/psms/dev/devtest1/_src_dataio/data_file_CSV.hpp