Feature Selection Toolbox FST4
UTIA · Czech Academy of Sciences
Loading...
Searching...
No Matches
result_tracker_stabileval.hpp
Go to the documentation of this file.
1#ifndef FSTRESULTTRACKERSTABILEVAL_H
2#define FSTRESULTTRACKERSTABILEVAL_H
3 /*
27=========================================================================
28Copyright and license:
29 * FST4 software (with the exception of any bundled or externally linked
30 third-party components) is Copyright (C) 2000-present Institute of
31 Information Theory and Automation of the Czech Academy of Sciences (UTIA).
32 * FST4 is free software: you can redistribute it and/or modify it under
33 the terms of the GNU Affero General Public License as published by the
34 Free Software Foundation, version 3. The full text of the license is in
35 the file LICENSE in the distribution root and at
36 https://www.gnu.org/licenses/agpl-3.0.html.
37 * FST4 is distributed in the hope that it will be useful, but WITHOUT ANY
38 WARRANTY; without even the implied warranty of MERCHANTABILITY or
39 FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public
40 License for more details.
41 * Closed-source use without the AGPL's copyleft obligations requires a
42 commercial license from UTIA; in case of interest write to
43 utia@utia.cas.cz. LICENSING.md in the distribution root explains both
44 options, the contributor license agreement that every contribution
45 requires, and the contacts (technical questions and pull requests:
46 somol@utia.cas.cz, with "FST" in the subject line).
47========================================================================== */
48
49#include <memory>
50#include <cassert>
51#include <iostream>
52#include <list>
53#include <vector>
54#include "error.hpp"
55#include "global.hpp"
56#include "result_tracker.hpp"
57
58/*============== Template parameter type naming conventions ==============
59--------- Numeric types: -------------------------------------------------
60DATATYPE - data sample values - usually real numbers (but may be integers
61 in text processing etc.)
62REALTYPE - must be real numbers - for representing intermediate results of
63 calculations like mean, covariance etc.
64IDXTYPE - index values for enumeration of data samples - (nonnegative) integers,
65 extent depends on numbers of samples in data
66DIMTYPE - index values for enumeration of features (dimensions), or classes (not
67 class sizes) - (nonnegative) integers, usually lower extent than IDXTYPE,
68 but be aware of expressions like _classes*_features*_features !
69 in linearized representations of feature matrices for all classes
70BINTYPE - feature selection marker type - represents ca. <10 different feature
71 states (selected, deselected, sel./desel. temporarily 1st nested loop, 2nd...)
72RETURNTYPE - criterion value: real value, but may be extended in future to support
73 multiple values
74--------- Class types: ---------------------------------------------------
75SUBSET - class of class type Subset
76CLASSIFIER - class implementing interface defined in abstract class Classifier
77EVALUATOR - class implementing interface defined in abstract class Sequential_Step
78DISTANCE - class implementing interface defined in abstract class Distance
79DATAACCESSOR - class implementing interface defined in abstract class Data_Accessor
80INTERVALCONTAINER - class of class type Data_Intervaller
81CONTAINER - STL container of class type Data_Interval
82========================================================================== */
83
84namespace FST {
85
108template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
109class Result_Tracker_Stability_Evaluator : public Clonable_Impl<Result_Tracker_Stability_Evaluator<RETURNTYPE,IDXTYPE,DIMTYPE,SUBSET>,Result_Tracker<RETURNTYPE,SUBSET>,CLONE_INHERIT,CLONE_INHERIT,CLONE_DEEP> {
110public:
111 typedef Result_Tracker<RETURNTYPE,SUBSET> parent;
112 typedef Clonable_Impl<Result_Tracker_Stability_Evaluator,parent,CLONE_INHERIT,CLONE_INHERIT,CLONE_DEEP> parent_impl;
113 typedef typename parent::ResultRec ResultRec;
114 typedef ResultRec* PResultRec;
115 typedef std::shared_ptr<SUBSET> PSubset;
117 Result_Tracker_Stability_Evaluator(const IDXTYPE capacity_limit=0):_capacity_limit(capacity_limit) {notify("Result_Tracker_Stability_Evaluator constructor.");}
118 Result_Tracker_Stability_Evaluator(const Result_Tracker_Stability_Evaluator& rtse):parent_impl(rtse), _capacity_limit(rtse._capacity_limit) {notify("Result_Tracker_Stability_Evaluator copy-constructor.");}
119 virtual ~Result_Tracker_Stability_Evaluator() {notify("Result_Tracker_Stability_Evaluator destructor.");}
120
121 virtual bool add(const RETURNTYPE value, const PSubset sub) override;
122 virtual void clear() override {results.clear();}
123 virtual std::size_t size() const override {return results.size();}
124 virtual bool get_first(PResultRec &r) override;
125 virtual bool get_next(PResultRec &r) override;
126
127 RETURNTYPE value_mean() const;
128 RETURNTYPE value_stddev() const;
129 RETURNTYPE size_mean() const;
130 RETURNTYPE size_stddev() const;
131
132 // Stability of THIS collection. All return a value in [0,1], higher = more stable
133 // (1.0 = all trials selected the same subset), and all return 1.0 for a single held
134 // result and 0.0 for an empty collection. Measures taking Ysiz need the size of the
135 // whole feature universe Y, i.e. the number of features the selection chose FROM.
136 RETURNTYPE stability_C() const;
137 RETURNTYPE stability_CW() const;
142 RETURNTYPE stability_CWrel(const DIMTYPE Ysiz) const;
143 RETURNTYPE stability_SH() const;
147 RETURNTYPE stability_PH(const DIMTYPE Ysiz) const;
148 RETURNTYPE stability_ATI() const;
152 RETURNTYPE stability_ANHI(const DIMTYPE Ysiz) const;
153
154 // Similarity of this collection to ANOTHER collection \p rtse. All return a value in
155 // [0,1], higher = more similar (1.0 = indistinguishable feature-occurrence pattern),
156 // and 0.0 with a warning if either collection is empty. The relation is symmetric.
157 RETURNTYPE similarity_IC(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const;
158 RETURNTYPE similarity_ICW(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const;
159 RETURNTYPE similarity_IATI(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const;
164 RETURNTYPE similarity_IANHI(const DIMTYPE Ysiz, const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const;
165
166
167
168 virtual std::ostream& print(std::ostream& os) const override {os << "Result_Tracker_Stability_Evaluator(limit=" << _capacity_limit << ") size " << size(); return os;}
169protected:
170 std::list<ResultRec> results;
171 const IDXTYPE _capacity_limit;
172
173 RETURNTYPE get_TI(const PSubset &sub1, const PSubset &sub2) const;
174 RETURNTYPE get_NHI(const DIMTYPE Ysiz, const PSubset &sub1, const PSubset &sub2) const;
175 DIMTYPE max_features() const;
176
177public:
178 typedef typename std::list<ResultRec>::const_iterator CONSTRESULTSITER;
179 typedef typename std::list<ResultRec>::iterator RESULTSITER;
180 RESULTSITER begin() {return results.begin();}
181 RESULTSITER end() {return results.end();}
182
183private:
184 RESULTSITER getiter;
185 // Scratch buffers for get_TI/get_NHI/similarity_IC/similarity_ICW. They are per-object
186 // members ON PURPOSE: shared (e.g. static) buffers would be written by every instance and
187 // every thread at once. Holding them as vectors keeps the class copyable for CLONE_DEEP.
188 // Consequently the const measure methods are NOT safe to call concurrently on one object.
189 mutable std::vector<int> _ti_mask;
190 mutable std::vector<int> _nhi_mask;
191 mutable std::vector<int> _sim_F1;
192 mutable std::vector<int> _sim_F2;
193};
194
195
196
197template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
198bool Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::add(const RETURNTYPE value, const PSubset sub)
199{
200 assert(sub);
201 ResultRec rr; rr.value=value; rr.sub.reset(new SUBSET(*sub));
202 if(!results.empty())
203 {
204 RESULTSITER iter=results.begin();
205 while(iter!=results.end() && (value<iter->value || (value==iter->value && sub->get_d()>iter->sub->get_d()))) iter++;
206 results.insert(iter,rr);
207 while(_capacity_limit>0 && results.size()>_capacity_limit) results.pop_back(); // if there is a limit, trim the tail so that the list size remains restricted
208 } else results.push_back(rr);
209 return true;
210}
211
212template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
213bool Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::get_first(PResultRec &r)
214{
215 if(results.empty()) return false;
216 getiter=results.begin();
217 r=&(*getiter);
218 ++getiter;
219 return true;
220}
221
222template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
223bool Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::get_next(PResultRec &r)
224{
225 if(getiter==results.end()) return false;
226 r=&(*getiter);
227 ++getiter;
228 return true;
229}
230
231template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
232RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::value_mean() const
233{
234 if(results.empty()) throw fst_error("Result_Tracker_Stability_Evaluator::value_mean() called on empty list.");
235 RETURNTYPE mean=0.0;
236 for(CONSTRESULTSITER iter=results.begin();iter!=results.end();iter++) mean+=iter->value;
237 return mean/(RETURNTYPE)results.size();
238}
239
240template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
241RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::value_stddev() const
242{
243 if(results.empty()) throw fst_error("Result_Tracker_Stability_Evaluator::value_stddev() called on empty list.");
244 const RETURNTYPE mean=value_mean();
245 RETURNTYPE sd=0.0;
246 for(CONSTRESULTSITER iter=results.begin();iter!=results.end();iter++) sd+=(iter->value-mean)*(iter->value-mean);
247 return sqrt(sd/(RETURNTYPE)results.size());
248}
249
250template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
251RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::size_mean() const
252{
253 if(results.empty()) throw fst_error("Result_Tracker_Stability_Evaluator::size_mean() called on empty list.");
254 RETURNTYPE mean=0.0;
255 for(CONSTRESULTSITER iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); mean+=iter->sub->get_d_raw();}
256 return mean/(RETURNTYPE)results.size();
257}
258
259template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
260RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::size_stddev() const
261{
262 if(results.empty()) throw fst_error("Result_Tracker_Stability_Evaluator::size_stddev() called on empty list.");
263 const RETURNTYPE mean=size_mean();
264 RETURNTYPE sd=0.0;
265 for(CONSTRESULTSITER iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); sd+=(iter->sub->get_d_raw()-mean)*(iter->sub->get_d_raw()-mean);}
266 return sqrt(sd/(RETURNTYPE)results.size());
267}
268
269template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
270DIMTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::max_features() const
271{
272 DIMTYPE maxn=0;
273 if(!results.empty())
274 {
275 CONSTRESULTSITER iter=results.begin();
276 while(iter!=results.end()) {
277 DIMTYPE ss=iter->sub->get_n_raw();
278 if(ss>maxn) maxn=ss;
279 iter++;
280 }
281 }
282 return maxn;
283}
284
285template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
286RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_C() const
287{
288 const long size=results.size();
289 if(size>1)
290 {
291 RETURNTYPE result=0.0;
292 DIMTYPE count=0;
293 for(DIMTYPE f=0;f<max_features();f++) {
294 long fcnt=0;
295 CONSTRESULTSITER iter;
296 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); if(iter->sub->selected_raw(f)) fcnt++;}
297 if(fcnt>0) {
298 result+=(RETURNTYPE)(fcnt-1)/(RETURNTYPE)(size-1);
299 count++;
300 }
301 }
302 if(count==0) return 0.0;
303 return result/(RETURNTYPE)count;
304 }
305 else if(size==1) return 1.0;
306 else return 0.0;
307}
308
309template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
310RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_CW() const
311{
312 const long size=results.size();
313 if(size>1)
314 {
315 RETURNTYPE result=0.0;
316 DIMTYPE count=0;
317 long fcntsum=0;
318 CONSTRESULTSITER iter;
319 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); fcntsum+=iter->sub->get_d_raw();}
320
321 for(DIMTYPE f=0;f<max_features();f++) {
322 long fcnt=0;
323 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); if(iter->sub->selected_raw(f)) fcnt++;}
324 if(fcnt>0) {
325 result+=((RETURNTYPE)(fcnt)/(RETURNTYPE)(fcntsum)) * ((RETURNTYPE)(fcnt-1)/(RETURNTYPE)(size-1));
326 count++;
327 }
328 }
329 // unlike "C", the weighting already normalizes the sum, so count only tells
330 // whether any feature was ever selected at all
331 if(count==0) return 0.0;
332 return result;
333 }
334 else if(size==1) return 1.0;
335 else return 0.0;
336}
337
338template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
339RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_CWrel(const DIMTYPE Ysiz) const
340{
341 assert(Ysiz>0);
342 const long size=results.size();
343 if(size>1)
344 {
345 RETURNTYPE result=0.0;
346 DIMTYPE count=0;
347 long N=0;
348 CONSTRESULTSITER iter;
349 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); N+=iter->sub->get_d_raw();}
350
351 for(DIMTYPE f=0;f<Ysiz;f++) { // over the whole feature universe Y, not only the features seen in the held subsets
352 long fcnt=0;
353 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); if(iter->sub->selected_raw(f)) fcnt++;}
354 if(fcnt>0) {
355 result+=((RETURNTYPE)(fcnt)/(RETURNTYPE)(N)) * ((RETURNTYPE)(fcnt-1)/(RETURNTYPE)(size-1));
356 count++;
357 }
358 }
359 if(count==0) return 0.0;
360 // Rescale the raw weighted consistency between the extremes attainable for this
361 // many subsets (size) of this total cardinality (N) over this feature universe (Y):
362 // CWmin is reached when the N selections are spread as evenly as possible over Y
363 // features, CWmax when they are packed into as few features as possible. D and H are
364 // the remainders of those two even/packed splits. Both closed forms are from the
365 // TPAMI paper cited in the class documentation.
366 long Y=Ysiz;
367 long D=N%Y, H=N%size;
368 RETURNTYPE CWmin=(RETURNTYPE)((RETURNTYPE)N*(RETURNTYPE)N-(RETURNTYPE)Y*(RETURNTYPE)(N-D)-D*D)/(RETURNTYPE)((RETURNTYPE)Y*(RETURNTYPE)N*(RETURNTYPE)(size-1));
369 RETURNTYPE CWmax=(RETURNTYPE)(H*H+N*(size-1)-H*size)/(RETURNTYPE)(N*(size-1));
370 if(CWmax>CWmin) return (result-CWmin)/(CWmax-CWmin); else return result;
371 }
372 else if(size==1) return 1.0;
373 else return 0.0;
374}
375
376template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
377RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_SH() const
378{
379 const long size=results.size();
380 if(size>1)
381 {
382 RETURNTYPE result=0.0;
383 DIMTYPE count=0;
384 long fcntsum=0;
385 CONSTRESULTSITER iter;
386 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); fcntsum+=iter->sub->get_d_raw();}
387
388 for(DIMTYPE f=0;f<max_features();f++) {
389 long fcnt=0;
390 for(iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); if(iter->sub->selected_raw(f)) fcnt++;}
391 if(fcnt>0) {
392 result+=fcnt * (log((double)fcnt)/log((double)2.0));
393 count++;
394 }
395 }
396 if(count==0) return 0.0;
397 return result/(fcntsum*(log((double)size)/log((double)2.0)));
398 }
399 else if(size==1) return 1.0;
400 else return 0.0;
401}
402
403template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
404RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_PH(const DIMTYPE Ysiz) const
405{
406 assert(Ysiz>0);
407 const long size=results.size();
408 if(size>1)
409 {
410 RETURNTYPE result=0.0;
411 for(DIMTYPE f=0;f<Ysiz;f++) {
412 long fcnt=0;
413 for(CONSTRESULTSITER iter=results.begin();iter!=results.end();iter++) {assert(iter->sub); if(iter->sub->selected_raw(f)) fcnt++;}
414 result+=((fcnt>size-fcnt)?(RETURNTYPE)fcnt/(RETURNTYPE)size:(RETURNTYPE)(size-fcnt)/(RETURNTYPE)size);
415 }
416 return 2.0*result/(RETURNTYPE)Ysiz -1.0;
417 }
418 else if(size==1) return 1.0;
419 else return 0.0;
420}
421
422template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
423RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::get_TI(const PSubset &sub1, const PSubset &sub2) const
424{
425 assert(sub1);
426 assert(sub2);
427 const DIMTYPE Ysiz=maxf<DIMTYPE>(sub1->get_n_raw(),sub2->get_n_raw());
428 std::vector<int> &TI_mask=_ti_mask; // per-object scratch, see the note at its declaration
429 assert(Ysiz>0);
430 if((DIMTYPE)TI_mask.size()<Ysiz) {
431 TI_mask.resize(Ysiz);
432 notify("get_ATI reallocation.");
433 }
434
435 DIMTYPE f;
436 for(f=0;f<Ysiz;f++) TI_mask[f]=0;
437 // mark each feature with how many of the two subsets contain it (0, 1 or 2)
438 for(DIMTYPE fsel : sub1->selected_features()) {assert(fsel<Ysiz); TI_mask[fsel]++;}
439 for(DIMTYPE fsel : sub2->selected_features()) {assert(fsel<Ysiz); TI_mask[fsel]++;}
440
441 DIMTYPE SAS=0, SUS=0;
442 for(f=0;f<Ysiz;f++) {if(TI_mask[f]==2) SAS++; if(TI_mask[f]!=0) SUS++;}
443 if(SUS==0) {notify("Warning: get_TI(): SUS==0."); return -1.0;}
444 else return (RETURNTYPE)SAS/(RETURNTYPE)SUS;
445}
446
447template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
448RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_ATI() const
449{
450 const long size=results.size();
451 if(size>1)
452 {
453 RETURNTYPE result=0.0;
454 CONSTRESULTSITER iter1,iter2;
455 for(iter1=results.begin();iter1!=results.end();iter1++)
456 {
457 iter2=iter1; iter2++;
458 while(iter2!=results.end()) {
459 result+=get_TI(iter1->sub,iter2->sub);
460 iter2++;
461 }
462 }
463 return (2.0/(size*(size-1)))*result;
464 }
465 else if(size==1) return 1.0;
466 else return 0.0;
467}
468
469template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
470RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::get_NHI(const DIMTYPE Ysiz, const PSubset &sub1, const PSubset &sub2) const
471{
472 assert(sub1);
473 assert(sub2);
474 assert(Ysiz>0);
475 std::vector<int> &NHI_mask=_nhi_mask; // per-object scratch, see the note at its declaration
476 if((DIMTYPE)NHI_mask.size()<Ysiz) {
477 NHI_mask.resize(Ysiz);
478 notify("get_ANHI reallocation.");
479 }
480
481 DIMTYPE f;
482 for(f=0;f<Ysiz;f++) NHI_mask[f]=0;
483 // mark each feature with how many of the two subsets contain it (0, 1 or 2)
484 for(DIMTYPE fsel : sub1->selected_features()) {assert(fsel<Ysiz); NHI_mask[fsel]++;}
485 for(DIMTYPE fsel : sub2->selected_features()) {assert(fsel<Ysiz); NHI_mask[fsel]++;}
486
487 DIMTYPE SES=0;
488 for(f=0;f<Ysiz;f++) if(NHI_mask[f]==1) SES++;
489 return 1.0-(RETURNTYPE)SES/(RETURNTYPE)Ysiz;
490}
491
492template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
493RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::stability_ANHI(const DIMTYPE Ysiz) const
494{
495 assert(Ysiz>0);
496 const long size=results.size();
497 if(size>1)
498 {
499 RETURNTYPE result=0.0;
500 CONSTRESULTSITER iter1,iter2;
501 for(iter1=results.begin();iter1!=results.end();iter1++)
502 {
503 iter2=iter1; iter2++;
504 while(iter2!=results.end()) {
505 result+=get_NHI(Ysiz,iter1->sub,iter2->sub);
506 iter2++;
507 }
508 }
509 return (2.0/(size*(size-1)))*result;
510 }
511 else if(size==1) return 1.0;
512 else return 0.0;
513}
514
515template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
516RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::similarity_IC(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const
517{
518 const long size1=results.size();
519 const long size2=rtse.results.size();
520 if(size1>0 && size2>0)
521 {
522 const DIMTYPE Ysiz=maxf<DIMTYPE>(max_features(),rtse.max_features());
523 assert(Ysiz>0);
524 std::vector<int> &F1=_sim_F1, &F2=_sim_F2; // per-object scratch, see the note at its declaration
525 if((DIMTYPE)F1.size()<Ysiz) {
526 F1.resize(Ysiz);
527 F2.resize(Ysiz);
528 notify("similarity_IC reallocation.");
529 }
530 DIMTYPE f;
531 for(f=0;f<Ysiz;f++) F1[f]=F2[f]=0;
532 for(f=0;f<Ysiz;f++) {
533 CONSTRESULTSITER iter1;
534 for(iter1=results.begin();iter1!=results.end();iter1++) {assert(iter1->sub); if(iter1->sub->selected_raw(f)) F1[f]++;}
535 CONSTRESULTSITER iter2;
536 for(iter2=rtse.results.begin();iter2!=rtse.results.end();iter2++) {assert(iter2->sub); if(iter2->sub->selected_raw(f)) F2[f]++;}
537 }
538 RETURNTYPE result=0.0;
539 DIMTYPE count=0;
540 for(f=0;f<Ysiz;f++) {
541 if(F1[f]>0 || F2[f]>0) {
542 result+=fabs( (RETURNTYPE)(F1[f])/(RETURNTYPE)(size1) - (RETURNTYPE)(F2[f])/(RETURNTYPE)(size2) );
543 count++;
544 }
545 }
546 assert(count>0);
547 return 1.0 - result/(RETURNTYPE)count;
548 } else {notify("Warning: similarity_IC(): nothing to compare."); return 0.0;}
549}
550
551template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
552RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::similarity_ICW(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const
553{
554 const long size1=results.size();
555 const long size2=rtse.results.size();
556 if(size1>0 && size2>0)
557 {
558 const DIMTYPE Ysiz=maxf<DIMTYPE>(max_features(),rtse.max_features());
559 assert(Ysiz>0);
560 std::vector<int> &F1=_sim_F1, &F2=_sim_F2; // per-object scratch, see the note at its declaration
561 if((DIMTYPE)F1.size()<Ysiz) {
562 F1.resize(Ysiz);
563 F2.resize(Ysiz);
564 notify("similarity_ICW reallocation.");
565 }
566 DIMTYPE f;
567 for(f=0;f<Ysiz;f++) F1[f]=F2[f]=0;
568 for(f=0;f<Ysiz;f++) {
569 CONSTRESULTSITER iter1;
570 for(iter1=results.begin();iter1!=results.end();iter1++) {assert(iter1->sub); if(iter1->sub->selected_raw(f)) F1[f]++;}
571 CONSTRESULTSITER iter2;
572 for(iter2=rtse.results.begin();iter2!=rtse.results.end();iter2++) {assert(iter2->sub); if(iter2->sub->selected_raw(f)) F2[f]++;}
573 }
574 RETURNTYPE fcntsum=0.0;
575 for(f=0;f<Ysiz;f++) {
576 fcntsum+= maxf<RETURNTYPE>( (RETURNTYPE)(F1[f])/(RETURNTYPE)(size1) , (RETURNTYPE)(F2[f])/(RETURNTYPE)(size2) );
577 }
578 assert(fcntsum>0);
579 RETURNTYPE result=0.0;
580 for(f=0;f<Ysiz;f++) {
581 result+=(maxf<RETURNTYPE>( (RETURNTYPE)(F1[f])/(RETURNTYPE)(size1) , (RETURNTYPE)(F2[f])/(RETURNTYPE)(size2) ) / fcntsum) * fabs( (RETURNTYPE)(F1[f])/(RETURNTYPE)(size1) - (RETURNTYPE)(F2[f])/(RETURNTYPE)(size2) );
582 }
583 return 1.0 - result;
584 }
585 else {notify("Warning: similarity_ICW(): nothing to compare."); return 0.0;}
586}
587
588template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
589RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::similarity_IATI(const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const
590{
591 const long size1=results.size();
592 const long size2=rtse.results.size();
593 if(size1>0 && size2>0)
594 {
595 RETURNTYPE result=0.0;
596 CONSTRESULTSITER iter1;
597 for(iter1=results.begin();iter1!=results.end();iter1++)
598 {
599 assert(iter1->sub);
600 CONSTRESULTSITER iter2;
601 for(iter2=rtse.results.begin();iter2!=rtse.results.end();iter2++)
602 {
603 assert(iter2->sub);
604 result+=get_TI(iter1->sub,iter2->sub);
605 }
606 }
607 return (1.0/((RETURNTYPE)size1*(RETURNTYPE)(size2)))*result;
608 }
609 else {notify("Warning: similarity_IATI(): nothing to compare."); return 0.0;}
610}
611
612template<class RETURNTYPE, typename IDXTYPE, typename DIMTYPE, class SUBSET>
613RETURNTYPE Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET>::similarity_IANHI(const DIMTYPE Ysiz, const Result_Tracker_Stability_Evaluator<RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET> &rtse) const
614{
615 const long size1=results.size();
616 const long size2=rtse.results.size();
617 if(size1>0 && size2>0)
618 {
619 assert(Ysiz>0);
620 RETURNTYPE result=0.0;
621 CONSTRESULTSITER iter1;
622 for(iter1=results.begin();iter1!=results.end();iter1++)
623 {
624 assert(iter1->sub);
625 CONSTRESULTSITER iter2;
626 for(iter2=rtse.results.begin();iter2!=rtse.results.end();iter2++)
627 {
628 assert(iter2->sub);
629 result+=get_NHI(Ysiz,iter1->sub,iter2->sub);
630 }
631 }
632 return (1.0/((RETURNTYPE)size1*(RETURNTYPE)(size2)))*result;
633 }
634 else {notify("Warning: similarity_IANHI(): nothing to compare."); return 0.0;}
635}
636
637} // namespace
638#endif // FSTRESULTTRACKERSTABILEVAL_H ///:~
FST::Clonable_Impl
CRTP mixin implementing the Clonable protocol for a concrete class.
Definition clonable.hpp:184
FST::Result_Tracker_Stability_Evaluator
Collects multiple results to evaluate various stability measures on the collection.
Definition result_tracker_stabileval.hpp:109
FST::Result_Tracker_Stability_Evaluator::print
virtual std::ostream & print(std::ostream &os) const override
writes a human-readable identification of the object; base default writes nothing
Definition result_tracker_stabileval.hpp:168
FST::Result_Tracker_Stability_Evaluator::Result_Tracker_Stability_Evaluator
Result_Tracker_Stability_Evaluator(const IDXTYPE capacity_limit=0)
Definition result_tracker_stabileval.hpp:117
FST::Result_Tracker_Stability_Evaluator::size_stddev
RETURNTYPE size_stddev() const
standard deviation of the size of ResultRec::sub subsets; throws fst_error on an empty collection
Definition result_tracker_stabileval.hpp:260
FST::Result_Tracker_Stability_Evaluator::value_stddev
RETURNTYPE value_stddev() const
standard deviation of the ResultRec::value values; throws fst_error on an empty collection
Definition result_tracker_stabileval.hpp:241
FST::Result_Tracker_Stability_Evaluator::results
std::list< ResultRec > results
held results, sorted descending by value (ties broken toward the smaller subset); duplicates are perm...
Definition result_tracker_stabileval.hpp:170
FST::Result_Tracker_Stability_Evaluator::_sim_F1
std::vector< int > _sim_F1
similarity_IC()/similarity_ICW() scratch: per-feature selection count over this tracker's results
Definition result_tracker_stabileval.hpp:191
FST::Result_Tracker_Stability_Evaluator::PSubset
std::shared_ptr< SUBSET > PSubset
Definition result_tracker_stabileval.hpp:115
FST::Result_Tracker_Stability_Evaluator::size_mean
RETURNTYPE size_mean() const
mean value of the size of ResultRec::sub subsets; throws fst_error on an empty collection
Definition result_tracker_stabileval.hpp:251
FST::Result_Tracker_Stability_Evaluator::stability_CW
RETURNTYPE stability_CW() const
Definition result_tracker_stabileval.hpp:310
FST::Result_Tracker_Stability_Evaluator::max_features
DIMTYPE max_features() const
size of the feature universe the held results were selected from: the largest sub->get_n_raw() over t...
Definition result_tracker_stabileval.hpp:270
FST::Result_Tracker_Stability_Evaluator::Result_Tracker_Stability_Evaluator
Result_Tracker_Stability_Evaluator(const Result_Tracker_Stability_Evaluator &rtse)
copies the capacity limit only; the results list is NOT copied (starts empty) – this constructor back...
Definition result_tracker_stabileval.hpp:118
FST::Result_Tracker_Stability_Evaluator::parent_impl
Clonable_Impl< Result_Tracker_Stability_Evaluator, parent, CLONE_INHERIT, CLONE_INHERIT, CLONE_DEEP > parent_impl
implements stateless_clone() via the copy-constructor (CLONE_DEEP); clone()/sharing_clone() are inher...
Definition result_tracker_stabileval.hpp:112
FST::Result_Tracker_Stability_Evaluator::add
virtual bool add(const RETURNTYPE value, const PSubset sub) override
records one (value, subset) result; returns false if the tracker declines to keep it (concrete tracke...
Definition result_tracker_stabileval.hpp:198
FST::Result_Tracker_Stability_Evaluator::CONSTRESULTSITER
std::list< ResultRec >::const_iterator CONSTRESULTSITER
read-only iterator over results, used throughout the stability/similarity measure computations
Definition result_tracker_stabileval.hpp:178
FST::Result_Tracker_Stability_Evaluator::similarity_ICW
RETURNTYPE similarity_ICW(const Result_Tracker_Stability_Evaluator< RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET > &rtse) const
similarity measure "ICW", the two-collection counterpart of stability_CW(); [0,1],...
Definition result_tracker_stabileval.hpp:552
FST::Result_Tracker_Stability_Evaluator::_sim_F2
std::vector< int > _sim_F2
similarity_IC()/similarity_ICW() scratch: per-feature selection count over the other tracker's result...
Definition result_tracker_stabileval.hpp:192
FST::Result_Tracker_Stability_Evaluator::stability_ATI
RETURNTYPE stability_ATI() const
Definition result_tracker_stabileval.hpp:448
FST::Result_Tracker_Stability_Evaluator::similarity_IATI
RETURNTYPE similarity_IATI(const Result_Tracker_Stability_Evaluator< RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET > &rtse) const
Definition result_tracker_stabileval.hpp:589
FST::Result_Tracker_Stability_Evaluator::stability_CWrel
RETURNTYPE stability_CWrel(const DIMTYPE Ysiz) const
stability measure "CW_{rel}": "CW" rescaled between the minimum and maximum attainable for the observ...
Definition result_tracker_stabileval.hpp:339
FST::Result_Tracker_Stability_Evaluator::get_NHI
RETURNTYPE get_NHI(const DIMTYPE Ysiz, const PSubset &sub1, const PSubset &sub2) const
pairwise normalized Hamming index of sub1 and sub2 over an explicit Ysiz-feature universe: 1 - (symme...
Definition result_tracker_stabileval.hpp:470
FST::Result_Tracker_Stability_Evaluator::getiter
RESULTSITER getiter
iteration cursor for get_first()/get_next() (single shared cursor, not reentrant)
Definition result_tracker_stabileval.hpp:184
FST::Result_Tracker_Stability_Evaluator::_ti_mask
std::vector< int > _ti_mask
get_TI() scratch: per-feature selection count (0/1/2) over the current pair
Definition result_tracker_stabileval.hpp:189
FST::Result_Tracker_Stability_Evaluator::parent
Result_Tracker< RETURNTYPE, SUBSET > parent
the abstract Result_Tracker interface this class implements
Definition result_tracker_stabileval.hpp:111
FST::Result_Tracker_Stability_Evaluator::similarity_IANHI
RETURNTYPE similarity_IANHI(const DIMTYPE Ysiz, const Result_Tracker_Stability_Evaluator< RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET > &rtse) const
similarity measure "IANHI": mean normalized Hamming index over all cross pairs (one subset from each ...
Definition result_tracker_stabileval.hpp:613
FST::Result_Tracker_Stability_Evaluator::stability_C
RETURNTYPE stability_C() const
stability measure "C" (consistency: mean over features of the relative frequency of co-occurrence in ...
Definition result_tracker_stabileval.hpp:286
FST::Result_Tracker_Stability_Evaluator::ResultRec
parent::ResultRec ResultRec
see Result_Tracker::ResultRec
Definition result_tracker_stabileval.hpp:113
FST::Result_Tracker_Stability_Evaluator::stability_ANHI
RETURNTYPE stability_ANHI(const DIMTYPE Ysiz) const
stability measure "ANHI" (average pairwise normalized Hamming index over all pairs of held subsets)....
Definition result_tracker_stabileval.hpp:493
FST::Result_Tracker_Stability_Evaluator::stability_PH
RETURNTYPE stability_PH(const DIMTYPE Ysiz) const
stability measure "PH" (relative Hamming distance between the held subsets, expressed as agreement)....
Definition result_tracker_stabileval.hpp:404
FST::Result_Tracker_Stability_Evaluator::get_TI
RETURNTYPE get_TI(const PSubset &sub1, const PSubset &sub2) const
pairwise Tanimoto-style index of sub1 and sub2: (size of selected-feature intersection) / (size of se...
Definition result_tracker_stabileval.hpp:423
FST::Result_Tracker_Stability_Evaluator::RESULTSITER
std::list< ResultRec >::iterator RESULTSITER
mutable iterator over results, as returned by begin()/end()
Definition result_tracker_stabileval.hpp:179
FST::Result_Tracker_Stability_Evaluator::_capacity_limit
const IDXTYPE _capacity_limit
set by the constructor; 0 = unlimited
Definition result_tracker_stabileval.hpp:171
FST::Result_Tracker_Stability_Evaluator::end
RESULTSITER end()
iterator past the last held result
Definition result_tracker_stabileval.hpp:181
FST::Result_Tracker_Stability_Evaluator::begin
RESULTSITER begin()
iterator to the best (first) held result
Definition result_tracker_stabileval.hpp:180
FST::Result_Tracker_Stability_Evaluator::value_mean
RETURNTYPE value_mean() const
mean value of the ResultRec::value values; throws fst_error on an empty collection
Definition result_tracker_stabileval.hpp:232
FST::Result_Tracker_Stability_Evaluator::similarity_IC
RETURNTYPE similarity_IC(const Result_Tracker_Stability_Evaluator< RETURNTYPE, IDXTYPE, DIMTYPE, SUBSET > &rtse) const
similarity measure "IC", the two-collection counterpart of stability_C(); [0,1], higher = more simila...
Definition result_tracker_stabileval.hpp:516
FST::Result_Tracker_Stability_Evaluator::clear
virtual void clear() override
discards all recorded results
Definition result_tracker_stabileval.hpp:122
FST::Result_Tracker_Stability_Evaluator::stability_SH
RETURNTYPE stability_SH() const
Definition result_tracker_stabileval.hpp:377
FST::Result_Tracker_Stability_Evaluator::get_first
virtual bool get_first(PResultRec &r) override
iterator
Definition result_tracker_stabileval.hpp:213
FST::Result_Tracker_Stability_Evaluator::size
virtual std::size_t size() const override
number of results currently held (std::size_t rather than 'long': on LLP64 platforms 'long' is 32 bit...
Definition result_tracker_stabileval.hpp:123
FST::Result_Tracker_Stability_Evaluator::_nhi_mask
std::vector< int > _nhi_mask
get_NHI() scratch: per-feature selection count (0/1/2) over the current pair
Definition result_tracker_stabileval.hpp:190
FST::Result_Tracker_Stability_Evaluator::PResultRec
ResultRec * PResultRec
raw pointer to a ResultRec, as used by get_first()/get_next() iteration
Definition result_tracker_stabileval.hpp:114
FST::Result_Tracker_Stability_Evaluator::get_next
virtual bool get_next(PResultRec &r) override
iterator
Definition result_tracker_stabileval.hpp:223
FST::Result_Tracker
Abstract class, defines interface for classes that enable collecting multiple results.
Definition result_tracker.hpp:91
FST::fst_error
FST4 specific error passing class.
Definition error.hpp:61
error.hpp
FST4 specific error passing class.
global.hpp
Global definitions.
FST
The Feature Selection Toolbox 4 library: every library component – data access, criteria and classifi...
Definition data_accessor.hpp:80
FST::notify
void notify(const std::string msg)
Prints msg followed by a newline via syncout::print(); a no-op unless the preprocessor macro DEBUG is...
Definition global.hpp:112
FST::maxf
VALUE maxf(const VALUE &x, const VALUE &y)
Returns the greater of x and y (strict '>' comparison; ties return y).
Definition global.hpp:178
result_tracker.hpp
Defines interface for classes that enable collecting multiple results.
FST::Result_Tracker::ResultRec
Structure to hold [subset,criterion value] temporary solutions in the course of search.
Definition result_tracker.hpp:99
FST::Result_Tracker::ResultRec::value
RETURNTYPE value
the criterion value of the recorded subset
Definition result_tracker.hpp:100
FST::Result_Tracker::ResultRec::sub
PSubset sub
the recorded subset itself
Definition result_tracker.hpp:101
Generated by doxygen 1.17.0
Copyright © 2000–present ÚTIA, Institute of Information Theory and Automation of the Czech Academy of Sciences · Contacts · This site sets no cookies and uses no analytics.