|
fallback solution
|
Henning Müller
|
Oct 22, 2001 02:37 PDT
|
Dear Benchathletes,
as Stephane said in his mail: It is very important that we can show a
working benchmark framework in January.
If we do not manage to get the complete benchmarking harness to work by
January, we can use the perl scripts of our benchmark and adapt them to
the performance measures, relevance judgements and image databases of
the Benchathlon.
The scripts are very general, so they can easily be adapted to new
databases or judgements.
All the scripts need, are an image database in form of a URL list, and
then a number of images chosen as query images for evaluation. For every
query image a number of relevance sets can be generated to be able to
evaluate the performance based on different relevances or different user
judgements. The final results are averaged over all the relevance sets
and all the queries.
Relevance feedback is being generated for every relevance set and thus
the performance with relevance feedback can be evaluated.
For the generation of relevance feedback we assume that a user would
mark all relevant images returned in the first n (=20 for example)
result images as positive feedback and everything else as negative
feedback.
The queries are all done in MRML at the moment.
More about the scripts can be read at:
http://viper.unige.ch/benchmarking/
I can also send you a copy of the scripts if anybody is interested.
Henning
--
----------------------------------------------------------
Henning Mueller, Computer Vision Group
Computer Science Department, University of Geneva
24, rue du General Dufour, CH-1211 Geneva 4, SWITZERLAND
Phone : +41(22)705 7633; fax: +41(22)705 7780
Henning.-@cui.unige.ch
----------------------------------------------------------
|
|
 |
|
