Reproducability, Reuse und Interactive Information RetrievalDr. Mark HallInstitut für Informatik, Martin-Luther-Universität Halle-Wittenberg
Cranfield & Sharing
It works
Interactive IR
- Fragestellung
- Tasks auswählen
- UI / System bauen
- Metriken
- Fragebögen
- Teilnehmer/innen finden
- Go!
- Auswerten
Shared IIR Evaluation
Konsequenzen
- Kaum Re-use
- Schwierige Reproduzierbarkeit
- Viel verlorener Aufwand
Wen interessiert es?
- SCST I
- SCST II
- BIIRRR
Was kann man recyceln?
- Tasks
- Metriken
- Protokolle
- Fragebögen
- Systeme / UIs
- Interaction Data
Fragebogen im Rahmen von BIIRRR
26 Forscher/innen aus dem IIR Bereich
Kommentare
Published papers often contain incomplete information; materials may need to be customized to current study
Difficulties in understanding just what had been done by the other study. Difficulties in rationalizing, normalizing and integrating data collected in different ways.
Kommentare
But the real question what is does reusability provide and is it worth it?
Experiments are costly. Making them reusable increasing costs. But will that make it useful others.
This survey is flawed because it assumes that this is somehow needed. It tries to justify the PROMISE made.
Certain thing should be reusable. Ie the method aspects. But others it does not. And is a total waste of time.
Kommentare
I am not sure we should strive for reusability. I believe IIR studies should be performed with both algorithm and users in the loop (as a living lab) which inherently prevents reusability as there will be interactions between user and algorithm.
Re-use ist nicht 1:1
iRepository
Shared Repository for IIR Resourcen
Low overhead
Open