检索中常用指标

Mean ReciProcal Rank(MRR)

多个查询的倒数排名的均值.

$$MRR = \frac{1}{N}\sum_{i=1}^{N}\frac{1}{Rank_{i}}$$

import numpy as np
y_pred = np.asarray([0.2,0.3,0.7,1.0])
y_true = np.asarray([1,0,0,0])

coupled_pair = list(zip(y_true, y_pred))
coupled_pair = np.asarray(sorted(coupled_pair, key=lambda x: x[1], reverse=True))
coupled_pair

array([[0. , 1. ],
       [0. , 0.7],
       [0. , 0.3],
       [1. , 0.2]])

for idx, (label, pred) in enumerate(coupled_pair):
    if label > 0.:
        print(1. / (idx + 1))
        break

0.25

from matchzoo.metrics import MeanReciprocalRank

MeanReciprocalRank()(y_true, y_pred)

0.25

Precision

通常记为P@k, k为排序后的前k位

y_true = [0,0,0,1]
y_pred = [0.2, 0.4, 0.3, 0.1]
coupled_pair = list(zip(y_true, y_pred))
coupled_pair = np.asarray(sorted(coupled_pair, key=lambda x: x[1], reverse=True))
coupled_pair

array([[0. , 0.4],
       [0. , 0.3],
       [0. , 0.2],
       [1. , 0.1]])

p@k 则是计算前k位

precision = 0.0
k = 4
for idx, (label, score) in enumerate(coupled_pair):
    if idx >= k:
        break
    if label > 0.:
        precision += 1
print(precision / k)

0.25

Average Precision(AerP)

计算从1到k位的P@k的平均值, AUC(area under PR curve), PR曲线下的面积

from matchzoo.metrics import Precision
y_true = [0, 1]
y_pred = [0.1, 0.6]

precision_metrics = [Precision(k + 1) for k in range(len(y_pred))]
out = [metric(y_true, y_pred) for metric in precision_metrics]
if not out:
    AP = 0.
else:
    AP = np.mean(out).item() 
AP

0.75

Mean average precision

若检索出四个相关文档（1，2，4，7）
则map = (1/1 + 2/2 + 3/4 + 4/7)/4，再对所有问题求平均

y_true = [0, 1, 0, 0]
y_pred = [0.1, 0.6, 0.2, 0.3]
coupled_pair = list(zip(y_true, y_pred))
coupled_pair = np.asarray(sorted(coupled_pair, key=lambda x: x[1], reverse=True))
coupled_pair

array([[1. , 0.6],
       [0. , 0.3],
       [0. , 0.2],
       [0. , 0.1]])

result = 0.
pos = 0
for idx, (label, score) in enumerate(coupled_pair):
    # 0. is threshold
    if label > 0.:
        pos += 1.
        result += pos / (idx + 1.)
if pos == 0:
    MAP = 0.
else:
    MAP = result / pos
MAP

1.0

Discounted Cumulative Gain(DCG)

需要考虑排序的顺序。

$$\mathrm{DCG} @ \mathrm{k}=\sum_{i=1}^{k} \frac{2^{\mathrm{reli}}-1}{\log _{2(i+1)}}$$

y_true = [0, 1, 2, 0]
y_pred = [0.4, 0.2, 0.5, 0.7]
coupled_pair = list(zip(y_true, y_pred))
coupled_pair = np.asarray(sorted(coupled_pair, key=lambda x: x[1], reverse=True))
coupled_pair

array([[0. , 0.7],
       [2. , 0.5],
       [0. , 0.4],
       [1. , 0.2]])

import math

result = 0.
k = 3
for i, (label, score) in enumerate(coupled_pair):
    if i >= k:
        break
    if label > 0.:
        result += (math.pow(2., label) - 1.) / math.log(2. + i)
print(round(result, 2))

2.73

NDCG (Normalized DCG)

首先要计算出排序完全正确，即理想状态下的IDCG

$$\operatorname{IDCG} @ \mathrm{k}=\sum_{i=1}^{| \mathrm{REL}} \frac{2^{\mathrm{rel}_{-}}-1}{\log _{2}(\mathrm{i}+1)}$$

其中REL是理想状态下排序.然后计算NDCG:

$$\mathrm{NDCG} @ \mathrm{Ok}=\frac{\mathrm{DCG} @ \mathrm{k}}{\mathrm{IDCG} @ \mathrm{k}}$$

coupled_pair

array([[0. , 0.7],
       [2. , 0.5],
       [0. , 0.4],
       [1. , 0.2]])

from matchzoo.metrics import DiscountedCumulativeGain
k = 2

dcg_metric = DiscountedCumulativeGain(k=k, threshold=0.)
idcg_val = dcg_metric(y_true, y_true)
dcg_val = dcg_metric(y_true, y_pred)
NDCG = dcg_val / idcg_val if idcg_val != 0. else 0.
NDCG

0.52129602861432

参考

matchzoo-py

信息检索评价指标