LM perplexity by using tensorflow

1、Language model perplexity是衡量語言模型好壞的重要指標，其計算公式P(sentence)^-(1/N)

2、tensorflow的RNN模型如何使用
參考API文件：tf.contrib.legacy_seq2seq.sequence_loss_by_example，這個函式會返回一個大小為N的列表，N為句子數目，每個值代表該句話的log-perplexity。

tf.contrib.legacy_seq2seq.sequence_loss_by_example(
    logits,
    targets,
    weights,
    average_across_timesteps=True
 
,
    softmax_loss_function=None,
    name=None
)

#計算這個batch的log-perplexity,返回的是shape=[batch]的值，代表每個句子的log-perplexity
loss = legacy_seq2seq.sequence_loss_by_example([self.logits],
                [tf.reshape(self.targets, [-1])],
                [tf.ones([args.batch_size * args.seq_length])],
                args.vocab
 
_size)
#計算這個batch中所有句子的平均log-perplexity          
self.cost = tf.reduce_sum(loss) / args.batch_size
#計算這個batch中所有句子的平均perplexity
self.perplexity = tf.exp(self.cost)

可以看一下tensorflow對sequence_loss_by_example的原始碼

def sequence_loss_by_example(logits,
                             targets,
                             weights,
                             average_across_timesteps=True,
                             softmax_loss_function=None,
                             name=None)
 
:
  """Weighted cross-entropy loss for a sequence of logits (per example).
  Args:
    logits: List of 2D Tensors of shape [batch_size x num_decoder_symbols].
    targets: List of 1D batch-sized int32 Tensors of the same length as logits.
    weights: List of 1D batch-sized float-Tensors of the same length as logits.
    average_across_timesteps: If set, divide the returned cost by the total
      label weight.
    softmax_loss_function: Function (labels-batch, inputs-batch) -> loss-batch
      to be used instead of the standard softmax (the default if this is None).
    name: Optional name for this operation, default: "sequence_loss_by_example".
  Returns:
    1D batch-sized float Tensor: The log-perplexity for each sequence.
  Raises:
    ValueError: If len(logits) is different from len(targets) or len(weights).
  """
  if len(targets) != len(logits) or len(weights) != len(logits):
    raise ValueError("Lengths of logits, weights, and targets must be the same "
                     "%d, %d, %d." % (len(logits), len(weights), len(targets)))
  with ops.name_scope(name, "sequence_loss_by_example",
                      logits + targets + weights):
    log_perp_list = []
    for logit, target, weight in zip(logits, targets, weights):
      if softmax_loss_function is None:
        # TODO(irving,ebrevdo): This reshape is needed because
        # sequence_loss_by_example is called with scalars sometimes, which
        # violates our general scalar strictness policy.
        target = array_ops.reshape(target, [-1])
        crossent = nn_ops.sparse_softmax_cross_entropy_with_logits(
            labels=target, logits=logit)
      else:
        crossent = softmax_loss_function(target, logit)
      log_perp_list.append(crossent * weight)
    log_perps = math_ops.add_n(log_perp_list)
    if average_across_timesteps:
      total_size = math_ops.add_n(weights)
      total_size += 1e-12  # Just to avoid division by 0 for all-0 weights.
      log_perps /= total_size
  return log_perps

計算的過程是：
1、求句子的每個時間點（RNN timestep）處的loss，然後對每個時間點的loss求和。
2、求句子的長度（timestep），然後loss/timestep。（預設需要對average_across_timesteps ）