后使用快捷导航没有帐号？
查看: 218|回复: 7
TensorFlow学习（四）：优化器Optimizer
新手上路, 积分 38, 距离下一级还需 12 积分
论坛徽章:0
转贴：http://blog.csdn.net/xierhacker/article/details/
反正是要学一些API的，不如直接从例子里面学习怎么使用API，这样同时可以复习一下一些基本的知识。但是一开始开始和以前一样，先直接讲类和常用函数用法，然后举例子。
这里主要是各种优化器，以及使用。因为大多数机器学习任务就是最小化损失，在损失定义的情况下，后面的工作就交给优化器啦。
因为常见的是对于梯度的优化，也就是说，优化器最后其实就是各种对于梯度下降的优化。
理论部分可以见斯坦福深度学习的课程。同时这里推荐一个博客，总结了这些优化器的原理以及性能，写的挺好的：An overview of gradient descent optimazation algorithms
从其中讲几个比较常用的，其他的可以自己去看文档。官方文档：TrainingOptimizer
GradientDescentOptimizer
AdagradOptimizer
AdagradDAOptimizer
MomentumOptimizer
AdamOptimizer
FtrlOptimizer
RMSPropOptimizer二.常用的optimizer类Ⅰ.class tf.train.Optimizer优化器（optimizers）类的基类。这个类定义了在训练模型的时候添加一个操作的API。你基本上不会直接使用这个类，但是你会用到他的子类比如GradientDescentOptimizer, AdagradOptimizer, MomentumOptimizer.等等这些。
后面讲的时候会详细讲一下GradientDescentOptimizer 这个类的一些函数，然后其他的类只会讲构造函数，因为类中剩下的函数都是大同小异的Ⅱ.class tf.train.GradientDescentOptimizer这个类是实现梯度下降算法的优化器。(结合理论可以看到，这个构造函数需要的一个学习率就行了)__init__(learning_rate, use_locking=False,name=’GradientDescent’)作用：创建一个梯度下降优化器对象
learning_rate: A Tensor or a floating point value. 要使用的学习率
use_locking: 要是True的话，就对于更新操作（update operations.）使用锁
name: 名字，可选，默认是”GradientDescent”.compute_gradients(loss,var_list=None,gate_gradients=GATE_OP,aggregation_method=None,colocate_gradients_with_ops=False,grad_loss=None)作用：对于在变量列表（var_list）中的变量计算对于损失函数的梯度,这个函数返回一个（梯度，变量）对的列表，其中梯度就是相对应变量的梯度了。这是minimize()函数的第一个部分，
loss: 待减小的值
var_list: 默认是在GraphKey.TRAINABLE_VARIABLES.
gate_gradients: How to gate the computation of gradients. Can be GATE_NONE, GATE_OP, or GATE_GRAPH.
aggregation_method: Specifies the method used to combine gradient terms. Valid values are defined in the class AggregationMethod.
colocate_gradients_with_ops: If True, try colocating gradients with the corresponding op.
grad_loss: Optional. A Tensor holding the gradient computed for loss.apply_gradients(grads_and_vars,global_step=None,name=None)作用：把梯度“应用”（Apply）到变量上面去。其实就是按照梯度下降的方式加到上面去。这是minimize（）函数的第二个步骤。 返回一个应用的操作。
grads_and_vars: compute_gradients()函数返回的(gradient, variable)对的列表
global_step: Optional Variable to increment by one after the variables have been updated.
name: 可选，名字get_name()minimize(loss,global_step=None,var_list=None,gate_gradients=GATE_OP,aggregation_method=None,colocate_gradients_with_ops=False,name=None,grad_loss=None)作用：非常常用的一个函数
通过更新var_list来减小loss，这个函数就是前面compute_gradients() 和apply_gradients().的结合Ⅲ.class tf.train.AdadeltaOptimizer实现了 Adadelta算法的优化器，可以算是下面的Adagrad算法改进版本构造函数：
tf.train.AdadeltaOptimizer.init(learning_rate=0.001, rho=0.95, epsilon=1e-08, use_locking=False, name=’Adadelta’)作用：构造一个使用Adadelta算法的优化器
learning_rate: tensor或者浮点数，学习率
rho: tensor或者浮点数. The decay rate.
epsilon: A Tensor or a floating point value. A constant epsilon used to better conditioning the grad update.
use_locking: If True use locks for update operations.
name: 【可选】这个操作的名字，默认是”Adadelta”IV.class tf.train.AdagradOptimizerOptimizer that implements the Adagrad algorithm.See this paper.
tf.train.AdagradOptimizer.__init__(learning_rate, initial_accumulator_value=0.1, use_locking=False, name=’Adagrad’)Construct a new Adagrad optimizer.
Args:learning_rate: A Tensor or a floating point value. The learning rate.initial_accumulator_value: A floating point value. Starting value for the accumulators, must be positive.use_locking: If True use locks for update operations.name: Optional name prefix for the operations created when applying gradients. Defaults to &Adagrad&.Raises:ValueError: If the initial_accumulator_value is invalid.The Optimizer base class provides methods to compute gradients for a loss and apply gradients to variables. A collection of subclasses implement classic optimization algorithms such as GradientDescent and Adagrad.You never instantiate the Optimizer class itself, but instead instantiate one of the subclasses.Ⅴ.class tf.train.MomentumOptimizerOptimizer that implements the Momentum algorithm.tf.train.MomentumOptimizer.__init__(learning_rate, momentum, use_locking=False, name=’Momentum’, use_nesterov=False)Construct a new Momentum optimizer.Args:learning_rate: A Tensor or a floating point value. The learning rate.
momentum: A Tensor or a floating point value. The momentum.
use_locking: If True use locks for update operations.
name: Optional name prefix for the operations created when applying gradients. Defaults to “Momentum”.Ⅵ.class tf.train.AdamOptimizer实现了Adam算法的优化器
构造函数：
tf.train.AdamOptimizer.__init__(learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08, use_locking=False, name=’Adam’)Construct a new Adam optimizer.Initialization:m_0 &- 0 (Initialize initial 1st moment vector)
v_0 &- 0 (Initialize initial 2nd moment vector)
t &- 0 (Initialize timestep)
The update rule for variable with gradient g uses an optimization described at the end of section2 of the paper:t &- t + 1
lr_t &- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)m_t &- beta1 * m_{t-1} + (1 - beta1) * g
v_t &- beta2 * v_{t-1} + (1 - beta2) * g * g
variable &- variable - lr_t * m_t / (sqrt(v_t) + epsilon)
The default value of 1e-8 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1.Note that in dense implement of this algorithm, m_t, v_t and variable will update even if g is zero, but in sparse implement, m_t, v_t and variable will not update in iterations g is zero.Args:learning_rate: A Tensor or a floating point value. The learning rate.
beta1: A float value or a constant float tensor. The exponential decay rate for the 1st moment estimates.
beta2: A float value or a constant float tensor. The exponential decay rate for the 2nd moment estimates.
epsilon: A small constant for numerical stability.
use_locking: If True use locks for update operations.
name: Optional name for the operations created when applying gradients. Defaults to “Adam”.三.例子I.线性回归要是有不知道线性回归的理论知识的，请到
http://blog.csdn.net/xierhacker/article/details/
http://blog.csdn.net/xierhacker/article/details/
熟悉的直接跳过。
直接上代码:import tensorflow as tfimport numpy as npimport matplotlib.pyplot as plt# Prepare train datatrain_X = np.linspace(-1, 1, <font color="#0)train_Y = 2 * train_X + np.random.randn(*train_X.shape) * <font color="#0 + 10# Define the modelX = tf.placeholder(&float&)Y = tf.placeholder(&float&)w = tf.Variable(<font color="#0, name=&weight&)b = tf.Variable(<font color="#0, name=&bias&)loss = tf.square(Y - X*w - b)train_op = tf.train.GradientDescentOptimizer(<font color="#0).minimize(loss)# Create session to runwith tf.Session() as sess:& & sess.run(tf.initialize_all_variables())& & epoch = 1& & for i in range(10):& && &&&for (x, y) in zip(train_X, train_Y):& && && && &_, w_value, b_value = sess.run([train_op, w, b],feed_dict={X: x,Y: y})& && &&&print(&Epoch: {}, w: {}, b: {}&.format(epoch, w_value, b_value))& && &&&epoch += 1#drawplt.plot(train_X,train_Y,&+&)plt.plot(train_X,train_X.dot(w_value)+b_value)plt.show()1234567891011121314151617181920212223242526272829303132
1234567891011121314151617181920212223242526272829303132
这里你可以使用更多的优化器来试一下各个优化器的性能和调参情况
注册会员, 积分 165, 距离下一级还需 35 积分
论坛徽章:4
感谢分享，可以看成是第二课的扩展了
金牌会员, 积分 1028, 距离下一级还需 1972 积分
论坛徽章:37
多谢分享，学习了~~~~
注册会员, 积分 79, 距离下一级还需 121 积分
论坛徽章:0
楼主如果把下面的优缺点写清楚就好了。以及适合用在哪里
GradientDescentOptimizer
AdagradOptimizer
AdagradDAOptimizer
MomentumOptimizer
AdamOptimizer
FtrlOptimizer
RMSPropOptimizer
中级会员, 积分 405, 距离下一级还需 95 积分
论坛徽章:8
内容丰富，调优是机器学习工作量大的一块
注册会员, 积分 120, 距离下一级还需 80 积分
论坛徽章:0
谢谢楼主的分享 谢谢..................
中级会员, 积分 351, 距离下一级还需 149 积分
论坛徽章:6
学习了，谢谢楼主分享
高级会员, 积分 994, 距离下一级还需 6 积分
论坛徽章:13
不错&&谢谢分享& && && && &优化器Optimizer - Keras中文文档极客学院团队出品 · 更新于
This library provides a set of classes and functions that helps train models.
Optimizers
The Optimizer base class provides methods to compute gradients for a loss and
apply gradients to variables.
A collection of subclasses implement classic
optimization algorithms such as GradientDescent and Adagrad.
You never instantiate the Optimizer class itself, but instead instantiate one
of the subclasses.
class tf.train.Optimizer
Base class for optimizers.
This class defines the API to add Ops to train a model.
You never use this
class directly, but instead instantiate one of its subclasses such as
GradientDescentOptimizer, AdagradOptimizer, or MomentumOptimizer.
# Create an optimizer with the desired parameters.
opt = GradientDescentOptimizer(learning_rate=0.1)
# Add Ops to the graph to minimize a cost by updating a list of variables.
# "cost" is a Tensor, and the list of variables contains variables.Variable
# objects.
opt_op = opt.minimize(cost, &list of variables&)
In the training program you will just have to run the returned Op.
# Execute opt_op to do one step of training:
opt_op.run()
Processing gradients before applying them.
Calling minimize() takes care of both computing the gradients and
applying them to the variables.
If you want to process the gradients
before applying them you can instead use the optimizer in three steps:
Compute the gradients with compute_gradients().
Process the gradients as you wish.
Apply the processed gradients with apply_gradients().
# Create an optimizer.
opt = GradientDescentOptimizer(learning_rate=0.1)
# Compute the gradients for a list of variables.
grads_and_vars = pute_gradients(loss, &list of variables&)
# grads_and_vars is a list of tuples (gradient, variable).
Do whatever you
# need to the 'gradient' part, for example cap them, etc.
capped_grads_and_vars = [(MyCapper(gv[0]), gv[1])) for gv in grads_and_vars]
# Ask the optimizer to apply the capped gradients.
opt.apply_gradients(capped_grads_and_vars)
tf.train.Optimizer.__init__(use_locking, name)
Create a new Optimizer.
This must be called by the constructors of subclasses.
use_locking: Bool. If True apply use locks to prevent concurrent updates
to variables.
name: A non-empty string.
The name to use for accumulators created
for the optimizer.
ValueError: if name is malformed.
tf.train.Optimizer.minimize(loss, global_step=None, var_list=None, gate_gradients=1, name=None)
Add operations to minimize 'loss' by updating 'var_list'.
This method simply combines calls compute_gradients() and
apply_gradients(). If you want to process the gradient before applying them
call compute_gradients() and apply_gradients() explicitly instead of using
this function.
loss: A Tensor containing the value to minimize.
global_step: Optional Variable to increment by one after the
variables have been updated.
var_list: Optional list of variables.Variable to update to minimize
Defaults to the list of variables collected in the graph
under the key GraphKeys.TRAINABLE_VARIABLES.
gate_gradients: How to gate the computation of gradients.
GATE_NONE, GATE_OP, or
GATE_GRAPH.
name: Optional name for the returned operation.
An Operation that updates the variables in 'var_list'.
If 'global_step'
was not None, that operation also increments global_step.
ValueError: if some of the variables are not variables.Variable objects.
tf.pute_gradients(loss, var_list=None, gate_gradients=1)
Compute gradients of &loss& for the variables in &var_list&.
This is the first part of minimize().
It returns a list
of (gradient, variable) pairs where &gradient& is the gradient
for &variable&.
Note that &gradient& can be a Tensor, a
IndexedSlices, or None if there is no gradient for the
given variable.
loss: A Tensor containing the value to minimize.
var_list: Optional list of variables.Variable to update to minimize
Defaults to the list of variables collected in the graph
under the key GraphKey.TRAINABLE_VARIABLES.
gate_gradients: How to gate the computation of gradients.
GATE_NONE, GATE_OP, or
GATE_GRAPH.
A list of (gradient, variable) pairs.
TypeError: If var_list contains anything else than variables.Variable.
ValueError: If some arguments are invalid.
tf.train.Optimizer.apply_gradients(grads_and_vars, global_step=None, name=None)
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that
applies gradients.
grads_and_vars: List of (gradient, variable) pairs as returned by
compute_gradients().
global_step: Optional Variable to increment by one after the
variables have been updated.
name: Optional name for the returned operation.
Default to the
name passed to the Optimizer constructor.
An Operation that applies the specified gradients. If 'global_step'
was not None, that operation also increments global_step.
TypeError: if grads_and_vars is malformed.
Gating Gradients
Both minimize() and compute_gradients() accept a gate_gradient argument
that controls the degree of parallelism during the application of the
gradients.
The possible values are: GATE_NONE, GATE_OP, and GATE_GRAPH.
GATE_NONE: Compute and apply gradients in parallel.
This provides the
maximum parallelism in execution, at the cost of some non-reproducibility in
the results.
For example the two gradients of MatMul depend on the input
values: With GATE_NONE one of the gradients could be applied to one of the
inputs before the other gradient is computed resulting in non-reproducible
GATE_OP: For each Op, make sure all gradients are computed before they
This prevents race conditions for Ops that generate gradients for
multiple inputs where the gradients depend on the inputs.
GATE_GRAPH: Make sure all gradients for all variables are computed
before any one of them is used.
This provides the least parallelism but can
be useful if you want to process all gradients before applying any of them.
Some optimizer subclasses, such as MomentumOptimizer and AdagradOptimizer
allocate and manage additional variables associated with the variables to
These are called Slots.
Slots have names and you can ask the
optimizer for the names of the slots that it uses.
Once you have a slot name
you can ask the optimizer for the variable it created to hold the slot value.
This can be useful if you want to log debug a training algorithm, report stats
about the slots, etc.
tf.train.Optimizer.get_slot_names()
Return a list of the names of slots created by the Optimizer.
See get_slot().
A list of strings.
tf.train.Optimizer.get_slot(var, name)
Return a slot named &name& created for &var& by the Optimizer.
Some Optimizer subclasses use additional variables.
For example
Momentum and Adagrad use variables to accumulate updates.
This method
gives access to these Variables if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
var: A variable passed to minimize() or apply_gradients().
name: A string.
The Variable for the slot if it was created, None otherwise.
class tf.train.GradientDescentOptimizer
Optimizer that implements the gradient descent algorithm.
tf.train.GradientDescentOptimizer.__init__(learning_rate, use_locking=False, name='GradientDescent')
Construct a new gradient descent optimizer.
learning_rate: A Tensor or a floating point value.
The learning
rate to use.
use_locking: If True use locks for update operation.s
name: Optional name prefix for the operations created when applying
gradients. Defaults to &GradientDescent&.
class tf.train.AdagradOptimizer
Optimizer that implements the Adagrad algorithm.
tf.train.AdagradOptimizer.__init__(learning_rate, initial_accumulator_value=0.1, use_locking=False, name='Adagrad')
Construct a new Adagrad optimizer.
learning_rate: A Tensor or a floating point value.
The learning rate.
initial_accumulator_value: A floating point value.
Starting value for the accumulators, must be positive.
use_locking: If True use locks for update operations.
name: Optional name prefix for the operations created when applying
gradients.
Defaults to &Adagrad&.
ValueError: If the initial_accumulator_value is invalid.
class tf.train.MomentumOptimizer
Optimizer that implements the Momentum algorithm.
tf.train.MomentumOptimizer.__init__(learning_rate, momentum, use_locking=False, name='Momentum')
Construct a new Momentum optimizer.
learning_rate: A Tensor or a floating point value.
The learning rate.
momentum: A Tensor or a floating point value.
The momentum.
use_locking: If True use locks for update operations.
name: Optional name prefix for the operations created when applying
gradients.
Defaults to &Momentum&.
class tf.train.AdamOptimizer
Optimizer that implements the Adam algorithm.
tf.train.AdamOptimizer.__init__(learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-08, use_locking=False, name='Adam')
Construct a new Adam optimizer.
Implementation is based on:
Initialization:
m_0 &- 0 (Initialize initial 1st moment vector)
v_0 &- 0 (Initialize initial 2nd moment vector)
t &- 0 (Initialize timestep)
The update rule for variable with gradient g uses an optimization
described at the end of section2 of the paper:
t &- t + 1
lr_t &- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)
m_t &- beta1 * m_{t-1} + (1 - beta1) * g
v_t &- beta2 * v_{t-1} + (1 - beta2) * g * g
variable &- variable - lr_t * m_t / (sqrt(v_t) + epsilon)
The default value of 1e-8 for epsilon might not be a good default in
general. For example, when training an Inception network on ImageNet a
current good choice is 1.0 or 0.1.
learning_rate: A Tensor or a floating point value.
The learning rate.
beta1: A float value or a constant float tensor.
The exponential decay rate for the 1st moment estimates.
beta2: A float value or a constant float tensor.
The exponential decay rate for the 2st moment estimates.
epsilon: A small constant for numerical stability.
use_locking: If True use locks for update operation.s
name: Optional name for the operations created when applying gradients.
Defaults to &Adam&.
class tf.train.FtrlOptimizer
Optimizer that implements the FTRL algorithm.
tf.train.FtrlOptimizer.__init__(learning_rate, learning_rate_power=-0.5, initial_accumulator_value=0.1, l1_regularization_strength=0.0, l2_regularization_strength=0.0, use_locking=False, name='Ftrl')
Construct a new FTRL optimizer.
The Ftrl-proximal algorithm, abbreviated for Follow-the-regularized-leader,
is described in the paper .
It can give a good performance vs. sparsity tradeoff.
Ftrl-proximal uses its own global base learning rate and can behave like
Adagrad with learning_rate_power=-0.5, or like gradient descent with
learning_rate_power=0.0.
The effective learning rate is adjusted per parameter, relative to this
base learning rate as:
effective_learning_rate_i = (learning_rate /
pow(k + summed_squared_gradients_for_i, learning_rate_power));
where k is the small constant initial_accumulator_value.
Note that the real regularization coefficient of |w|^2 for objective
function is 1 / lambda_2 if specifying l2 = lambda_2 as argument when
using this function.
learning_rate: A float value or a constant float Tensor.
learning_rate_power: A float value, must be less or equal to zero.
initial_accumulator_value: The starting value for accumulators.
Only positive values are allowed.
l1_regularization_strength: A float value, must be greater than or
equal to zero.
l2_regularization_strength: A float value, must be greater than or
equal to zero.
use_locking: If True use locks for update operations.
name: Optional name prefix for the operations created when applying
gradients.
Defaults to &Ftrl&.
ValueError: if one of the arguments is invalid.
class tf.train.RMSPropOptimizer
Optimizer that implements the RMSProp algorithm.
tf.train.RMSPropOptimizer.__init__(learning_rate, decay, momentum=0.0, epsilon=1e-10, use_locking=False, name='RMSProp')
Construct a new RMSProp optimizer.
learning_rate: A Tensor or a floating point value.
The learning rate.
decay: discounting factor for the history/coming gradient
momentum: a scalar tensor.
epsilon: small value to avoid zero denominator.
use_locking: If True use locks for update operation.
name: Optional name prefic for the operations created when applying
gradients. Defaults to &RMSProp&.
Gradient Computation
TensorFlow provides functions to compute the derivatives for a given
TensorFlow computation graph, adding operations to the graph. The
optimizer classes automatically compute derivatives on your graph, but
creators of new Optimizers or expert users can call the lower-level
functions below.
tf.gradients(ys, xs, grad_ys=None, name='gradients', colocate_gradients_with_ops=False, gate_gradients=False, aggregation_method=None)
Constructs symbolic partial derivatives of ys w.r.t. x in xs.
ys and xs are each a Tensor or a list of tensors.
is a list of Tensor, holding the gradients received by the
ys. The list must be the same length as ys.
gradients() adds ops to the graph to output the partial
derivatives of ys with respect to xs.
It returns a list of
Tensor of length len(xs) where each tensor is the sum(dy/dx)
for y in ys.
grad_ys is a list of tensors of the same length as ys that holds
the initial gradients for each y in ys.
When grad_ys is None,
we fill in a tensor of '1's of the shape of y for each y in ys.
user can provide their own initial 'grad_ys` to compute the
derivatives using a different initial gradient for each y (e.g., if
one wanted to weight the gradient differently for each value in
ys: A Tensor or list of tensors to be differentiated.
xs: A Tensor or list of tensors to be used for differentiation.
grad_ys: Optional. A Tensor or list of tensors the same size as
ys and holding the gradients computed for each y in ys.
name: Optional name to use for grouping all the gradient ops together.
defaults to 'gradients'.
colocate_gradients_with_ops: If True, try colocating gradients with
the corresponding op.
gate_gradients: If True, add a tuple around the gradients returned
for an operations.
This avoids some race conditions.
aggregation_method: Specifies the method used to combine gradient terms.
Accepted values are constants defined in the class AggregationMethod.
A list of sum(dy/dx) for each x in xs.
LookupError: if one of the operations between x and y does not
have a registered gradient function.
ValueError: if the arguments are invalid.
class tf.AggregationMethod
A class listing aggregation methods used to combine gradients.
Computing partial derivatives can require aggregating gradient
contributions. This class lists the various methods that can
be used to combine gradients in the graph:
ADD_N: All of the gradient terms are summed as part of one
operation using the &AddN& op. It has the property that all
gradients must be ready before any aggregation is performed.
DEFAULT: The system-chosen default aggregation method.
tf.stop_gradient(input, name=None)
Stops gradient computation.
When executed in a graph, this op outputs its input tensor as-is.
When building ops to compute gradients, this op prevents the contribution of
its inputs to be taken into account.
Normally, the gradient generator adds ops
to a graph to compute the derivatives of a specified 'loss' by recursively
finding out inputs that contributed to its computation.
If you insert this op
in the graph it inputs are masked from the gradient generator.
They are not
taken into account for computing gradients.
This is useful any time you want to compute a value with TensorFlow but need
to pretend that the value was a constant. Some examples include:
The EM algorithm where the M-step should not involve backpropagation
through the output of the E-step.
Contrastive divergence training of Boltzmann machines where, when
differentiating the energy function, the training must not backpropagate
through the graph that generated the samples from the model.
Adversarial training, where no backprop should happen through the adversarial
example generation process.
input: A Tensor.
name: A name for the operation (optional).
A Tensor. Has the same type as input.
Gradient Clipping
TensorFlow provides several operations that you can use to add clipping
functions to your graph. You can use these functions to perform general data
clipping, but they're particularly useful for handling exploding or vanishing
gradients.
tf.clip_by_value(t, clip_value_min, clip_value_max, name=None)
Clips tensor values to a specified min and max.
Given a tensor t, this operation returns a tensor of the same type and
shape as t with its values clipped to clip_value_min and clip_value_max.
Any values less than clip_value_min are set to clip_value_min. Any values
greater than clip_value_max are set to clip_value_max.
t: A Tensor.
clip_value_min: A 0-D (scalar) Tensor. The minimum value to clip by.
clip_value_max: A 0-D (scalar) Tensor. The maximum value to clip by.
name: A name for the operation (optional).
A clipped Tensor.
tf.clip_by_norm(t, clip_norm, name=None)
Clips tensor values to a maximum L2-norm.
Given a tensor t, and a maximum clip value clip_norm, this operation
normalizes t so that its L2-norm is less than or equal to clip_norm'. Specifically, if the L2-norm is already less than or equal toclip_norm, thentis not modified. If the L2-norm is greater thanclip_norm, then this operation returns a tensor of the same type and shape ast` with its
values set to:
t * clip_norm / l2norm(t)
In this case, the L2-norm of the output tensor is clip_norm.
This operation is typically used to clip gradients before applying them with
an optimizer.
t: A Tensor.
clip_norm: A 0-D (scalar) Tensor & 0. A maximum clipping value.
name: A name for the operation (optional).
A clipped Tensor.
tf.clip_by_average_norm(t, clip_norm, name=None)
Clips tensor values to a maximum average L2-norm.
Given a tensor t, and a maximum clip value clip_norm, this operation
normalizes t so that its average L2-norm is less than or equal to
clip_norm'. Specifically, if the average L2-norm is already less than or equal toclip_norm, thentis not modified. If the average L2-norm is greater thanclip_norm, then this operation returns a tensor of the same type and shape ast` with its values set to:
t * clip_norm / l2norm_avg(t)
In this case, the average L2-norm of the output tensor is clip_norm.
This operation is typically used to clip gradients before applying them with
an optimizer.
t: A Tensor.
clip_norm: A 0-D (scalar) Tensor & 0. A maximum clipping value.
name: A name for the operation (optional).
A clipped Tensor.
tf.clip_by_global_norm(t_list, clip_norm, use_norm=None, name=None)
Clips values of multiple tensors by the ratio of the sum of their norms.
Given a tuple or list of tensors t_list, and a clipping ratio clip_norm,
this operation returns a list of clipped tensors list_clipped
and the global norm (global_norm) of all tensors in t_list. Optionally,
if you've already computed the global norm for t_list, you can specify
the global norm with use_norm.
To perform the clipping, the values t_list[i] are set to:
t_list[i] * clip_norm / max(global_norm, clip_norm)
global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))
If clip_norm & global_norm then the entries in t_list remain as they are,
otherwise they're all shrunk by the global ratio.
Any of the entries of t_list that are of type None are ignored.
This is the correct way to perform gradient clipping (for example, see
R. Pascanu, T. Mikolov, and Y. Bengio, &On the difficulty of training
Recurrent Neural Networks&.
However, it is slower than clip_by_norm() because all the parameters must be
ready before the clipping operation can be performed.
t_list: A tuple or list of mixed Tensors, IndexedSlices, or None.
clip_norm: A 0-D (scalar) Tensor & 0. The clipping ratio.
use_norm: A 0-D (scalar) Tensor of type float (optional). The global
norm to use. If not provided, global_norm() is used to compute the norm.
name: A name for the operation (optional).
list_clipped: A list of Tensors of the same type as list_t.
global_norm: A 0-D (scalar) Tensor representing the global norm.
TypeError: If t_list is not a sequence.
tf.global_norm(t_list, name=None)
Computes the global norm of multiple tensors.
Given a tuple or list of tensors t_list, this operation returns the
global norm of the elements in all tensors in t_list. The global norm is
computed as:
global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))
Any entries in t_list that are of type None are ignored.
t_list: A tuple or list of mixed Tensors, IndexedSlices, or None.
name: A name for the operation (optional).
A 0-D (scalar) Tensor of type float.
TypeError: If t_list is not a sequence.
Decaying the learning rate
tf.train.exponential_decay(learning_rate, global_step, decay_steps, decay_rate, staircase=False, name=None)
Applies exponential decay to the learning rate.
When training a model, it is often recommended to lower the learning rate as
the training progresses.
This function applies an exponential decay function
to a provided initial learning rate.
It requires a global_step value to
compute the decayed learning rate.
You can just pass a TensorFlow variable
that you increment at each training step.
The function returns the decayed learning rate.
It is computed as:
decayed_learning_rate = learning_rate *
decay_rate ^ (global_step / decay_steps)
If the argument staircase is True, then global_step /decay_steps is an
integer division and the decayed learning rate follows a staircase function.
Example: decay every 100000 steps with a base of 0.96:
global_step = tf.Variable(0, trainable=False)
starter_learning_rate = 0.1
learning_rate = tf.exponential_decay(starter_learning_rate, global_step,
.96, staircase=True)
optimizer = tf.GradientDescent(learning_rate)
# Passing global_step to minimize() will increment it at each step.
optimizer.minimize(...my loss..., global_step=global_step)
learning_rate: A scalar float32 or float64 Tensor or a
Python number.
The initial learning rate.
global_step: A scalar int32 or int64 Tensor or a Python number.
Global step to use for the decay computation.
Must not be negative.
decay_steps: A scalar int32 or int64 Tensor or a Python number.
Must be positive.
See the decay computation above.
decay_rate: A scalar float32 or float64 Tensor or a
Python number.
The decay rate.
staircase: Boolean.
It True decay the learning rate at discrete intervals.
name: string.
Optional name of the operation.
Defaults to 'ExponentialDecay'
A scalar Tensor of the same type as learning_rate.
The decayed
learning rate.
Moving Averages
Some training algorithms, such as GradientDescent and Momentum often benefit
from maintaining a moving average of variables during optimization.
moving averages for evaluations often improve results significantly.
class tf.train.ExponentialMovingAverage
Maintains moving averages of variables by employing and exponential decay.
When training a model, it is often beneficial to maintain moving averages of
the trained parameters.
Evaluations that use averaged parameters sometimes
produce significantly better results than the final trained values.
The apply() method adds shadow copies of trained variables and add ops that
maintain a moving average of the trained variables in their shadow copies.
It is used when building the training model.
The ops that maintain moving
averages are typically run after each training step.
The average() and average_name() methods give access to the shadow
variables and their names.
They are useful when building an evaluation
model, or when restoring a model from a checkpoint file.
They help use the
moving averages in place of the last trained values for evaluations.
The moving averages are computed using exponential decay.
You specify the
decay value when creating the ExponentialMovingAverage object.
The shadow
variables are initialized with the same initial values as the trained
variables.
When you run the ops to maintain the moving averages, each
shadow variable is updated with the formula:
shadow_variable -= (1 - decay) * (shadow_variable - variable)
This is mathematically equivalent to the classic formula below, but the use
of an assign_sub op (the "-=" in the formula) allows concurrent lockless
updates to the variables:
shadow_variable = decay * shadow_variable + (1 - decay) * variable
Reasonable values for decay are close to 1.0, typically in the
multiple-nines range: 0.999, 0.9999, etc.
Example usage when creating a training model:
# Create variables.
var0 = tf.Variable(...)
var1 = tf.Variable(...)
# ... use the variables to build a training model...
# Create an op that applies the optimizer.
This is what we usually
# would use as a training op.
opt_op = opt.minimize(my_loss, [var0, var1])
# Create an ExponentialMovingAverage object
ema = tf.train.ExponentialMovingAverage(decay=0.9999)
# Create the shadow variables, and add ops to maintain moving averages
# of var0 and var1.
maintain_averages_op = ema.apply([var0, var1])
# Create an op that will update the moving averages after each training
This is what we will use in place of the usuall trainig op.
with tf.control_dependencies([opt_op]):
training_op = tf.group(maintain_averages_op)
...train the model by running training_op...
There are two ways to use the moving averages for evaluations:
Build a model that uses the shadow variables instead of the variables.
For this, use the average() method which returns the shadow variable
for a given variable.
Build a model normally but load the checkpoint files to evaluate by using
the shadow variable names.
For this use the average_name() method.
information on restoring saved variables.
Example of restoring the shadow variable values:
# Create a Saver that loads variables from their saved shadow values.
shadow_var0_name = ema.average_name(var0)
shadow_var1_name = ema.average_name(var1)
saver = tf.train.Saver({shadow_var0_name: var0, shadow_var1_name: var1})
saver.restore(...checkpoint filename...)
# var0 and var1 now hold the moving average values
tf.train.ExponentialMovingAverage.__init__(decay, num_updates=None, name='ExponentialMovingAverage')
Creates a new ExponentialMovingAverage object.
The Apply() method has to be called to create shadow variables and add
ops to maintain moving averages.
The optional num_updates parameter allows one to tweak the decay rate
dynamically. .
It is typical to pass the count of training steps, usually
kept in a variable that is incremented at each step, in which case the
decay rate is lower at the start of training.
This makes moving averages
move faster.
If passed, the actual decay rate used is:
min(decay, (1 + num_updates) / (10 + num_updates))
decay: Float.
The decay to use.
num_updates: Optional count of number of updates applied to variables.
name: String. Optional prefix name to use for the name of ops added in
tf.train.ExponentialMovingAverage.apply(var_list=None)
Maintains moving averages of variables.
var_list must be a list of Variable or Tensor objects.
This method
creates shadow variables for all elements of var_list.
Shadow variables
for Variable objects are initialized to the variable's initial value.
For Tensor objects, the shadow variables are initialized to 0.
shadow variables are created with trainable=False and added to the
GraphKeys.ALL_VARIABLES collection.
They will be returned by calls to
tf.all_variables().
Returns an op that updates all shadow variables as described above.
Note that apply() can be called multiple times with different lists of
variables.
var_list: A list of Variable or Tensor objects. The variables
and Tensors must be of types float32 or float64.
An Operation that updates the moving averages.
TypeError: If the arguments are not all float32 or float64.
ValueError: If the moving average of one of the variables is already
being computed.
tf.train.ExponentialMovingAverage.average_name(var)
Returns the name of the Variable holding the average for var.
The typical scenario for ExponentialMovingAverage is to compute moving
averages of variables during training, and restore the variables from the
computed moving averages during evaluations.
To restore variables, you have to know the name of the shadow variables.
That name and the original variable can then be passed to a Saver() object
to restore the variable from the moving average value with:
saver = tf.train.Saver({ema.average_name(var): var})
average_name() can be called whether or not apply() has been called.
var: A Variable object.
A string: the name of the variable that will be used or was used
by the ExponentialMovingAverage class to hold the moving average of
tf.train.ExponentialMovingAverage.average(var)
Returns the Variable holding the average of var.
var: A Variable object.
A Variable object or None if the moving average of var
is not maintained..
Coordinator and QueueRunner
for how to use threads and queues.
For documentation on the Queue API,
class tf.train.Coordinator
A coordinator for threads.
This class implements a simple mechanism to coordinate the termination of a
set of threads.
# Create a coordinator.
coord = Coordinator()
# Start a number of threads, passing the coordinator to each of them.
...start thread 1...(coord, ...)
...start thread N...(coord, ...)
# Wait for all the threads to terminate.
coord.join(threads)
Any of the threads can call coord.request_stop() to ask for all the threads
To cooperate with the requests, each thread must check for
coord.should_stop() on a regular basis.
coord.should_stop() returns
True as soon as coord.request_stop() has been called.
A typical thread running with a Coordinator will do something like:
while not coord.should_stop():
...do some work...
Exception handling:
A thread can report an exception to the Coordinator as part of the
should_stop() call.
The exception will be re-raised from the
coord.join() call.
Thread code:
while not coord.should_stop():
...do some work...
except Exception, e:
coord.request_stop(e)
Main code:
coord = Coordinator()
# Start a number of threads, passing the coordinator to each of them.
...start thread 1...(coord, ...)
...start thread N...(coord, ...)
# Wait for all the threads to terminate.
coord.join(threads)
except Exception, e:
...exception that was passed to coord.request_stop()
Grace period for stopping:
After a thread has called coord.request_stop() the other threads have a
fixed time to stop, this is called the 'stop grace period' and defaults to 2
If any of the threads is still alive after the grace period expires
coord.join() raises a RuntimeException reporting the laggards.
coord = Coordinator()
# Start a number of threads, passing the coordinator to each of them.
...start thread 1...(coord, ...)
...start thread N...(coord, ...)
# Wait for all the threads to terminate, give them 10s grace period
coord.join(threads, stop_grace_period_secs=10)
except RuntimeException:
...one of the threads took more than 10s to stop after request_stop()
...was called.
except Exception:
...exception that was passed to coord.request_stop()
tf.train.Coordinator.__init__()
Create a new Coordinator.
tf.train.Coordinator.join(threads, stop_grace_period_secs=120)
Wait for threads to terminate.
Blocks until all 'threads' have terminated or request_stop() is called.
After the threads stop, if an 'exc_info' was passed to request_stop, that
exception is re-reaised.
Grace period handling: When request_stop() is called, threads are given
'stop_grace_period_secs' seconds to terminate.
If any of them is still
alive after that period expires, a RuntimeError is raised.
Note that if
an 'exc_info' was passed to request_stop() then it is raised instead of
that RuntimeError.
threads: List threading.Threads. The started threads to join.
stop_grace_period_secs: Number of seconds given to threads to stop after
request_stop() has been called.
RuntimeError: If any thread is still alive after request_stop()
is called and the grace period expires.
tf.train.Coordinator.request_stop(ex=None)
Request that the threads stop.
After this is called, calls to should_stop() will return True.
ex: Optional Exception, or Python 'exc_info' tuple as returned by
sys.exc_info().
If this is the first call to request_stop() the
corresponding exception is recorded and re-raised from join().
tf.train.Coordinator.should_stop()
Check if stop was requested.
True if a stop was requested.
tf.train.Coordinator.wait_for_stop(timeout=None)
Wait till the Coordinator is told to stop.
timeout: float.
Sleep for up to that many seconds waiting for
should_stop() to become True.
True if the Coordinator is told stop, False if the timeout expired.
class tf.train.QueueRunner
Holds a list of enqueue operations for a queue, each to be run in a thread.
Queues are a convenient TensorFlow mechanism to compute tensors
asynchronously using multiple threads. For example in the canonical 'Input
Reader' setup one set of threads generates
a second set
of threads read records from the files, processes them, and enqueues tensors
a third set of threads dequeues these input records to
construct batches and runs them through training operations.
There are several delicate issues when running multiple threads that way:
closing the queues in sequence as the input is exhausted, correctly catching
and reporting exceptions, etc.
The QueueRunner, combined with the Coordinator, helps handle these issues.
tf.train.QueueRunner.__init__(queue, enqueue_ops)
Create a QueueRunner.
On construction the QueueRunner adds an op to close the queue.
will be run if the enqueue ops raise exceptions.
When you later call the create_threads() method, the QueueRunner will
create one thread for each op in enqueue_ops.
Each thread will run its
enqueue op in parallel with the other threads.
The enqueue ops do not have
to all be the same op, but it is expected that they all enqueue tensors in
queue: A Queue.
enqueue_ops: List of enqueue ops to run in threads later.
tf.train.QueueRunner.create_threads(sess, coord=None, daemon=False, start=False)
Create threads to run the enqueue ops.
This method requires a session in which the graph was launched.
It creates
a list of threads, optionally starting them.
There is one thread for each
op passed in enqueue_ops.
The coord argument is an optional coordinator, that the threads will use
to terminate together and report exceptions.
If a coordinator is given,
this method starts an additional thread to close the queue when the
coordinator requests a stop.
This method may be called again as long as all threads from a previous call
have stopped.
sess: A Session.
coord: Optional Coordinator object for reporting errors and checking
stop conditions.
daemon: Boolean.
If True make the threads daemon threads.
start: Boolean.
If True starts the threads.
If False the
caller must call the start() method of the returned threads.
A list of threads.
RuntimeError: If threads from a previous call to create_threads() are
still running.
tf.train.QueueRunner.exceptions_raised
Exceptions raised but not handled by the QueueRunner threads.
Exceptions raised in queue runner threads are handled in one of two ways
depending on whether or not a Coordinator was passed to
create_threads():
With a Coordinator, exceptions are reported to the coordinator and
forgotten by the QueueRunner.
Without a Coordinator, exceptions are captured by the QueueRunner and
made available in this exceptions_raised property.
A list of Python Exception objects.
The list is empty if no exception
was captured.
(No exceptions are captured when using a Coordinator.)
tf.train.add_queue_runner(qr, collection='queue_runners')
Adds a QueueRunner to a collection in the graph.
When building a complex model that uses many queues it is often difficult to
gather all the queue runners that need to be run.
This convenience function
allows you to add a queue runner to a well known collection in the graph.
The companion method start_queue_runners() can be used to start threads for
all the collected queue runners.
qr: A QueueRunner.
collection: A GraphKey specifying the graph collection to add
the queue runner to.
Defaults to GraphKeys.QUEUE_RUNNERS.
tf.train.start_queue_runners(sess=None, coord=None, daemon=True, start=True, collection='queue_runners')
Starts all queue runners collected in the graph.
This is a companion method to add_queue_runner().
It just starts
threads for all queue runners collected in the graph.
It returns
the list of all threads.
sess: Session used to run the queue ops.
Defaults to the
default session.
coord: Optional Coordinator for coordinating the started threads.
daemon: Whether the threads should be marked as daemons, meaning
they don't block program exit.
start: Set to False to only create the threads, not start them.
collection: A GraphKey specifying the graph collection to
get the queue runners from.
Defaults to GraphKeys.QUEUE_RUNNERS.
A list of threads.
Summary Operations
The following ops output
protocol buffers as serialized string tensors.
You can fetch the output of a summary op in a session, and pass it to
to append it
to an event file.
Event files contain
protos that can contain Summary protos along with the timestamp and
You can then use TensorBoard to visualize the contents of the
event files.
tf.scalar_summary(tags, values, collections=None, name=None)
Outputs a Summary protocol buffer with scalar values.
The input tags and values must have the same shape.
The generated
summary has a summary value for each tag-value pair in tags and values.
tags: A 1-D string Tensor.
Tags for the summaries.
values: A 1-D float32 or float64 Tensor.
Values for the summaries.
collections: Optional list of graph collections keys. The new summary op is
added to these collections. Defaults to [GraphKeys.SUMMARIES].
name: A name for the operation (optional).
A scalar Tensor of type string. The serialized Summary protocol
tf.image_summary(tag, tensor, max_images=None, collections=None, name=None)
Outputs a Summary protocol buffer with images.
The summary has up to max_images summary values containing images. The
images are built from tensor which must be 4-D with shape [batch_size, height, width, channels] and where channels can be:
1: tensor is interpreted as Grayscale.
3: tensor is interpreted as RGB.
4: tensor is interpreted as RGBA.
The images have the same number of channels as the input tensor. Their values
are normalized, one image at a time, to fit in the range [0, 255].
op uses two different normalization algorithms:
If the input values are all positive, they are rescaled so the largest one
If any input value is negative, the values are shifted so input value 0.0
is at 127.
They are then rescaled so that either the smallest value is 0,
or the largest one is 255.
The tag argument is a scalar Tensor of type string.
It is used to
build the tag of the summary values:
If max_images is 1, the summary value tag is 'tag/image'.
If max_images is greater than 1, the summary value tags are
generated sequentially as 'tag/image/0', 'tag/image/1', etc.
tag: A scalar Tensor of type string. Used to build the tag
of the summary values.
tensor: A 4-D float32 Tensor of shape [batch_size, height, width, channels] where channels is 1, 3, or 4.
max_images: Max number of batch elements to generate images for.
collections: Optional list of ops.GraphKeys.
The collections to add the
summary to.
Defaults to [ops.GraphKeys.SUMMARIES]
name: A name for the operation (optional).
A scalar Tensor of type string. The serialized Summary protocol
tf.histogram_summary(tag, values, collections=None, name=None)
Outputs a Summary protocol buffer with a histogram.
The generated
has one summary value containing a histogram for values.
This op reports an OutOfRange error if any value is not finite.
tag: A string Tensor. 0-D.
Tag to use for the summary value.
values: A float32 Tensor. Any shape. Values to use to build the
histogram.
collections: Optional list of graph collections keys. The new summary op is
added to these collections. Defaults to [GraphKeys.SUMMARIES].
name: A name for the operation (optional).
A scalar Tensor of type string. The serialized Summary protocol
tf.nn.zero_fraction(value, name=None)
Returns the fraction of zeros in value.
If value is empty, the result is nan.
This is useful in summaries to measure and report sparsity.
For example,
z = tf.Relu(...)
summ = tf.scalar_summary('sparsity', tf.zero_fraction(z))
value: A tensor of numeric type.
name: A name for the operation (optional).
The fraction of zeros in value, with type float32.
tf.merge_summary(inputs, collections=None, name=None)
Merges summaries.
This op creates a
protocol buffer that contains the union of all the values in the input
summaries.
When the Op is run, it reports an InvalidArgument error if multiple values
in the summaries to merge use the same tag.
inputs: A list of string Tensor objects containing serialized Summary
protocol buffers.
collections: Optional list of graph collections keys. The new summary op is
added to these collections. Defaults to [GraphKeys.SUMMARIES].
name: A name for the operation (optional).
A scalar Tensor of type string. The serialized Summary protocol
buffer resulting from the merging.
tf.merge_all_summaries(key='summaries')
Merges all summaries collected in the default graph.
key: GraphKey used to collect the summaries.
Defaults to
GraphKeys.SUMMARIES.
If no summaries were collected, returns None.
Otherwise returns a scalar
Tensor of typestring containing the serialized Summary protocol
buffer resulting from the merging.
Adding Summaries to Event Files
overview of summaries, event files, and visualization in TensorBoard.
class tf.train.SummaryWriter
Writes Summary protocol buffers to event files.
The SummaryWriter class provides a mechanism to create an event file in a
given directory and add summaries and events to it. The class updates the
file contents asynchronously. This allows a training program to call methods
to add data to the file directly from the training loop, without slowing down
tf.train.SummaryWriter.__init__(logdir, graph_def=None, max_queue=10, flush_secs=120)
Creates a SummaryWriter and an event file.
On construction the summary writer creates a new event file in logdir.
This event file will contain Event protocol buffers constructed when you
call one of the following functions: add_summary(), add_event(), or
add_graph().
If you pass a graph_def protocol buffer to the constructor it is added to
the event file. (This is equivalent to calling add_graph() later).
TensorBoard will pick the graph from the file and display it graphically so
you can interactively explore the graph you built. You will usually pass
the graph from the session in which you launched it:
...create a graph...
# Launch the graph in a session.
sess = tf.Session()
# Create a summary writer, add the 'graph_def' to the event file.
writer = tf.train.SummaryWriter(&some-directory&, sess.graph_def)
The other arguments to the constructor control the asynchronous writes to
the event file:
flush_secs: How often, in seconds, to flush the added summaries
and events to disk.
max_queue: Maximum number of summaries or events pending to be
written to disk before one of the 'add' calls block.
logdir: A string. Directory where event file will be written.
graph_def: A GraphDef protocol buffer.
max_queue: Integer. Size of the queue for pending events and summaries.
flush_secs: Number. How often, in seconds, to flush the
pending events and summaries to disk.
tf.train.SummaryWriter.add_summary(summary, global_step=None)
Adds a Summary protocol buffer to the event file.
This method wraps the provided summary in an Event procotol buffer
and adds it to the event file.
You can pass the output of any summary op, as-is, to this function. You
can also pass a Summary procotol buffer that you manufacture with your
own data. This is commonly done to report evaluation results in event
summary: A Summary protocol buffer, optionally serialized as a string.
global_step: Number. Optional global step value to record with the
tf.train.SummaryWriter.add_event(event)
Adds an event to the event file.
event: An Event protocol buffer.
tf.train.SummaryWriter.add_graph(graph_def, global_step=None)
Adds a GraphDef protocol buffer to the event file.
The graph described by the protocol buffer will be displayed by
TensorBoard. Most users pass a graph in the constructor instead.
graph_def: A GraphDef protocol buffer.
global_step: Number. Optional global step counter to record with the
tf.train.SummaryWriter.flush()
Flushes the event file to disk.
Call this method to make sure that all pending events have been written to
tf.train.SummaryWriter.close()
Flushes the event file to disk and close the file.
Call this method when you do not need the summary writer anymore.
tf.train.summary_iterator(path)
An iterator for reading Event protocol buffers from an event file.
You can use this function to read events written to an event file. It returns
a Python iterator that yields Event protocol buffers.
Example: Print the contents of an events file.
for e in tf.summary_iterator(path to events file):
Example: Print selected summary values.
# This example supposes that the events file contains summaries with a
# summary value tag 'loss'.
These could have been added by calling
# `add_summary()`, passing the output of a scalar summary op created with
# with: `tf.scalar_summary(['loss'], loss_tensor)`.
for e in tf.summary_iterator(path to events file):
for v in e.summary.value:
if v.tag == 'loss':
print v.simple_value
See the protocol buffer definitions of
for more information about their attributes.
path: The path to an event file created by a SummaryWriter.
Event protocol buffers.
Training utilities
tf.train.global_step(sess, global_step_tensor)
Small helper to get the global step.
# Creates a variable to hold the global_step.
global_step_tensor = tf.Variable(10, trainable=False, name='global_step')
# Creates a session.
sess = tf.Session()
# Initializes the variable.
sess.run(global_step_tensor.initializer)
print 'global_step:', tf.train.global_step(sess, global_step_tensor)
global_step: 10
sess: A brain Session object.
global_step_tensor: Tensor or the name of the operation that contains
the global step.
The global step value.
tf.train.write_graph(graph_def, logdir, name, as_text=True)
Writes a graph proto on disk.
The graph is written as a binary proto unless as_text is True.
v = tf.Variable(0, name='my_variable')
sess = tf.Session()
tf.train.write_graph(sess.graph_def, '/tmp/my-model', 'train.pbtxt')
graph_def: A GraphDef protocol buffer.
logdir: Directory where to write the graph.
name: Filename for the graph.
as_text: If True, writes the graph as an ASCII proto.