theano scan optimization

发布时间:2017-6-26 9:50:21 编辑:www.fx114.net 分享查询网我要评论
本篇文章主要介绍了"theano scan optimization ",主要涉及到theano scan optimization 方面的内容,对于theano scan optimization 感兴趣的同学可以参考一下。

selected from Theano Doc

Optimizing Scan performance

Minimizing Scan Usage

performan as much of the computation as possible outside of Scan. This may have the effect increasing memory usage but also reduce the overhead introduce by Scan.

Explicitly passing inputs of the inner function to scan

It's more efficient to explicitly pass parameter as non-sequence inputs.

Examples: Gibbs Sampling

Version One:

import theanofrom theano import tensor as TW = theano.shared(W_values) # we assume that ``W_values`` contains the                            # initial values of your weight matrixbvis = theano.shared(bvis_values)bhid = theano.shared(bhid_values)trng = T.shared_randomstreams.RandomStreams(1234)def OneStep(vsample) :    hmean = T.nnet.sigmoid(theano.dot(vsample, W) + bhid)    hsample = trng.binomial(size=hmean.shape, n=1, p=hmean)    vmean = T.nnet.sigmoid(theano.dot(hsample, W.T) + bvis)    return trng.binomial(size=vsample.shape, n=1, p=vmean,                         dtype=theano.config.floatX)sample = theano.tensor.vector()values, updates = theano.scan(OneStep, outputs_info=sample, n_steps=10)gibbs10 = theano.function([sample], values[-1], updates=updates)

Version Two:

W = theano.shared(W_values) # we assume that ``W_values`` contains the                            # initial values of your weight matrixbvis = theano.shared(bvis_values)bhid = theano.shared(bhid_values)trng = T.shared_randomstreams.RandomStreams(1234)# OneStep, with explicit use of the shared variables (W, bvis, bhid)def OneStep(vsample, W, bvis, bhid):    hmean = T.nnet.sigmoid(theano.dot(vsample, W) + bhid)    hsample = trng.binomial(size=hmean.shape, n=1, p=hmean)    vmean = T.nnet.sigmoid(theano.dot(hsample, W.T) + bvis)    return trng.binomial(size=vsample.shape, n=1, p=vmean,                     dtype=theano.config.floatX)sample = theano.tensor.vector()# The new scan, with the shared variables passed as non_sequencesvalues, updates = theano.scan(fn=OneStep,                              outputs_info=sample,                              non_sequences=[W, bvis, bhid],                              n_steps=10)gibbs10 = theano.function([sample], values[-1], updates=updates)

Deactivating garbage collecting in Scan

Deactivating garbage collecting in Scan can allow it to reuse memory between executins instead of always having to allocate new memory. Scan reuses memory between iterations of the same execution but frees the memory after the last iteration.
config.scan.allow_gc=False

Graph Optimizations

There are patterns that Theano can't optimize. the LSTM tutorial provides an example of optimization that theano can't perform. Instead of performing many matrix multiplications between matrix \(x_t\) and each of the shared msatrices \(W_i,W_c,W_f\) and \(W_o\), the matrixes \(W_{*}\) are merged into a single shared \(W\) and the graph performans a single larger matrix multiplication between \(W\) and \(x_t\). The resulting matrix is then sliced to obtain the results of that the small individial matrix multiplications by a single larger one and thus improves performance at the cost of a potentially higher memory usage.

上一篇:Ubuntu 16.04 64位 搭建 node.js NodeJS 环境
下一篇:最后想的还是你啊

相关文章

相关评论

本站评论功能暂时取消,后续此功能例行通知。

一、不得利用本站危害国家安全、泄露国家秘密,不得侵犯国家社会集体的和公民的合法权益,不得利用本站制作、复制和传播不法有害信息!

二、互相尊重,对自己的言论和行为负责。

最近更新