This is very promising since dilation is built into the convolutional operator in most frameworks, including cuDNN. Lightvector updated his blog with some more result, of especial interest is the fact that adding dilation has a similar effect as adding chain pooling but since you only apply dilation over some of the channel local shape information is not lost. To implement this we would have to do a custom CUDA kernel, which could probably be done relatively easily using some flood filling technique: But it would probably help in most situations. This is only one layer of chain pooling so we expect it to have issues identifying problems with loosely connected groups or certain types of false eyes. To help with the performance impact, one could limit the chain pooling to the final convolutional layer in the value head with input x: However Chain Pooling might greatly help with this issue. It has issues identifying whether large dragons are alive or dead.įollowing the logic posted in lightvector's repository one could reasonably expect Parametric ReLUs and Global Pooling Properties to have little effect on this problem.I would be more interesting in seeing if this improves the value head, in which you can observe the following issue consistently: I suspect this is too expensive to do in practice (no good support for it in cuDNN), but it is a very interesting idea.Īll these enhancements are for the policy network, which is generally doing pretty well even without these tweaks. Concatenate y₁ and yₛ channel-wise into y₁ₛ (so y₁ₛ has shape ).Concatenate x and yₛ channel-wise into yₓₛ (so yₓₛ has shape ).Given input x compute the the chain pooling of each channel and store the result in yₛ.This way each residual block would benefit from both the pooling and the local shape: Might be interesting to do a dense block approach where each residual blocks becomes a DenseNet. I used to experiment with other activation functions, like selu, but they were not as good.Ĭhain Pooling This probably gives worse results since the pooling destroy the local shape information. Otherwise I've had similar experiences a long time ago when I was playing around with neural network architectures for Go where PReLU is just better. Parametric ReLUs Unfortunately there is no support for PReLU in cuDNN so we cannot do this. Need to experiment a bit with this to tell. Unclear if we want to add them early, before each residual block, or only in the policy and value head. Global Pooled Properties This should be easy to implement and makes a lot of sense. Initial thoughts on the concepts without any further research to back it up: Check out some of ideas mentioned here to enhance the neural
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |