Checkers playing Neural Network evolved with Genetic Algorithm becomes too sensitive to input data changes
$begingroup$
I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:
Neural Network Structure:
Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.
Genetic Algorithm
30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:
$m' = m * e^{r*0.9} $
$w' = w * m' * r$
Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.
machine-learning neural-network overfitting genetic-algorithms
asked Apr 11 '17 at 20:10
DanielDaniel
211
$endgroup$
bumped to the homepage by Community♦ 13 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
|
show 7 more comments
$begingroup$
I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:
Neural Network Structure:
Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.
Genetic Algorithm
30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:
$m' = m * e^{r*0.9} $
$w' = w * m' * r$
Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.
machine-learning neural-network overfitting genetic-algorithms
asked Apr 11 '17 at 20:10
DanielDaniel
211
$endgroup$
bumped to the homepage by Community♦ 13 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26
|
show 7 more comments
$begingroup$
I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:
Neural Network Structure:
Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.
Genetic Algorithm
30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:
$m' = m * e^{r*0.9} $
$w' = w * m' * r$
Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.
machine-learning neural-network overfitting genetic-algorithms
asked Apr 11 '17 at 20:10
DanielDaniel
211
$endgroup$
I recently embarked on a very ambitious project and I have to say it has turned out a lot better than I expected, I succeeded in coding from scratch a neural network that plays checkers at a very acceptable level.
I have one problem though which prevents me from having a stronger AI, I guess some people would call it overfitting, I'm not sure if the term is correct in this situation.
The main issue is that after a certain amout of generations the weights become so great that the smallest change in the state of the board changes the evaluation of it drastically, effectively limiting any kind of nuanced analysis.
A bit more information:
Neural Network Structure:
Input layer is a series of indipendent layers which take as input a section of the board (1, 0.5, -0.5 and -1 are the inputs taken from the board).
I am using 2 hidden layers of approximately 30 and 10 nodes each, the output node is an evaluation of the board.
Genetic Algorithm
30 Neural Networks are generated randomly with weights from -0.2 to 0.2, they play 20 games against random opponents and they are rated accordingly (-2 points for a loss, -1 for a draw [120 moves with no winner] and 1 point for a win).
The 15 best are kept and copy/pasted on top of the 15 worst ones, then each weight of the new networks are mutated using the following equation:
$m' = m * e^{r*0.9} $
$w' = w * m' * r$
Where $m$ is the mutation parameter of that specific weight which starts off at 0.05, $r$ is a random standard Gaussian number and $w$ is the weight.
I hope someone here can help me figure out how to prevent the weights from going haywire after the 40th or so generation.
machine-learning neural-network overfitting genetic-algorithms
machine-learning neural-network overfitting genetic-algorithms
asked Apr 11 '17 at 20:10
DanielDaniel
211
asked Apr 11 '17 at 20:10
DanielDaniel
211
asked Apr 11 '17 at 20:10
DanielDaniel
211
asked Apr 11 '17 at 20:10
DanielDaniel
211
asked Apr 11 '17 at 20:10
DanielDaniel
211
211
bumped to the homepage by Community♦ 13 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 13 hours ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26
|
show 7 more comments
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26
|
show 7 more comments
1 Answer
1
active
oldest
votes
$begingroup$
Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.
First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?
That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f18278%2fcheckers-playing-neural-network-evolved-with-genetic-algorithm-becomes-too-sensi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.
First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?
That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
$endgroup$
add a comment |
$begingroup$
Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.
First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?
That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
$endgroup$
add a comment |
$begingroup$
Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.
First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?
That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
$endgroup$
Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.
First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?
That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
answered Dec 20 '18 at 1:11
kbrosekbrose
814112
814112
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f18278%2fcheckers-playing-neural-network-evolved-with-genetic-algorithm-becomes-too-sensi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Try using activation functions (sigmoid/tanh) after your hidden layers or gradient clipping; these should limit the magnitude of your weight updates.
$endgroup$
– liangjy
Apr 11 '17 at 21:09
$begingroup$
@liangjy: There are no gradient calculations involved in this model's training. The weights are mutated randomly instead. For some reason, higher weight individuals are winning initial games, but it is not really clear why to me.
$endgroup$
– Neil Slater
Apr 11 '17 at 21:19
$begingroup$
@liangjy I am using tanh as an activation function for each neuron.
$endgroup$
– Daniel
Apr 11 '17 at 23:43
$begingroup$
The algorithm looks as much like particle swarm optimisation as a GA, due to amount of movement in the weights and lack of cross-over between selected parents to create offspring. Given the simplicity of the network and search algorithm, I would not expect much learning to happen at all. How are you measuring results such that you observe the agent "plays checkers at a very acceptable level"?
$endgroup$
– Neil Slater
Apr 12 '17 at 7:49
$begingroup$
@NeilSlater The testing of the actual strength of the network has been one of the issues, thankfully I've had a lot of people willing to test it out, none of them great checkers players by any means but they all reported having a lot of trouble beating the AI. Myself included, I have probably won about 5% of the games I've played against one of the networks in the 30-40th generation, but then again I'm terrible at checkers. I also tested it against some online checkers engine and it has beaten all of them. Against really advanced engines like cakeMD it struggles a lot in the end game.
$endgroup$
– Daniel
Apr 12 '17 at 10:26