Preparing data for training, where to find suitable text
$begingroup$
I've been tasked with producing training data for a supervised ML system using Tensorflow. However, I'm having a hard time generating the training data.
The system's ultimate goal, is to be able to categorise documents, for example this document is about money, or this is about science, etc. So if we'd like the system to recognise documents relating to money, I need a lot of text examples that relate to money, but where do I find those?
We'd also like the system to be able to recognise the various slang terms for money, and we've collected about 300 different slang terms for money. So that also compounds the question, of how do I put these slang terms in there?
I've taken a look at various newsgroups around the topic of money/finance, and there is some text in there, but there is also a huge amount of spam and other garbage, so it's going to take a fair while to separate out any actual useful text.
I had another idea to just randomly generate phrases that contained a random assortment of verbs and money terms but I've got a feeling that's going to produce a poorly trained model.
Does anyone have any suggestions? I'm just using money here, but ultimately we'd like to cover as many categories as is possible.
machine-learning tensorflow
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I've been tasked with producing training data for a supervised ML system using Tensorflow. However, I'm having a hard time generating the training data.
The system's ultimate goal, is to be able to categorise documents, for example this document is about money, or this is about science, etc. So if we'd like the system to recognise documents relating to money, I need a lot of text examples that relate to money, but where do I find those?
We'd also like the system to be able to recognise the various slang terms for money, and we've collected about 300 different slang terms for money. So that also compounds the question, of how do I put these slang terms in there?
I've taken a look at various newsgroups around the topic of money/finance, and there is some text in there, but there is also a huge amount of spam and other garbage, so it's going to take a fair while to separate out any actual useful text.
I had another idea to just randomly generate phrases that contained a random assortment of verbs and money terms but I've got a feeling that's going to produce a poorly trained model.
Does anyone have any suggestions? I'm just using money here, but ultimately we'd like to cover as many categories as is possible.
machine-learning tensorflow
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I've been tasked with producing training data for a supervised ML system using Tensorflow. However, I'm having a hard time generating the training data.
The system's ultimate goal, is to be able to categorise documents, for example this document is about money, or this is about science, etc. So if we'd like the system to recognise documents relating to money, I need a lot of text examples that relate to money, but where do I find those?
We'd also like the system to be able to recognise the various slang terms for money, and we've collected about 300 different slang terms for money. So that also compounds the question, of how do I put these slang terms in there?
I've taken a look at various newsgroups around the topic of money/finance, and there is some text in there, but there is also a huge amount of spam and other garbage, so it's going to take a fair while to separate out any actual useful text.
I had another idea to just randomly generate phrases that contained a random assortment of verbs and money terms but I've got a feeling that's going to produce a poorly trained model.
Does anyone have any suggestions? I'm just using money here, but ultimately we'd like to cover as many categories as is possible.
machine-learning tensorflow
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I've been tasked with producing training data for a supervised ML system using Tensorflow. However, I'm having a hard time generating the training data.
The system's ultimate goal, is to be able to categorise documents, for example this document is about money, or this is about science, etc. So if we'd like the system to recognise documents relating to money, I need a lot of text examples that relate to money, but where do I find those?
We'd also like the system to be able to recognise the various slang terms for money, and we've collected about 300 different slang terms for money. So that also compounds the question, of how do I put these slang terms in there?
I've taken a look at various newsgroups around the topic of money/finance, and there is some text in there, but there is also a huge amount of spam and other garbage, so it's going to take a fair while to separate out any actual useful text.
I had another idea to just randomly generate phrases that contained a random assortment of verbs and money terms but I've got a feeling that's going to produce a poorly trained model.
Does anyone have any suggestions? I'm just using money here, but ultimately we'd like to cover as many categories as is possible.
machine-learning tensorflow
machine-learning tensorflow
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
TonyTony
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
TonyTony
101
asked yesterday
TonyTony
101
101
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Tony is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Tony is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47538%2fpreparing-data-for-training-where-to-find-suitable-text%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Tony is a new contributor. Be nice, and check out our Code of Conduct.
Tony is a new contributor. Be nice, and check out our Code of Conduct.
Tony is a new contributor. Be nice, and check out our Code of Conduct.
Tony is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47538%2fpreparing-data-for-training-where-to-find-suitable-text%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
