Clustering and graphing similarities of sentence subjects
$begingroup$
I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject.
Ex. "I love reading math books" Subjects for the above sentence = [Math: 10, Reading: 14, Romance: 1, Leisure: 4]
Now I would like to create a graph of nodes, where each node is a sentence and place these nodes at the origin in a 2D plane. Each subject forms the circumference of a circle surrounding the nodes. The "closeness" of the nodes to their respective subjects is represented by their positions in the 2D plane. I figured I could do this by taking each score for a subject of a sentence and apply it as a vector. Then add all the vectors, for all the subjects, together to settle on a final position
The resulting plane could look like this
The idea here is that we can now bind each sentence node to each other with edges to create a proximity graph. Using a Gabriel graph, we can only bind the closest nodes together
The entire goal here is to construct a script from a bunch of sentences where we can flow down the script going from subject to subject without too much discontinuity.
As you can see I already have a method that seems to make sense. I was wondering however if there was already a set of methods for doing this kind of thing in data science. I was looking into spectral clustering, and similarity measure. I even looked into bioinformatics and found Needleman–Wunsch algorithm and Smith–Waterman algorithm. But I'm not knowledgable in data science or bioinformatics. Can I get some directions as to where I should be headed to solve this kind of problem. Is there already an established set of tools and methods for accomplishing it?
clustering similarity natural-language-process
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject.
Ex. "I love reading math books" Subjects for the above sentence = [Math: 10, Reading: 14, Romance: 1, Leisure: 4]
Now I would like to create a graph of nodes, where each node is a sentence and place these nodes at the origin in a 2D plane. Each subject forms the circumference of a circle surrounding the nodes. The "closeness" of the nodes to their respective subjects is represented by their positions in the 2D plane. I figured I could do this by taking each score for a subject of a sentence and apply it as a vector. Then add all the vectors, for all the subjects, together to settle on a final position
The resulting plane could look like this
The idea here is that we can now bind each sentence node to each other with edges to create a proximity graph. Using a Gabriel graph, we can only bind the closest nodes together
The entire goal here is to construct a script from a bunch of sentences where we can flow down the script going from subject to subject without too much discontinuity.
As you can see I already have a method that seems to make sense. I was wondering however if there was already a set of methods for doing this kind of thing in data science. I was looking into spectral clustering, and similarity measure. I even looked into bioinformatics and found Needleman–Wunsch algorithm and Smith–Waterman algorithm. But I'm not knowledgable in data science or bioinformatics. Can I get some directions as to where I should be headed to solve this kind of problem. Is there already an established set of tools and methods for accomplishing it?
clustering similarity natural-language-process
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago
add a comment |
$begingroup$
I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject.
Ex. "I love reading math books" Subjects for the above sentence = [Math: 10, Reading: 14, Romance: 1, Leisure: 4]
Now I would like to create a graph of nodes, where each node is a sentence and place these nodes at the origin in a 2D plane. Each subject forms the circumference of a circle surrounding the nodes. The "closeness" of the nodes to their respective subjects is represented by their positions in the 2D plane. I figured I could do this by taking each score for a subject of a sentence and apply it as a vector. Then add all the vectors, for all the subjects, together to settle on a final position
The resulting plane could look like this
The idea here is that we can now bind each sentence node to each other with edges to create a proximity graph. Using a Gabriel graph, we can only bind the closest nodes together
The entire goal here is to construct a script from a bunch of sentences where we can flow down the script going from subject to subject without too much discontinuity.
As you can see I already have a method that seems to make sense. I was wondering however if there was already a set of methods for doing this kind of thing in data science. I was looking into spectral clustering, and similarity measure. I even looked into bioinformatics and found Needleman–Wunsch algorithm and Smith–Waterman algorithm. But I'm not knowledgable in data science or bioinformatics. Can I get some directions as to where I should be headed to solve this kind of problem. Is there already an established set of tools and methods for accomplishing it?
clustering similarity natural-language-process
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have a bunch of sentences. Each sentence is given a weight of how "close" it is to a particular subject.
Ex. "I love reading math books" Subjects for the above sentence = [Math: 10, Reading: 14, Romance: 1, Leisure: 4]
Now I would like to create a graph of nodes, where each node is a sentence and place these nodes at the origin in a 2D plane. Each subject forms the circumference of a circle surrounding the nodes. The "closeness" of the nodes to their respective subjects is represented by their positions in the 2D plane. I figured I could do this by taking each score for a subject of a sentence and apply it as a vector. Then add all the vectors, for all the subjects, together to settle on a final position
The resulting plane could look like this
The idea here is that we can now bind each sentence node to each other with edges to create a proximity graph. Using a Gabriel graph, we can only bind the closest nodes together
The entire goal here is to construct a script from a bunch of sentences where we can flow down the script going from subject to subject without too much discontinuity.
As you can see I already have a method that seems to make sense. I was wondering however if there was already a set of methods for doing this kind of thing in data science. I was looking into spectral clustering, and similarity measure. I even looked into bioinformatics and found Needleman–Wunsch algorithm and Smith–Waterman algorithm. But I'm not knowledgable in data science or bioinformatics. Can I get some directions as to where I should be headed to solve this kind of problem. Is there already an established set of tools and methods for accomplishing it?
clustering similarity natural-language-process
clustering similarity natural-language-process
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
SephSeph
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
SephSeph
1061
asked 2 days ago
SephSeph
1061
1061
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Seph is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago
add a comment |
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Seph is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46795%2fclustering-and-graphing-similarities-of-sentence-subjects%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Seph is a new contributor. Be nice, and check out our Code of Conduct.
Seph is a new contributor. Be nice, and check out our Code of Conduct.
Seph is a new contributor. Be nice, and check out our Code of Conduct.
Seph is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46795%2fclustering-and-graphing-similarities-of-sentence-subjects%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Do you already have these similarity measures? I would stay away from the bioinformatics algorithms you linked. They would give very high similarity between the words math and bath, they don't know any context. In your example bath and math should probably be far apart.
$endgroup$
– Pallie
yesterday
$begingroup$
No, the weights for the subjects of a node are going to be handwritten in. I don't have any data to work with, yet. The idea is to get a model first and tune it. Then collect data and use that.
$endgroup$
– Seph
yesterday
$begingroup$
So maybe it would be possible to calculate an n nearest neighbours for each point and then draw this data as a graph with force directed layout?
$endgroup$
– Pallie
14 hours ago