Date Extraction in Python
$begingroup$
I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:
- 2019-02-20
- 20.02.2019 ("German format")
- 02/2019 ("February 2019")
- "tomorrow" (datetime.timedelta(days=1))
- "yesterday" (datetime.timedelta(days=-1))
Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?
I'm interested in Information Extraction from German and English texts. Mainly German, though.
Constraints
I don't have the complete dataset by now, but I have some idea about it:
- 10 years of interesting dates which could be in the dataset
- I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
- I have millions of documents. Every document has around 20 sentences, I guess.
- Most of the data is in German
python text-mining
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
$endgroup$
add a comment |
$begingroup$
I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:
- 2019-02-20
- 20.02.2019 ("German format")
- 02/2019 ("February 2019")
- "tomorrow" (datetime.timedelta(days=1))
- "yesterday" (datetime.timedelta(days=-1))
Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?
I'm interested in Information Extraction from German and English texts. Mainly German, though.
Constraints
I don't have the complete dataset by now, but I have some idea about it:
- 10 years of interesting dates which could be in the dataset
- I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
- I have millions of documents. Every document has around 20 sentences, I guess.
- Most of the data is in German
python text-mining
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
$endgroup$
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday
add a comment |
$begingroup$
I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:
- 2019-02-20
- 20.02.2019 ("German format")
- 02/2019 ("February 2019")
- "tomorrow" (datetime.timedelta(days=1))
- "yesterday" (datetime.timedelta(days=-1))
Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?
I'm interested in Information Extraction from German and English texts. Mainly German, though.
Constraints
I don't have the complete dataset by now, but I have some idea about it:
- 10 years of interesting dates which could be in the dataset
- I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
- I have millions of documents. Every document has around 20 sentences, I guess.
- Most of the data is in German
python text-mining
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
$endgroup$
I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes:
- 2019-02-20
- 20.02.2019 ("German format")
- 02/2019 ("February 2019")
- "tomorrow" (datetime.timedelta(days=1))
- "yesterday" (datetime.timedelta(days=-1))
Is there a Python package / library which offers this already or do i have to write all of those regexes/logic myself?
I'm interested in Information Extraction from German and English texts. Mainly German, though.
Constraints
I don't have the complete dataset by now, but I have some idea about it:
- 10 years of interesting dates which could be in the dataset
- I guess the interesting date types are: (1) 28.02.2019, (2) relative ones like "3 days ago" (3) 28/02/2019, (4) 02/28/2019 (5) 2019-02-28 (6) 2019/02/28 (7) 2019/28/02 (8) 28.2.2019 (9) 28.2 (10) ... -- all of which could have spaces in various places
- I have millions of documents. Every document has around 20 sentences, I guess.
- Most of the data is in German
python text-mining
python text-mining
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
edited 10 hours ago
Martin Thoma
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
asked yesterday
Martin ThomaMartin Thoma
6,2681353129
6,2681353129
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday
add a comment |
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Stanford CoreNLP has a very good implementation of NER for date/time.
https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process)
Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime). List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html
answered yesterday
Shamit VermaShamit Verma
57516
$endgroup$
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45854%2fdate-extraction-in-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Stanford CoreNLP has a very good implementation of NER for date/time.
https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process)
Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime). List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html
answered yesterday
Shamit VermaShamit Verma
57516
$endgroup$
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
add a comment |
$begingroup$
Stanford CoreNLP has a very good implementation of NER for date/time.
https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process)
Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime). List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html
answered yesterday
Shamit VermaShamit Verma
57516
$endgroup$
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
add a comment |
$begingroup$
Stanford CoreNLP has a very good implementation of NER for date/time.
https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process)
Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime). List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html
answered yesterday
Shamit VermaShamit Verma
57516
$endgroup$
Stanford CoreNLP has a very good implementation of NER for date/time.
https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process)
Though this is written in Java, there are quite a few Python wrappers for this library (Such as : https://github.com/FraBle/python-sutime). List of such libraries : https://stanfordnlp.github.io/CoreNLP/other-languages.html
answered yesterday
Shamit VermaShamit Verma
57516
answered yesterday
Shamit VermaShamit Verma
57516
answered yesterday
Shamit VermaShamit Verma
57516
answered yesterday
Shamit VermaShamit Verma
57516
57516
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
add a comment |
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
At least the web interface only offers English.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
$begingroup$
These Languages are built-in : github.com/stanfordnlp/CoreNLP/tree/master/src/edu/stanford/nlp/… . You can look for German rules (if someone has created it) OR estimate work required for writing these based on number of rules in other language files.
$endgroup$
– Shamit Verma
yesterday
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45854%2fdate-extraction-in-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
I had looked into this about 6 months ago and could not find anything that works out of the box for both English and German. What seemed promising was using some fuzzy matching, given you can make some half-descent assumptions about the possible formats, as in your examples. The same would go for a regex solution, I suppose. You could combine the approaches even.
$endgroup$
– n1k31t4
yesterday
$begingroup$
fuzzywuzzy up to my knowledge is a bad match, as it essentially uses the Levensthein distance. For dates I need regexes ... Although I could list all reasonable dates (10 years = 3653 elements) and all formats I'm interested in (maybe 10), doing fuzzy matching for roughly 36'530 elements over millions of documents is not feasible.
$endgroup$
– Martin Thoma
yesterday
$begingroup$
I agree it isn't optimal, but using heuristic parameters could work fairly well (it did for me). You could brute force it as you suggest - you hadn't mentioned millions of documents. Being more specific; it is really the number of tokens which is important (how big is a document?). Perhaps you could update your question to include those additional computation considerations/constraints.
$endgroup$
– n1k31t4
yesterday