Time-efficient matrix elements grouping and summing
$begingroup$
I'm interested in finding the quickest way of grouping the elements of a large matrix in sub-groups of NxM elements and them summing them together.
To be completely clear, I'm actually not interested in the "regrouped" matrice, but only in the final results where the elements are summed.
I'll show you an example below:
Say I have the following matrix 9x8:
test = Array[Subscript[a, ##] &, {8, 9}]
I regroup it in sub-matrices NxM, in this example 3x2:
subtest = Partition[test, {2, 3}]
and then I sum them together (as suggested in the comment by @:
out = MapAt[Total[#, -1] &, subtest, {All, All}];
I could use other ways of summing the subgroups, as for example:
out = Total /@ Flatten /@ # & /@ subtest;
Or using two nested tables, or for loops, etc.
My question is what is the fastest method for doing this? I need to do it on a 48k x 48k matrix, so I'd really need something reasonably quick.
Should I look into compiling nested for loops in C (not sure, I haven't ever tried)?
Something worth mentioning is that the entries of the matrix are all integers larger or equal to 0.
I'll add a (redundant) example with numeric values, thac can be however modified to larger matrices:
test = RandomInteger[1, {8, 9}];
{{0, 0, 1, 0, 1, 1, 0, 0, 0}, {1, 1, 0, 1, 0, 0, 1, 0, 0}, {0, 0, 1,
1, 1, 1, 0, 0, 0}, {0, 0, 1, 0, 0, 1, 1, 0, 1}, {0, 0, 1, 1, 0, 0,
1, 1, 1}, {1, 0, 0, 1, 0, 1, 1, 1, 1}, {1, 0, 1, 1, 1, 0, 1, 1,
0}, {0, 0, 1, 1, 1, 0, 1, 0, 1}}
m = 3
n = 2
out = MapAt[Total[#, -1] &, Partition[test, {n, m}], {All, All}]
{{3, 3, 1}, {2, 4, 2}, {2, 3, 6}, {3, 4, 4}}
matrix
$endgroup$
|
show 3 more comments
$begingroup$
I'm interested in finding the quickest way of grouping the elements of a large matrix in sub-groups of NxM elements and them summing them together.
To be completely clear, I'm actually not interested in the "regrouped" matrice, but only in the final results where the elements are summed.
I'll show you an example below:
Say I have the following matrix 9x8:
test = Array[Subscript[a, ##] &, {8, 9}]
I regroup it in sub-matrices NxM, in this example 3x2:
subtest = Partition[test, {2, 3}]
and then I sum them together (as suggested in the comment by @:
out = MapAt[Total[#, -1] &, subtest, {All, All}];
I could use other ways of summing the subgroups, as for example:
out = Total /@ Flatten /@ # & /@ subtest;
Or using two nested tables, or for loops, etc.
My question is what is the fastest method for doing this? I need to do it on a 48k x 48k matrix, so I'd really need something reasonably quick.
Should I look into compiling nested for loops in C (not sure, I haven't ever tried)?
Something worth mentioning is that the entries of the matrix are all integers larger or equal to 0.
I'll add a (redundant) example with numeric values, thac can be however modified to larger matrices:
test = RandomInteger[1, {8, 9}];
{{0, 0, 1, 0, 1, 1, 0, 0, 0}, {1, 1, 0, 1, 0, 0, 1, 0, 0}, {0, 0, 1,
1, 1, 1, 0, 0, 0}, {0, 0, 1, 0, 0, 1, 1, 0, 1}, {0, 0, 1, 1, 0, 0,
1, 1, 1}, {1, 0, 0, 1, 0, 1, 1, 1, 1}, {1, 0, 1, 1, 1, 0, 1, 1,
0}, {0, 0, 1, 1, 1, 0, 1, 0, 1}}
m = 3
n = 2
out = MapAt[Total[#, -1] &, Partition[test, {n, m}], {All, All}]
{{3, 3, 1}, {2, 4, 2}, {2, 3, 6}, {3, 4, 4}}
matrix
$endgroup$
3
$begingroup$
No need to map if you use the second argument ofTotal:Total[Partition[test, {2, 3}], {3, 4}].
$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
1
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated byRandomInteger[1, {n, n}]are packed which can be checked withDeveloper`PackedArrayQ[test].
$endgroup$
– Henrik Schumacher
3 hours ago
|
show 3 more comments
$begingroup$
I'm interested in finding the quickest way of grouping the elements of a large matrix in sub-groups of NxM elements and them summing them together.
To be completely clear, I'm actually not interested in the "regrouped" matrice, but only in the final results where the elements are summed.
I'll show you an example below:
Say I have the following matrix 9x8:
test = Array[Subscript[a, ##] &, {8, 9}]
I regroup it in sub-matrices NxM, in this example 3x2:
subtest = Partition[test, {2, 3}]
and then I sum them together (as suggested in the comment by @:
out = MapAt[Total[#, -1] &, subtest, {All, All}];
I could use other ways of summing the subgroups, as for example:
out = Total /@ Flatten /@ # & /@ subtest;
Or using two nested tables, or for loops, etc.
My question is what is the fastest method for doing this? I need to do it on a 48k x 48k matrix, so I'd really need something reasonably quick.
Should I look into compiling nested for loops in C (not sure, I haven't ever tried)?
Something worth mentioning is that the entries of the matrix are all integers larger or equal to 0.
I'll add a (redundant) example with numeric values, thac can be however modified to larger matrices:
test = RandomInteger[1, {8, 9}];
{{0, 0, 1, 0, 1, 1, 0, 0, 0}, {1, 1, 0, 1, 0, 0, 1, 0, 0}, {0, 0, 1,
1, 1, 1, 0, 0, 0}, {0, 0, 1, 0, 0, 1, 1, 0, 1}, {0, 0, 1, 1, 0, 0,
1, 1, 1}, {1, 0, 0, 1, 0, 1, 1, 1, 1}, {1, 0, 1, 1, 1, 0, 1, 1,
0}, {0, 0, 1, 1, 1, 0, 1, 0, 1}}
m = 3
n = 2
out = MapAt[Total[#, -1] &, Partition[test, {n, m}], {All, All}]
{{3, 3, 1}, {2, 4, 2}, {2, 3, 6}, {3, 4, 4}}
matrix
$endgroup$
I'm interested in finding the quickest way of grouping the elements of a large matrix in sub-groups of NxM elements and them summing them together.
To be completely clear, I'm actually not interested in the "regrouped" matrice, but only in the final results where the elements are summed.
I'll show you an example below:
Say I have the following matrix 9x8:
test = Array[Subscript[a, ##] &, {8, 9}]
I regroup it in sub-matrices NxM, in this example 3x2:
subtest = Partition[test, {2, 3}]
and then I sum them together (as suggested in the comment by @:
out = MapAt[Total[#, -1] &, subtest, {All, All}];
I could use other ways of summing the subgroups, as for example:
out = Total /@ Flatten /@ # & /@ subtest;
Or using two nested tables, or for loops, etc.
My question is what is the fastest method for doing this? I need to do it on a 48k x 48k matrix, so I'd really need something reasonably quick.
Should I look into compiling nested for loops in C (not sure, I haven't ever tried)?
Something worth mentioning is that the entries of the matrix are all integers larger or equal to 0.
I'll add a (redundant) example with numeric values, thac can be however modified to larger matrices:
test = RandomInteger[1, {8, 9}];
{{0, 0, 1, 0, 1, 1, 0, 0, 0}, {1, 1, 0, 1, 0, 0, 1, 0, 0}, {0, 0, 1,
1, 1, 1, 0, 0, 0}, {0, 0, 1, 0, 0, 1, 1, 0, 1}, {0, 0, 1, 1, 0, 0,
1, 1, 1}, {1, 0, 0, 1, 0, 1, 1, 1, 1}, {1, 0, 1, 1, 1, 0, 1, 1,
0}, {0, 0, 1, 1, 1, 0, 1, 0, 1}}
m = 3
n = 2
out = MapAt[Total[#, -1] &, Partition[test, {n, m}], {All, All}]
{{3, 3, 1}, {2, 4, 2}, {2, 3, 6}, {3, 4, 4}}
matrix
matrix
edited 3 hours ago
Fraccalo
asked 5 hours ago
FraccaloFraccalo
2,481518
2,481518
3
$begingroup$
No need to map if you use the second argument ofTotal:Total[Partition[test, {2, 3}], {3, 4}].
$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
1
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated byRandomInteger[1, {n, n}]are packed which can be checked withDeveloper`PackedArrayQ[test].
$endgroup$
– Henrik Schumacher
3 hours ago
|
show 3 more comments
3
$begingroup$
No need to map if you use the second argument ofTotal:Total[Partition[test, {2, 3}], {3, 4}].
$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
1
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated byRandomInteger[1, {n, n}]are packed which can be checked withDeveloper`PackedArrayQ[test].
$endgroup$
– Henrik Schumacher
3 hours ago
3
3
$begingroup$
No need to map if you use the second argument of
Total: Total[Partition[test, {2, 3}], {3, 4}].$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
No need to map if you use the second argument of
Total: Total[Partition[test, {2, 3}], {3, 4}].$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
1
1
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated by
RandomInteger[1, {n, n}] are packed which can be checked with Developer`PackedArrayQ[test].$endgroup$
– Henrik Schumacher
3 hours ago
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated by
RandomInteger[1, {n, n}] are packed which can be checked with Developer`PackedArrayQ[test].$endgroup$
– Henrik Schumacher
3 hours ago
|
show 3 more comments
1 Answer
1
active
oldest
votes
$begingroup$
Partition[test, {2,3}] is quite slow in this case because it has to rearrange the elements in the data vector that represents the entries of a packed array in the backend:
Flatten[test] == Flatten[Partition[test, {2, 3}]]
False
Using Span (;;) as follows employs 6 monotonically increasing read operations; in this specific case, these operations are faster than using Partition:
n = 24000;
test = RandomInteger[1, {n, n}];
a = Total[Partition[test, {2, 3}], {3, 4}]; //
AbsoluteTiming // First
b = Sum[test[[i ;; ;; 2, j ;; ;; 3]], {i, 1, 2}, {j, 1, 3}]; //
AbsoluteTiming // First
a == b
245.89
117.943
True
However, this performance advantage seems to decay when the matrix test becomes bigger (so swapping is required). E.g., for $n = 4800$, method b is aboutten times faster thana`, but for $n = 24000$, it's only a factor of 4.6 and here it has degraded to a factor of 2 or so...
SparseArray method
Have I said already that I love SparseArrays?
AbsoluteTiming[
c = Dot[
KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
],
Dot[
test,
KroneckerProduct[
IdentityMatrix[n/3, SparseArray],
ConstantArray[1, {3, 1}]
]
]
]
][[1]]
a == c
76.3822
True
The story goes on...
A combination of the SparseArray method from above with a CompiledFunction):
cf = Compile[{{x, _Integer, 1}, {k, _Integer}},
Table[
Sum[Compile`GetElement[x, i + j], {j, 1, k}],
{i, 0, Length[x] - 1, k}],
CompilationTarget -> "C",
RuntimeAttributes -> {Listable},
Parallelization -> True,
RuntimeOptions -> "Speed"
];
d = KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
].cf[test, 3]; // AbsoluteTiming // First
a == d
33.5677
True
$endgroup$
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "387"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f192187%2ftime-efficient-matrix-elements-grouping-and-summing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Partition[test, {2,3}] is quite slow in this case because it has to rearrange the elements in the data vector that represents the entries of a packed array in the backend:
Flatten[test] == Flatten[Partition[test, {2, 3}]]
False
Using Span (;;) as follows employs 6 monotonically increasing read operations; in this specific case, these operations are faster than using Partition:
n = 24000;
test = RandomInteger[1, {n, n}];
a = Total[Partition[test, {2, 3}], {3, 4}]; //
AbsoluteTiming // First
b = Sum[test[[i ;; ;; 2, j ;; ;; 3]], {i, 1, 2}, {j, 1, 3}]; //
AbsoluteTiming // First
a == b
245.89
117.943
True
However, this performance advantage seems to decay when the matrix test becomes bigger (so swapping is required). E.g., for $n = 4800$, method b is aboutten times faster thana`, but for $n = 24000$, it's only a factor of 4.6 and here it has degraded to a factor of 2 or so...
SparseArray method
Have I said already that I love SparseArrays?
AbsoluteTiming[
c = Dot[
KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
],
Dot[
test,
KroneckerProduct[
IdentityMatrix[n/3, SparseArray],
ConstantArray[1, {3, 1}]
]
]
]
][[1]]
a == c
76.3822
True
The story goes on...
A combination of the SparseArray method from above with a CompiledFunction):
cf = Compile[{{x, _Integer, 1}, {k, _Integer}},
Table[
Sum[Compile`GetElement[x, i + j], {j, 1, k}],
{i, 0, Length[x] - 1, k}],
CompilationTarget -> "C",
RuntimeAttributes -> {Listable},
Parallelization -> True,
RuntimeOptions -> "Speed"
];
d = KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
].cf[test, 3]; // AbsoluteTiming // First
a == d
33.5677
True
$endgroup$
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
add a comment |
$begingroup$
Partition[test, {2,3}] is quite slow in this case because it has to rearrange the elements in the data vector that represents the entries of a packed array in the backend:
Flatten[test] == Flatten[Partition[test, {2, 3}]]
False
Using Span (;;) as follows employs 6 monotonically increasing read operations; in this specific case, these operations are faster than using Partition:
n = 24000;
test = RandomInteger[1, {n, n}];
a = Total[Partition[test, {2, 3}], {3, 4}]; //
AbsoluteTiming // First
b = Sum[test[[i ;; ;; 2, j ;; ;; 3]], {i, 1, 2}, {j, 1, 3}]; //
AbsoluteTiming // First
a == b
245.89
117.943
True
However, this performance advantage seems to decay when the matrix test becomes bigger (so swapping is required). E.g., for $n = 4800$, method b is aboutten times faster thana`, but for $n = 24000$, it's only a factor of 4.6 and here it has degraded to a factor of 2 or so...
SparseArray method
Have I said already that I love SparseArrays?
AbsoluteTiming[
c = Dot[
KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
],
Dot[
test,
KroneckerProduct[
IdentityMatrix[n/3, SparseArray],
ConstantArray[1, {3, 1}]
]
]
]
][[1]]
a == c
76.3822
True
The story goes on...
A combination of the SparseArray method from above with a CompiledFunction):
cf = Compile[{{x, _Integer, 1}, {k, _Integer}},
Table[
Sum[Compile`GetElement[x, i + j], {j, 1, k}],
{i, 0, Length[x] - 1, k}],
CompilationTarget -> "C",
RuntimeAttributes -> {Listable},
Parallelization -> True,
RuntimeOptions -> "Speed"
];
d = KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
].cf[test, 3]; // AbsoluteTiming // First
a == d
33.5677
True
$endgroup$
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
add a comment |
$begingroup$
Partition[test, {2,3}] is quite slow in this case because it has to rearrange the elements in the data vector that represents the entries of a packed array in the backend:
Flatten[test] == Flatten[Partition[test, {2, 3}]]
False
Using Span (;;) as follows employs 6 monotonically increasing read operations; in this specific case, these operations are faster than using Partition:
n = 24000;
test = RandomInteger[1, {n, n}];
a = Total[Partition[test, {2, 3}], {3, 4}]; //
AbsoluteTiming // First
b = Sum[test[[i ;; ;; 2, j ;; ;; 3]], {i, 1, 2}, {j, 1, 3}]; //
AbsoluteTiming // First
a == b
245.89
117.943
True
However, this performance advantage seems to decay when the matrix test becomes bigger (so swapping is required). E.g., for $n = 4800$, method b is aboutten times faster thana`, but for $n = 24000$, it's only a factor of 4.6 and here it has degraded to a factor of 2 or so...
SparseArray method
Have I said already that I love SparseArrays?
AbsoluteTiming[
c = Dot[
KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
],
Dot[
test,
KroneckerProduct[
IdentityMatrix[n/3, SparseArray],
ConstantArray[1, {3, 1}]
]
]
]
][[1]]
a == c
76.3822
True
The story goes on...
A combination of the SparseArray method from above with a CompiledFunction):
cf = Compile[{{x, _Integer, 1}, {k, _Integer}},
Table[
Sum[Compile`GetElement[x, i + j], {j, 1, k}],
{i, 0, Length[x] - 1, k}],
CompilationTarget -> "C",
RuntimeAttributes -> {Listable},
Parallelization -> True,
RuntimeOptions -> "Speed"
];
d = KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
].cf[test, 3]; // AbsoluteTiming // First
a == d
33.5677
True
$endgroup$
Partition[test, {2,3}] is quite slow in this case because it has to rearrange the elements in the data vector that represents the entries of a packed array in the backend:
Flatten[test] == Flatten[Partition[test, {2, 3}]]
False
Using Span (;;) as follows employs 6 monotonically increasing read operations; in this specific case, these operations are faster than using Partition:
n = 24000;
test = RandomInteger[1, {n, n}];
a = Total[Partition[test, {2, 3}], {3, 4}]; //
AbsoluteTiming // First
b = Sum[test[[i ;; ;; 2, j ;; ;; 3]], {i, 1, 2}, {j, 1, 3}]; //
AbsoluteTiming // First
a == b
245.89
117.943
True
However, this performance advantage seems to decay when the matrix test becomes bigger (so swapping is required). E.g., for $n = 4800$, method b is aboutten times faster thana`, but for $n = 24000$, it's only a factor of 4.6 and here it has degraded to a factor of 2 or so...
SparseArray method
Have I said already that I love SparseArrays?
AbsoluteTiming[
c = Dot[
KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
],
Dot[
test,
KroneckerProduct[
IdentityMatrix[n/3, SparseArray],
ConstantArray[1, {3, 1}]
]
]
]
][[1]]
a == c
76.3822
True
The story goes on...
A combination of the SparseArray method from above with a CompiledFunction):
cf = Compile[{{x, _Integer, 1}, {k, _Integer}},
Table[
Sum[Compile`GetElement[x, i + j], {j, 1, k}],
{i, 0, Length[x] - 1, k}],
CompilationTarget -> "C",
RuntimeAttributes -> {Listable},
Parallelization -> True,
RuntimeOptions -> "Speed"
];
d = KroneckerProduct[
IdentityMatrix[n/2, SparseArray],
ConstantArray[1, {1, 2}]
].cf[test, 3]; // AbsoluteTiming // First
a == d
33.5677
True
edited 2 hours ago
answered 3 hours ago
Henrik SchumacherHenrik Schumacher
54.9k474153
54.9k474153
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
add a comment |
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Thanks! The sparse array will need some studying on my side for fully understand what's going on there, but it look quite promising in terms of speed up!
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
@Fraccalo Have look my latest edits. I seem to have found a method that is twice as fast.
$endgroup$
– Henrik Schumacher
2 hours ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
$begingroup$
This looks amazing, thank you so much @Henrik Schumacher! I'll go through the details first thing tomorrow morning, tons of stuff to learn (I'm not very familiar with the compile function, and haven't used sparse arrays more then a couple of times so far, so I really have lot of new things to learn here :) )
$endgroup$
– Fraccalo
9 mins ago
add a comment |
Thanks for contributing an answer to Mathematica Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f192187%2ftime-efficient-matrix-elements-grouping-and-summing%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown



3
$begingroup$
No need to map if you use the second argument of
Total:Total[Partition[test, {2, 3}], {3, 4}].$endgroup$
– J. M. is computer-less♦
4 hours ago
$begingroup$
oh yes, you are absolutely right, I will change my post accordingly, thanks! Still, I'm not 100% sure this is the fastest solution
$endgroup$
– Fraccalo
3 hours ago
$begingroup$
Converting entries to machine precision numbers will speed up the process. Integers in Mathematica are of arbitrary precision and often slows down the calculation.
$endgroup$
– ukar
3 hours ago
$begingroup$
@ukar True, but I think that converting a 48k x 48k matrix with N with take a lot of time, worth doing some testing though
$endgroup$
– Fraccalo
3 hours ago
1
$begingroup$
@ukar: Wrong. When the integers do not exceed the bounda for machine integera (64 bit integers), Mathematica has a chance to use packed arrays and machine integer computations. And indeed, the matrices generated by
RandomInteger[1, {n, n}]are packed which can be checked withDeveloper`PackedArrayQ[test].$endgroup$
– Henrik Schumacher
3 hours ago