Htydjtk

E.g. three players want to hurl a gigantic boulder at a monster during combat. Neither one can achieve this alone and must muster their combined strength to lift and hurl the massive stone.

The help action, in any case, won't work, since on p-175 of the Player's Handbook on working together states:

A character can only provide help if the task is one that he or she could attempt alone.

So what should the players do? This should be a team effort and not an assist, so all three players actions must be used.

By RAW, you're right

You've cited the relevant rule already. If a character doesn't have the Strength to lift/drag the boulder (see Lifting and Carrying, PHB p.176), there's no way they're going to get it moving, and that falls into the 'debatable' category of being able to attempt the action, at best.

But clearly that makes no sense

Lifting or pushing a large object in tandem with others is the poster child for tasks that benefit from multiple participants (so long as there's room for everyone).

You (assuming you're the DM in this scenario; if you're not, pitch this to them) would be well within your rights to rule that anything with a Strength score would be able to use the Help action to contribute their excess Lift/Drag capacity to the task. You could also, as detailed in this answer, just use raw Strength scores, though you'd have to make a judgment call on what the threshold would be in either case.

So how might you adjudicate this action?

Effectively, your players would be creating an improvised Rolling Sphere trap, from which we can borrow mechanics:

... a 10 foot diameter rolling sphere of solid stone ... moves 60 feet in a straight line. [It] can move through creatures' spaces, and creatures can move through its space, treating it as difficult terrain.

Whenever the sphere enters a creature's space or a creature enters its space while it's rolling, that creature must succeed on a DC 15 Dexterity saving throw or take 55 (10d10) bludgeoning damage and be knocked prone.

... As an action, a creature within 5 feet of the sphere can attempt to slow it down with a DC 20 Strength check. On a successful check, the sphere's speed is reduced by 15 feet. If the sphere's speed drops to 0, it stops moving and is no longer a threat.

[DMG p.123]

But first they would need to actually get the boulder rolling, which would require enough of the party to spend an action trying to move the boulder (pooling their strength as stated above, until they had enough to move the boulder).

My own solution is to add a caveat to the the help action:

If the creature with the highest ability modifier can not complete the task alone, then the help action becomes a "group effort", becoming a group check without advantage.

The GM should therefore adjust the DC according to who is participating and how many are participating.