I understand that you have to use the two weapon fighting style to add your ability modifier to damage, but I was wondering if there were any penalties to the offhand attack modifier?

No, there is no concept of offhand attack (or related penalties) in 5e

Here are the general rules for two-weapon fighting:

When you take the Attack action and attack with a light melee weapon that you're holding in one hand, you can use a bonus action to attack with a different light melee weapon that you're holding in the other hand. You don't add your ability modifier to the damage of the bonus attack, unless that modifier is negative. If either weapon has the thrown property, you can throw the weapon, instead of making a melee attack with it. (PHB, p. 195)

and here is the specific rules for the Two-Weapon Fighting fighting style:

When you engage in two-weapon fighting, you can add your ability modifier to the damage of the second attack. (PHB, p. 72)

There is no mention of any kind of penalty or any reference to "primary" and "off" hands. The reason for that is that those do not exist in 5e. It is one of the changes 5e made from older editions.

So, both hands' attack rolls are rolled without penalty even without the Two-weapon fighting fighting style. And neither hand is designated as the primary or offhand. Either hand can be used to make the first attack (using an action) or the second attack attack (using a bonus action) and they are mechanically identical.

No, there is no attack penalty (nor an actual concept of an off-hand)

The entirety of the default two-weapon fighting rules are described in the combat section of the basic rules:

Two-Weapon Fighting

When you take the Attack action and attack with a light melee weapon that you're holding in one hand, you can use a bonus action to attack with a different light melee weapon that you're holding in the other hand. You don't add your ability modifier to the damage of the bonus attack, unless that modifier is negative.

If either weapon has the thrown property, you can throw the weapon, instead of making a melee attack with it.

The only difference between a normal attack as you might make with the Attack action and the bonus attack granted by two-weapon fighting is that you don't get a positive ability score modifier to damage. The attack roll itself is not called out as being different in any way - you apply all appropriate modifiers to that as normal.

It's also worth noting that there is no actual concept of an off-hand in 5e's two-weapon fighting. If you're holding two light weapons, the Attack action attack which allows you to qualify for the TWF bonus action could be made with either hand, and gets your normal damage bonus; the bonus action is then made with the other hand, and doesn't get the bonus. You could mix up which weapon you use for your "primary" attack from turn to turn as you like.

If you have the Extra Attack class feature, you can even use each weapon at least once as part of your normal Attack action, and then both hands/weapons qualify for use with the bonus action, and you can choose which one you use for the bonus attack.

Off your turn, if you get the opportunity to make an opportunity attack, you can do it with either weapon you're holding and benefit from all your normal modifiers.