There are other factors (such as notoriety level), but the main thing is it depends on how many times you hit your target. The more hits (ie the less damage you do per hit), the more rep you gain. This is why putting skill points into your weapons results in less rep - because you kill them in fewer hits.
Additionally not every hit is the same. For example, a thunderbolt cannonball hit gives you more rep than a round shot cannonball hit, regardless of damage. Blade storm gives you more rep than a slash. I even think they made it so a slash on a cutlass gives you more rep than a cut on a rapier (rapier is faster and less damaging, so they made it less rep per hit).
But you also receive diminishing returns on each hit (past a certain point), until eventually it stops giving you rep. Around 115 base (before bonuses) rep per enemy per weapon used seems to be the max. If you're getting over 100 base rep per enemy, you're actually not being very efficient, as it's fastest to have a constant stream of 80-100 base rep per enemy. This of course requires you to always have an enemy available to you, which often isn't the case (respawn times).
