Currently, the happiness an inn can provides is a function only of the fruit it has within. Thus, a unit chooses the closest Inn for every team and converts based on which has the largest happiness.
All we have to do to modify the level of conversion is to give units a "loyalty" factor, rather than make them entirely hedonistic and always willing to convert to get more happiness. Basically, the local team's happiness level is multiplied by this 'loyalty' factor, which we'd have to determine through experimentation. This gives units a bias to stay on the current team, and we can easily control the amount of bias until we get the right level of conversion.