When inviting people to a meeting, look at the situation from their perspective. Do they need to be there? If not, don’t invite them. Do they need to attend the whole meeting? If they don’t, use your agenda to invite them to only the portions they need to see and manage the agenda so that when they join you will be on that topic. An alternative is to schedule more meetings, but invite only the people necessary to each. Although you will have more meetings, they will be shorter, and everyone else will have fewer and shorter meetings to deal with.
Fine-tuning your meeting schedule to accomplish this depends on getting reliable commitments to attend from all who must be there, so confirm attendance with each person involved prior to the meeting.