Simple way to compare manytomany relationships? Laravel 5.3

I have a set of relationships that looks like this:

The users and agencies have a lot of data stored in them, in addition to the pivot tables you see there.

What I'd like to do is find the agencies or users that match the preferences of the currently logged in individual, whether they are an agency or user.

It's a straight comparison, so nothing fancy. But I honestly have no idea how to write the eloquent query to account for the pivot tables. Can someone point me in the right direction?

I'm looking at something like this, which is understandably failing:

$loggedinagency = Auth::user()->id;
$match_user_agency = User::with('work_prefs')
            ->where('work_prefs', 'like', 
            Agency::find($loggedinagency)->work_prefs)
            ->get();

via Chebli Mohamed