Laravel whereHas not working as expected

I have a relationship on my User table called 'roles'.

This relationship is:

public function roles()
{
    return $this->hasMany('App\RoleUser');
}

I am only trying to fetch users that have a role of 3.

My query is:

User::whereHas('roles', function ($q) {
    $q->where('id', 3);
})->get();

This, however, is returning all users no matter what their role. I can confirm that all users have roles and the roles relationship is working.

The fields in the role_user table are 'id', 'user_id'

via Chebli Mohamed