Laravel 5.8 - Eloquent not listening to where on relation

I'm trying to do a where query on my relation called employee_label.

The table employee_label looks like this:

+----+-------------+----------+
| id | employee_id | label_id |
+----+-------------+----------+
|  1 |         123 |      123 |
|  2 |         456 |      456 |
|  3 |         768 |      768 |
+----+-------------+----------+

The other where() and orWhere() get passed an empty array() by default, but could contain data as for example: ['inactive' => 0].

Expected result:
I expect to only receive the employees with for example label 123 when I provide label 123 as where() for the relation.

Actual result:
All employees are returned, the data is not filtered.

Code:

$employees = Employee::with(['employee_label' => function($query) use ($request) {
    $query->where('label_id', '=', $request->get('label'));
}])->where($searchQuery)->orWhere($orQuery)->orderBy('inactive', 'asc')->paginate(20);

What I've tried:
I've tried to change the $query->where('label_id') to $query->where('employee_label.label_id') with no change in the result.

via Chebli Mohamed