Laravel 5.4 eloquent relationship and local scope

I use laravel 5.4 and have 2 models, User and Provinces.

In model User:

    public function scopeWithSort($query, $sorting, $by)
    {
        return $query->orderBy($sorting, $by);
    }
    public function province()
    {
        return $this->belongsTo('App\Province', 'loction_id');
    }

In controller User:

$query = User::with('province')->select('*');
if ($request->sorting == 'location_id') {
   $query->withSort('provinces.name', $request->by); // ** error line
}

How can I sorting by using this relation. Thanks advances!

via Chebli Mohamed