Join and OrderBy within with laravel eloquent query

I want to display ordered list of stores based on cities from a brand.

This is the code I tried

 $brand = Brand::where('slug','=',$slugurl)
      ->with(['stores' => function($q){
        $q->where('status', 1)
        ->join('cities', function ($join){
              $join->on('cities.id', '=', 'stores.city_id')->orderBy('cities.sort_number', 'DESC');
        });

      }])
      ->firstOrFail();

The relationship of the tables :

Brand hasMany Stores and Stores belongTo Cities

The listings results output is not ordered based on cities sort_number. Any idea how to achieve this ?

via Chebli Mohamed