Laravel order by a nested eager load relationship

Firstly, I tried looking at this, but I couldn't get the solution to work, as it had pivot tables involved, which mine does not. My brain couldn't abstract the solution from that solution.

I have this query:

$orderLines = PurchasingOrderLine::where('season_id', '=', $seasonRecID)
        ->with('factory')
        ->with('purchasingFactoryDates.milestone')
        ->with('divergedColors.purchasingColorDates.milestone')
        ->get();

So it's quite a big relationship.

I want to sort the purchasingFactoryDates by their milestone's "number" attribute.

Here is a sample of the JSON response for this query (a sample)..

  {
    id: 1,
    season_id: 258,
    factory_id: 38,
    archived: 0,
    created_at: "2016-10-03 00:00:00",
    updated_at: "2016-10-04 00:00:00",
    factory: {
       ID: 38,
       name: "testfac",
       address: "testadrr",
       postCode: "",
       city: "Foo",
       countryID: 27,
       podID: 2,
       warehouse_id: 3,
       paymentTermID: 11,
       factoryCode: "Bazz",
       active: 1
    },
    purchasing_factory_dates: [
      {
        id: 1,
        purchasing_order_line_id: 1,
        milestone_id: 1,
        milestone_date: "1993-10-22 19:15:51",
        milestone_status_id: 2,
        created_at: "2016-10-04 07:38:30",
        updated_at: "2016-10-18 19:15:51",
        milestone: {
          id: 1,
          name: "test1",
          description: null,
          number: 1,
          created_at: "-0001-11-30 00:00:00",
          updated_at: "-0001-11-30 00:00:00"
        }
      },
    .........

Any ideas on how to do this? I don't want the entire relation to be sorted by the Milestone.number, only the purchasingFactoryDates.

via Chebli Mohamed