Laravel paginate many to many relationship with eager and orderby

I'm struggling paginating the results of a many-to-many relationship in Laravel. I want to paginate the combination: Not 1 course with 10 availabilities and inversed. I want 10 courses with 10 availabilities. Easy thing if you query the relation table, but I want to order by a table's field (availability date).

Database structure (simplified)

courses
  id - integer
  name - string

course_availability //relation table
  course_id - integer
  availability_id - integer

availability
  id - integer
  date - date

The Availability model has a scope with active dates and ordered by date, the courseAvailability relationship has this included.

Attempts:

Availability view Problem: I get 1 availability with multiple courses (100+)

Availability::whereHas('courseAvailability', function ($q) use($course_ids) {
  $q->whereIn('courses_id', $course_ids);
})->with('courseAvailability' => function($q) use($course_ids) {
  $q->whereIn('courses_id', $course_ids);
}])->forPage(1, 10)->get();

Course Availability view Problem: Sorting on the 10 results, not on the actual data, the relationship values

CourseAvailability::whereIn('courses_id', $course_ids)
  ->whereIn('availability_id', $availability_ids)
  ->forPage(1, 10)->get()->sortBy('availability.date');

via Chebli Mohamed