Is there really no way to perform limit() in with() method? (Eloquent)

Mode model has a hasMany relationship with DataSet model. I want to load data sets with the with()-method. It works fine, unless I need a limit on DataSet per Mode. The following code does not work as expected. It limits the DataSet results all together and not individually per Mode.

$modes = $modes->with(['dataSets' => function ($query) use ($user, $count) {

    // Select data sets of user from each mode

    $query->join('data_set_user', 'id', '=', 'data_set_id')
          ->where('user_id', $user->id);

    if(isset($count))
           $query->limit($count);

 }])->orderBy('order_value')
            ->get();

Have a look at the query which this code produces:

SELECT * FROM `data_sets` INNER JOIN `data_set_user` on `id` = `data_set_id` where `data_sets`.`mode_id` in (?, ?, ?) and `user_id` = ? limit 5"

Is there any Laravel-way to make the limit work for each mode individually?

via Chebli Mohamed