How to get record order by pivot column using laravel elquent?

I am trying to sort records order by highscore desc that I get top high scored users. I have tried a lot and still trying but could achieve the task.

Following is working but it does sorting

$GameLog = User::with(['games' => function ($q){
    $q->withPivot('highscore','level');
}])->get();

I also tried following to achieve it but this is also not working

$GameLog = User::with(['games' => function ($q){
    $q->withPivot('highscore','level');
}])->groupBy(DB::raw('highscore DESC'))
->get();

I also tried sortBy() function in the loop but still I am facing issue. The first query is returning following result which should be sort by highscore

{
    "status": "success",
    "data": [
        {
            "id": 1,
            "name": "fasdfsad",
            "games": [
                {
                    "id": 1,
                    "pivot": {
                        "highscore": 506,
                    }
                }
            ]
        },
        {
            "id": 8,
            "name": "john",
            "favorite_game_id": null,
            "games": [
                {
                    "id": 1,
                    "pivot": {
                        "highscore": 2340,
                    }
                }
            ]
        },
        {
            "id": 10,
            "name": "tfyuyu",
            "games": [
                {
                    "id": 1,
                    "pivot": {
                        "highscore": 100,
                    }
                }
            ]
        }
    ]
}

Can someone kindly give me idea how can I fix it I would appreciate. Thank you so much

via Chebli Mohamed