Laravel 5 orderBy relationship count and pagination

Im working on a link sharing website and need to order the links by their votes.

I have a 'links' table and a link_votes table.

The link_votes table has a 'link_id' reference field to the links table 'id' field.

I have a hasMany relationship in my Link model:

public function votes()
{
    return $this->hasMany('\App\LinkVote');
}

This returns all rows (each row is a single vote) int he link_votes table.

My standard query to get all my Links and order them by date created is:

$links = Link::withCount('votes')->orderBy('votes_count', 'desc')->paginate(10); 

What I wish to do now is order my links by the amount of votes they have.

What I have currently is:

$links = Link::orderBy('created_at', 'desc')->with('votes')->paginate(20);

This works, but almost too well. I have 2 types of vote in my link_votes table by having a vote_type field. If the vote is an upvote its a type of 1 but if its a downvote its a type of 2.

The issue is here by ordering by vote count its taking all votes as a count and putting votes with a negative vote above those links that don't have any votes.

I need to find a way of ordering by the sum of the upvotes (vote_type = 1) minus the downvotes (vote_type = 2).

In my view I show the vote count by doing that maths.

via Chebli Mohamed