Laravel - query takes seconds after adding a simple condition

I got the query below, which works ok:

        $data = User::select('users.id')->distinct()->join('messages as m', function ($join) {
            $join->on('users.id', 'm.user_id_origin')
            ->orOn('users.id', 'm.user_id_destination');
        })
        ->where('m.updated_at','>',date ("Y-m-d H:i:s", $tf))
        ->whereNull('m.deleted_at')
        ->where(function($q) use ($user) {
            $q->where('m.user_id_origin',$user->id);
            $q->orWhere('m.user_id_destination',$user->id);
        })->orderBy('m.created_at','desc');

But as soon as I add the following condition to the query, it takes up to 5 seconds:

            ->where('users.id','!=',$user->id) //ignoring current user from the list

What could be the reason?

via Chebli Mohamed