How to disable/override $with attribute in Laravel models

I have this model:

class Payment extends MyBaseModel
{  
    protected $table = 'payments';
    protected $with = ['company', 'account', 'manager', 'block', 'network'];
    protected $appends = ['someAttribute', 'someOtherAttribute'];

   ...
}

In another model I have a relation that looks like this:

public function payment() {
      return $this->belongsTo('App\Payment', 'payment_id', 'id')->withTrashed();
}

What I need is to be able to, sometimes, retrieve $this->payment without all the the relationships specified in $with (i.e. company, account, manager, block, network) also would like to (sometimes) disable the appended attributes.

Can't find a way to accomplish that. I've tried the following with no luck.

public function payment() {
    $this->belongsTo('App\Payment', 'payment_id', 'id')->setEagerLoads([])->withTrashed();
}

via Chebli Mohamed