Laravel debugger - enable/disable depending on IP address while caching implemented

We need to eneble/disable Laravel debugbar depending on IP address. It works if we clear/disable the caching.

But it does not work when caching enabled. here is my code

//Enabling DEBUGBAR in Production Only for developers
if(in_array($request->ip(), [ip addresses])) {
    config(['app.debug' => true]);
}

We are using configuration and route caching. What would be the best way to achieve this?

Laravel version - 5.4

Debugbar version - 2.2

via Chebli Mohamed