Very slow migrations

I have a (KVM) Debian 8 VM server it's running MySQL 5.5.58 and PHP7.1 locally I have a similar machine (Vagrant VM, Debian 8, VirtualBox). Locally my migrations run < 20 seconds if I dump them into a SQL file and import the SQL file it takes about 2-3 seconds.

However on the (KVM) Debian 8 VM server migrations run between 10-20minutes. I have tried increasing CPU count and RAM to no avail.

There are however two things that greately improve performance but still not really usable:

1. Convert migration files into raw SQL and run that takes about 20-50 seconds.
2. Set innodb_flush_log_at_trx_commit=0 and run migrations this reduces migration run time down to about 4-5 minutes which is still not usable.

my.cnf

[mysqld]
sync_binlog = 0
federated = 1
innodb_use_sys_malloc = 0
innodb_file_per_table = 1
innodb_stats_on_metadata = 0
innodb_buffer_pool_instances = 1
query_cache_type = 0
innodb_buffer_pool_size = 1G
innodb_log_file_size = 768M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
user        = mysql
pid-file    = /var/run/mysqld/mysqld.pid
socket      = /var/run/mysqld/mysqld.sock
port        = 3306
basedir     = /usr
datadir     = /var/lib/mysql
tmpdir      = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
bind-address        = 127.0.0.1
key_buffer      = 16M
max_allowed_packet  = 16M
thread_stack        = 192K
thread_cache_size       = 8
myisam-recover         = BACKUP
#max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10
query_cache_limit   = 1M
query_cache_size        = 16M

Filesystem:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         32G  6.0G   25G  20% /
udev             11M     0   11M   0% /dev
tmpfs           422M   17M  406M   4% /run
tmpfs           1.1G     0  1.1G   0% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs           1.1G     0  1.1G   0% /sys/fs/cgroup
tmpfs           211M     0  211M   0% /run/user/2031

What I have tested and checked:

1. Made sure I'm using same version of everything as locally myqsl, php, composer packages no discrepencies here.
2. Tested VM's disk FIO speeds to make sure there is no bottleneck here tried writing/reading single large file as well as tousands of small files getting high speeds across the board (disk is SSD and healthy).
3. Added extra cores and monitored CPU usage during migration it's pretty small.
4. Timed execution of Laravel getUp method from Migrator. Specifically timed reading migration file from disk, creating a new class and executing the query all seems to be executing < 1s per migration used microtime(true).
5. Checked mysql slow queries for any slow queries nothing was found, migrations are really very basic CREATE TABLE statements.
6. Tried installing MySQL 5.7 there was no improvement.
7. Tried increasing MySQL pool and log RAM size.
8. Made sure I'm using InnoDB.

Note: This happens in all ways of using MySQL I tested with MySQL installed on host. I installed docker mysql same issue. I installed vagrant instance and mysql inside that same issue.

This innodb_flush_log_at_trx_comit seems to make the most difference indicating I messed up something with mysql just not sure why. Migrations each runs as it's own transaction seems something is causing wait time here not sure what else to check.

via Chebli Mohamed