As long as you turn on JS defer, it won't slow down your page loading. Also if you have HTTP/2 push on, it will be faster.
WP puts JQ before <body> tag, if you move it to bottom, you added a new one manually? In this case, you can disable LSCWP's optimizer to move it by adding attribute...