[1607] Dynamic imports not working in vue-server-renderer

I have this bit of code in one of my SSR plugins:

let ScrollSnap = null
if (process.browser) {
  (async function () {
    ScrollSnap = await import('./ScrollSnap')
  })()
}

But when running the server, I get this error:

[vue-router] Failed to resolve async component default: SyntaxError: Unexpected token import
[vue-router] uncaught error during route navigation:
pages/index.7e17fc3259340ada06a8.js:244
            return import('./ScrollSnap');
                   ^^^^^^

SyntaxError: Unexpected token import
    at getCompiledScript (/path/to/project/node_modules/vue-server-renderer/build.js:7714:18)
    at evaluateModule (/path/to/project/node_modules/vue-server-renderer/build.js:7729:18)
    at r (/path/to/project/node_modules/vue-server-renderer/build.js:7737:16)
    at Function.requireEnsure [as e] (server-bundle.js:41:25)
    at _a7b33a52 (server-bundle.js:2880:29)
    at /path/to/project/node_modules/vue-router/dist/vue-router.common.js:1711:17
    at /path/to/project/node_modules/vue-router/dist/vue-router.common.js:1738:66
    at Array.map (<anonymous>)
    at /path/to/project/node_modules/vue-router/dist/vue-router.common.js:1738:38
    at Array.map (<anonymous>)
    at flatMapComponents (/path/to/project/node_modules/vue-router/dist/vue-router.common.js:1737:26)
    at /path/to/project/node_modules/vue-router/dist/vue-router.common.js:1673:5
    at iterator (/path/to/project/node_modules/vue-router/dist/vue-router.common.js:1872:7)
    at step (/path/to/project/node_modules/vue-router/dist/vue-router.common.js:1654:9)
    at step (/path/to/project/node_modules/vue-router/dist/vue-router.common.js:1658:9)
    at runQueue (/path/to/project/node_modules/vue-router/dist/vue-router.common.js:1662:3)

I looks like Nuxt.js is using babel-preset-vue-app which uses babel-plugin-syntax-dynamic-import, so I was surprised by this not working...

But then I realized it happens on the Node.js side of things, so I tried to install babel-plugin-dynamic-import-node and use it in Babel by adding this to my nuext.config.js, but that didn't have any effect either:

  babel: {
    plugins: [
      'dynamic-import-node'
    ]
  }

What am I missing? Is there a way to get the vue-server-renderer to use the same babel configuration as the browser?