Texonom Recommender 연구노트 11/07

wasm onnx transformers js edge

1. 그냥 올리니 wasm이라 안된다

2. vercel 말대로 edge 하자 wasm은 되지만 dynamic compile 안되서 precompile 못돌림 결론적으로
1. edge crypto도 안되서 pg to prisma 했음
2. 그래도 메인 이유를 확인하며 못돌리는거 확인 (path 도 안댐 node 기본 모듈들 없는런타임인듯)
3. 다시 function으로 돌아와서 onnx-runtime으로 시도

function by serverComponentsExternalPackages in next.js

[Question] Issue with deploying model to Vercel using NextJS and tRPC
Updated 2023 Sep 12 15:23

…ages/api/recommend-pages.js:1:941)
SyntaxError: Unexpected token v in JSON at position 0
    at JSON.parse (<anonymous>)
    at getModelJSON (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/utils/hub.js:551:17)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/tokenizers.js:52:16)
    at async AutoTokenizer.from_pretrained (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/tokenizers.js:3824:48)
    at async Promise.all (index 0)
    at async loadItems (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/pipelines.js:2305:5)
    at async pipeline (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/pipelines.js:2251:19)
    at async searchNotion (/var/task/.next/server/pages/api/recommend-pages.js:1:941)
Error: Failed to load model because protobuf parsing failed.
    at new OnnxruntimeSessionHandler (/var/task/node_modules/.pnpm/onnxruntime-node@1.14.0/node_modules/onnxruntime-node/dist/backend.js:27:92)
    at /var/task/node_modules/.pnpm/onnxruntime-node@1.14.0/node_modules/onnxruntime-node/dist/backend.js:64:29
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
Something went wrong during model construction (most likely a missing operation). Using `wasm` as a fallback. 
Aborted(Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/dist/ort-wasm-simd.wasm')
failed to asynchronously prepare wasm: RuntimeError: Aborted(Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/dist/ort-wasm-simd.wasm'). Build with -sASSERTIONS for more info.
Aborted(RuntimeError: Aborted(Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/dist/ort-wasm-simd.wasm'). Build with -sASSERTIONS for more info.)
f36662ce-7752-5077-9e5c-d31e1385577c
SyntaxError: Unexpected token v in JSON at position 0
    at JSON.parse (<anonymous>)
    at getModelJSON (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/utils/hub.js:551:17)
    at async Promise.all (index 0)
    at async loadTokenizer (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/tokenizers.js:52:16)
    at async AutoTokenizer.from_pretrained (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/tokenizers.js:3824:48)
    at async Promise.all (index 0)
    at async loadItems (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/pipelines.js:2305:5)
    at async pipeline (file:///var/task/node_modules/.pnpm/@xenova+transformers@2.7.0/node_modules/@xenova/transformers/src/pipelines.js:2251:19)
    at async searchNotion (/var/task/.next/server/pages/api/recommend-pages.js:1:941)
Error: Runtime exited with error: exit status 1
Runtime.ExitError

wasm next serverless function 문제