[BUG] Socket closed unexpectedly - Redis

[danielstankw]

I have installed the Flowiseai using the traditional method (docker) as well as Kubernetes pods using unofficial chart.
In both of those cases whenever I attempt to connect to the redis store to store vector embeddings I get the following error:

2024-04-15 07:03:35 [INFO]: Starting Flowise...
2024-04-15 07:03:35 [INFO]: ⚡️ [server]: Flowise Server is listening at 3000
2024-04-15 07:03:36 [INFO]: 📦 [server]: Data Source is being initialized!
2024-04-15 07:03:38 [INFO]: 📦 [server]: Data Source has been initialized!
2024-04-15 07:24:32 [INFO]: ⬆️ POST /api/v1/vector/internal-upsert/962166d0-816d-4c8d-8a2e-341a53495b37
2024-04-15 07:25:32 [ERROR]: uncaughtException:  Socket closed unexpectedly
Error: Socket closed unexpectedly
    at Socket.<anonymous> (/usr/local/lib/node_modules/flowise/node_modules/@redis/client/dist/lib/client/socket.js:194:118)
    at Object.onceWrapper (node:events:632:26)
    at Socket.emit (node:events:517:28)
    at TCP.<anonymous> (node:net:350:12)

Not sure if this is the issue but it seems like its Redis communication problem:
https://stackoverflow.com/questions/71574823/reusing-redis-connection-socket-closed-unexpectedly-node-redis
https://community.fly.io/t/redis-socketclosedunexpectedlyerror-socket-closed-unexpectedly/5897

[danielstankw]

Do I need to install any additional dependencies to run flowiseai with redis?

I am accessing Redis that is hosted externally.
When I use In memory vector store I am also failing to upsert with different error (see below)

2024-04-16 13:21:32 [ERROR]: Error: Connection error.
Error: Error: Connection error.
    at InMemoryVectorStore_VectorStores.upsert (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/InMemory/InMemoryVectorStore.js:26:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async buildFlow (/usr/local/lib/node_modules/flowise/dist/utils/index.js:342:37)
    at async upsertVector (/usr/local/lib/node_modules/flowise/dist/utils/upsertVector.js:117:32)
    at async Object.upsertVectorMiddleware (/usr/local/lib/node_modules/flowise/dist/services/vectors/index.js:6:9)
    at async createInternalUpsert (/usr/local/lib/node_modules/flowise/dist/controllers/vectors/index.js:26:16)
2024-04-16 13:21:32 [ERROR]: [server]: Error: Error: Error: Connection error.
Error: Error: Error: Connection error.
    at buildFlow (/usr/local/lib/node_modules/flowise/dist/utils/index.js:405:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async upsertVector (/usr/local/lib/node_modules/flowise/dist/utils/upsertVector.js:117:32)
    at async Object.upsertVectorMiddleware (/usr/local/lib/node_modules/flowise/dist/services/vectors/index.js:6:9)
    at async createInternalUpsert (/usr/local/lib/node_modules/flowise/dist/controllers/vectors/index.js:26:16)

I tried:
Enter container and ping the AzureOpenAi and Redis endpoints, I also tried to run test scripts for retrieving data from redis and embeddings from within the container and was successful.

Do you have any suggestion of pipeline that will let me test if the issue is with embedding or vector store?

[HenryHengZJ]

Strange, you should be fine with InMemory since it doesn't make connection. Id suggest try with different vector store, then embeddings, basically divide and conquer

[danielstankw]

@HenryHengZJ
The Issue persists and I boiled it down to the problem with redis vector store - flowiseai implementation.
I checked here, and I believe it might be due to the timeout on the redis connection, but I am not 100% sure.

I think fix requires modification of redis module implementation in flowiseai, swapping vector store to the In-memory solved the connection problem.

Even if I dont execute any queries there error below happens.

2024-05-23 08:10:08 [ERROR]: uncaughtException:  Socket closed unexpectedly
Error: Socket closed unexpectedly
    at Socket.<anonymous> (/usr/local/lib/node_modules/flowise/node_modules/@redis/client/dist/lib/client/socket.js:194:118)
    at Object.onceWrapper (node:events:632:26)
    at Socket.emit (node:events:517:28)
    at TCP.<anonymous> (node:net:350:12)

[danielstankw]

@HenryHengZJ is there any update on the redis vector store fix?

[Nek-11]

Any update ?
1 year later and we still have the same issue...

I am using Redis for queue mode and i also have the exact same [ERROR]: uncaughtException: Socket closed unexpectedly coming from redis.

@HenryHengZJ we need some sort of keep-alive mechanism or even just a ping at regular intervals.

[Nek-11]

@HenryHengZJ Probably found a fix:

Node-redis supports OS-level keep-alives and protocol-level pings via client options:
https://github.com/redis/node-redis/blob/master/docs/client-configuration.md#client-configuration

By changing every place you build or pass a Redis config (whether via createClient() or a config object in your queue manager, vector store, pub/sub, etc.) to include:

--- before
-  // your existing Redis options:
-  url: process.env.REDIS_URL,
-  // or host/port + auth + tls in queuemanager.ts
+  // your existing Redis options:
+  url: process.env.REDIS_URL,
+  // or host/port + auth + tls…
+  
+  // send OS-level TCP keep-alive probes (1st after 4 min):
+  socket: {
+    /* existing socket opts */,
+    keepAlive: true,
+    keepAliveInitialDelay: 240_000,
+  },
+  // send a Redis PING every 4 min to reset Azure’s 10 min idle timer:
+  pingInterval: 240_000,
+  // avoid buffering commands if the socket blips:
+  disableOfflineQueue: true,

you ensure the same Flowise TCP connection is never idle long enough to trigger Azure’s per-connection 10 min timeout.

⚠️ This bug affects all Redis clients in Flowise (queue manager, vector store, SSE, etc.), so I think it should be quite critical for the next release.

[Nek-11]

@HenryHengZJ
I made the necessary changes and tested them, seems to solve it. Will do a PR tomorrow

[Nek-11]

@HenryHengZJ @danielstankw

Here's the PR for this issue:
#4377

[HenryHengZJ]

question: why dont just disabled timeout on Redis? Because the recent versions of Redis doesnt have a timeout already https://redis.io/docs/latest/develop/reference/clients/#client-timeouts

[Nek-11]

question: why dont just disabled timeout on Redis? Because the recent versions of Redis doesnt have a timeout already https://redis.io/docs/latest/develop/reference/clients/#client-timeouts

Some providers do enable timeouts. For example Azure Cache for Redis enforces a non-configurable 10-minute idle timeout on server-side connections. Not sure when/if they will update that (they're still on Redis 6.0) but since it's used for a lot of production environments this keepalive mechanism is needed in this case..