Skip to content

Having "Unable to receive result for write task: channel closed" error on v1.6.23 #10578

@Mhs-220

Description

@Mhs-220

Describe the bug
After upgrading cube-js from 1.5.15 to 1.6.23 we're seeing a random failure in our self-hosted CubeJS instance. Suddenly nothing works and we see the following log on the main Cube instances:

QueryError: Internal: Unable to receive result for write task: channel closed
    at WebSocket.<anonymous> (/cube/node_modules/@cubejs-backend/cubestore-driver/src/WebSocketConnection.ts:163:32)
    at WebSocket.emit (node:events:519:28)
    at Receiver.receiverOnMessage (/cube/node_modules/ws/lib/websocket.js:1070:20)
    at Receiver.emit (node:events:519:28)
    at Receiver.dataMessage (/cube/node_modules/ws/lib/receiver.js:502:14)
    at Receiver.getData (/cube/node_modules/ws/lib/receiver.js:435:17)
    at Receiver.startLoop (/cube/node_modules/ws/lib/receiver.js:143:22)
    at Receiver._write (/cube/node_modules/ws/lib/receiver.js:78:10)
    at writeOrBuffer (node:internal/streams/writable:572:12)
    at _write (node:internal/streams/writable:501:10)

And on the API side, we also see this error:

{
    "errors": [
        {
            "message": "QueryError: Internal: Unable to receive result for write task: channel closed",
            "locations": [
                {
                    "line": 231,
                    "column": 5
                }
            ],
            "path": [
                "cube"
            ]
        }
    ],
    "data": null,
    "extensions": {}
}

The issue persist until we restart the workers and the router.

To Reproduce
It seems to be random or at least we don't know what is causing it.

Expected behavior
Have a self-healing mechanism for cases like this.

Version:
1.6.23

Additional context
We're self hosting on ECS + Fargate and cube instances are connected via AWS Service Discover.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions