OpenEBS Mayastor-io-engine failing with error allocating rte services array

General Discussions microk8s
1

Hello,

I’m fairly new to microk8s. Trying to setup mayastor on my locally hosted 3 node HA cluster.
I’m getting the error:

$ RUST_BACKTRACE=full microk8s.kubectl logs -n mayastor daemonset/mayastor-io-engine
Found 3 pods, using pod/mayastor-io-engine-vhxh4
Defaulted container “io-engine” out of: io-engine, agent-core-grpc-probe (init), etcd-probe (init), initialize-pool (init)
[2024-07-03T18:27:15.636685831+00:00 INFO io_engine:io-engine.rs:179] Engine responsible for managing I/Os version 1.0.0, revision b0734db654d8 (v2.0.0)
[2024-07-03T18:27:15.636820800+00:00 INFO io_engine:io-engine.rs:158] free_pages 2MB: 2048 nr_pages 2MB: 2048
[2024-07-03T18:27:15.636828975+00:00 INFO io_engine:io-engine.rs:159] free_pages 1GB: 0 nr_pages 1GB: 0
[2024-07-03T18:27:15.636919560+00:00 INFO io_engine:io-engine.rs:211] kernel io_uring support: yes
[2024-07-03T18:27:15.636932205+00:00 INFO io_engine:io-engine.rs:215] kernel nvme initiator multipath support: yes
[2024-07-03T18:27:15.636971899+00:00 INFO io_engine::core::env:env.rs:791] loading mayastor config YAML file /var/local/io-engine/config.yaml
[2024-07-03T18:27:15.636982985+00:00 INFO io_engine::subsys::config:mod.rs:168] Config file /var/local/io-engine/config.yaml is empty, reverting to default config
[2024-07-03T18:27:15.636993454+00:00 INFO io_engine::subsys::config::opts:opts.rs:151] Overriding NVMF_TCP_MAX_QUEUE_DEPTH value to ‘32’
[2024-07-03T18:27:15.636999945+00:00 INFO io_engine::subsys::config::opts:opts.rs:151] Overriding NVME_QPAIR_CONNECT_ASYNC value to ‘true’
[2024-07-03T18:27:15.637004392+00:00 INFO io_engine::subsys::config:mod.rs:216] Applying Mayastor configuration settings
EAL: alloc_pages_on_heap(): couldn’t allocate memory due to IOVA exceeding limits of current DMA mask
EAL: alloc_pages_on_heap(): Please try initializing EAL with --iova-mode=pa parameter
EAL: error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
thread ‘main’ panicked at ‘Failed to init EAL’, io-engine/src/core/env.rs:628:13
stack backtrace:
0: std::panicking::begin_panic
1: io_engine::core::env::MayastorEnvironment::init
2: io_engine::main
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

Have tried to edit and add the --iova-mode=pa as suggested to no avail:

$ microk8s.kubectl edit -n mayastor daemonset mayastor-io-engine
…
containers:
- args:
- -g$(MY_POD_IP)
- -N$(MY_NODE_NAME)
- -Rhttps://mayastor-agent-core:50051
- -y/var/local/io-engine/config.yaml
- -l1,2
- -p=etcd-client:2379
- --ptpl-dir=/var/local/io-engine/ptpl/
- --api-versions=v1
- --iova-mode=pa
command:
- io-engine
env:
- name: RUST_LOG
value: info,io_engine=info
…

Any more ideas on what could go wrong? I’m running latest stable versions as of date.

2

Dear Alberto,

please excuse my quite late question, but I’ve ran into the same problem on my fresh 3-node cluster on physical machines with 16 GB RAM each.

Have you finally been able to resolve this issue? I’m currently completely stuck here, the mayastor pod won’t come up…