NN Model loading from Memory instead of Flash or SD card failing with exceptions

FreeRTOS Multimedia
1

Hi Realtek Team,

We need to keep NN model data and firmware in single partition to avoid multiple OTA updates. so we tried below code to initialize NN model with model_src = MODEL_SRC_MEM and with static buffer initialized with NN model data as below.

image
image1010×784 120 KB

image
image804×594 66.5 KB

The application failed with usage fault.

fwin(1),enc_en(0),IQ_OFFSET = 0x17a00
fwin(1),enc_en(0),SENSOR_OFFSET = 0x29b00
sensor id 1 iq_data 17a00 sensor_data 29b00
npu[12345678] vipdrv vip_drv_init, video memory heap base: 0x77000000, size: 0x01000000
NN IRQ default priority : 0, set to 9
npu[12345678] HASHMAP 0x0x70946760(process-ID) INIT SUCCESS
npu[12345678] DATABASE 0x0x7094786c(task-desc) INIT SUCCESS
npu[12345678] DATABASE 0x0x70947818(memory-mgt) INIT SUCCESS
set yolo confidence thresh to 0.500000
set yolo NMS thresh to 0.300000

Usage Fault:
SCB Configurable Fault Status Reg = 0x00020000

Usage Fault Status:
Invalid state UsageFault, the processor has attempted to execute an instruction that makes illegal use of the EPSR

S-domain exception from Thread mode, Extended Stack frame on S-PSP
Registers Saved to stack

Stacked:
R0 = 0x707948d4
R1 = 0x70bc2e80
R2 = 0x00000001
R3 = 0x702509d4
R4 = 0x71103360
R5 = 0x71103360
R6 = 0x00000000
R7 = 0x701238b5
R8 = 0x707d6cd8
R9 = 0x707d6cc0
R10 = 0x707d6cc0
R11 = 0x70793e54
R12 = 0x7fa66666
LR = 0x701234df
PC = 0x702509d4
PSR = 0x60100000

Current:
LR = 0xffffffed
MSP = 0x20003fd8
PSP = 0x70bc2c80
xPSR = 0x60000006
CFSR = 0x00020000
HFSR = 0x00000000
DFSR = 0x00000000
MMFAR = 0x00000000
BFAR = 0x00000000
AFSR = 0x00000000
PriMask = 0x00000000
SVC priority: 0x00
PendSVC priority: 0xf0
Systick priority: 0xf0

MSP Data:
20003FD8: 70828400 FFFFFFF9 20004000 000003E8
20003FE8: E000E000 70828400 07070707 701409E9
20003FF8: 70140A58 21000000 5E3F7E33 9B5A4DED
20004008: FD7FBF5F DBC3ED7A FFB57D39 EF46DF3A
20004018: 072E9635 8BA53CC6 A7FEFBD5 F5F36E37
20004028: DB52FF7F 77EFDC00 1E7EEC44 4F5F7E20
20004038: 6F737F9F 6BEFCCFB 451830C7 F5700D1E
20004048: 47A692FD 76934419 26F389F2 91484013
20004058: 20CBD387 03BF7E6B C54244AC 810D9935
20004058: 20CBD387 03BF7E6B C54244AC 810D9935
20004068: FB80D66D 7780F10D 0D7C15A9 4E2699E3
20004078: 74F8CF8E 721F645E BD6A80F1 848408A0
20004088: 360A2F3E D07572CF 6069806A AB789E65
20004098: 6B71BD0F 12F95DEE 83C788A7 AADD4138
200040A8: 73E516F8 B638A75A 4E59C54E 630751F0
200040B8: 8F985B6A 6B75F6CE 361675F4 1190B39A
200040C8: 92563CE8 CE468FC2 20C7294E 6A3EAF5E

PSP Data:
70BC2C80: 707948D4 70BC2E80 00000001 702509D4
70BC2C90: 7FA66666 701234DF 702509D4 60100000
70BC2CA0: 00000001 70BBABA0 00000000 00000000
70BC2CB0: 70BBABA0 7013C247 00000000 7013D677
70BC2CC0: 70BBABA0 7013C65F 707968EC 00002710
70BC2CD0: 056F3BA0 71125E00 707968EC 708283E8
70BC2CE0: 00000000 70644D48 71103360 70BC2E80
70BC2CF0: 00000001 70BBAAC0 71103360 00000000
70BC2D00: 701238B5 707D6CD8 707D6CC0 707D6CC0
70BC2D20: 000000A0 7020911F 70BBAAC0 000000A0
70BC2D30: 00000000 00000000 707D6D44 707D6CCC
70BC2D40: 707D77CC 707D77C4 707D6CD8 707D6CC0
70BC2D50: 707D6CC0 701336E5 00000001 00000001
70BC2D60: 00000500 000002D0 00200000 00000001
70BC2D70: 00000001 00000280 000001E0 00200000

== NS Dump ==
CFSR_NS = 0x00000000
HFSR_NS = 0x00000000
DFSR_NS = 0x00000000
MMFAR_NS = 0x00000000
BFAR_NS = 0x00000000
AFSR_NS = 0x00000000
MSP_NS = 0x00000000
PSP_NS = 0x00000000
NS HardFault Status Reg = 0x00000000
SCB Configurable Fault Status Reg = 0x00000000
No Back Trace!

To confirm the issue, we disabled NN model code and tried then no exceptions and also checked by enabling NN model code with flash loading again and it is also working fine. Now can you tell us is anything wrong in initialization or model data C array creation from nb file?

I didn’t find any example which demonstrates MODEL_SRC_MEM in GitHub - Ameba-AIoT/ameba-rtos-pro2: Realtek Official IoT Software Development Kit for Ameba Series SoC: (Ameba Pro2) · GitHub so I am not sure whether the changes I made are valid or not. can you provide working example for MODEL_SRC_MEM?

Edit: Earlier I assigned direct variables in nnmodel_t structure members nb and nb_size, Now I changed them to functions and tested again, Now it is doing object detections, but failing after some time. I noticed a function vipnn_allocate_videomemory which is used to allocate 4K aligned memory while reading NN model data from flash or SD card, Is this 4K alignment mandatory? If my buffer is not 4K aligned one will it create any excpetions like I reported?

image
image1006×505 79.1 KB

Thanks,

Yugandhar

2

Hi @yugandhar ,

Kindly ensure that your static NN model data array is 4096-byte aligned.

__attribute__((aligned(4096))) const unsigned char model_data[] = { ... };

Please add cache maintenance before calling vip_create_network() in the MODEL_SRC_MEM path (line 908 of module_vipnn.c:

if (ctx->params.model->model_src == MODEL_SRC_MEM) {
    void *nn_model = ctx->params.model->nb();
    dcache_clean_invalidate_by_addr((uint32_t *)nn_model, ctx->params.model->nb_size());
    status = vip_create_network(nn_model, ctx->params.model->nb_size(), VIP_CREATE_NETWORK_FROM_MEMORY, &ctx->network);
    ctx->params.model->freemodel(nn_model);
}

Thank you.