Hm, the AX860i did have some issues related to random shutdowns but since you're running fine at 4.5GHz that shouldn't be the problem.
Can you try booting up with 46x ratio and 1.35V? Even if it's not XTU stable, it should be able to pass POST (boot) and even go into Windows.
It almost sounds like there's an issue with the µITX board. OCP is possible; might also be BIOS/board related. I haven't seen many builds with the ITX, let alone overclocks, so it's hard to say if asrock build in some protection. I would assume they did since the PCB real estate is very small on ITX.
Maybe someone closer to Asrock can check?