I think custom RMB functionality doesn’t have a way to specify a layout. However you can create the RMB as normal together with a report rule which will pre-select the layout you need. But this rule will affect when ordering the same report from other places as well. If you use preselect property report rule action type, it will select that layout by default when opening print dialog but user can change it as required.

Hi, this could help.

Thanks.