Restarting print job due to rendering container failure

None of the operational reports are working in one of my customer’s user place environments. They are on IFS Cloud 21R2 SU10. On premise customer. They have many other environments but the issue surfaced only in one of the environments. I would really appreciate it if you can give me some pointers to debug this issue.

Here are the things I have checked so far. 

1. Verify  Message Queue  and Report Messages Queues are setup properly.
2. Print Task Template is pointing to BATCH2
3. Report Print Task Template is set up properly
4. cleared the tables report_batch_pr_queue_tab and report_application_message_tab

Observation

Report Application messages are keep running for the same report.

I see the error below in the ifsapp-reporting-ren container log

{"type":"liberty_message","host":"ifsapp-reporting-ren-8bfb468cb-xwgkd","ibm_userDir":"\/opt\/ifs\/wlp\/usr\/","ibm_serverName":"defaultServer","message":"4735 ERROR [Framework]{ifsapp-reporting-ren} org.apache.fop.apps.Driver.renderFo(): Throwable catched during rendering process [Error GUID:2dd0191b-78c6-481e-a90e-8cca5b162ba5]\njava.lang.NullPointerException\n\tat org.apache.fop.render.pdf.fonts.LazyFont.getAscender(LazyFont.java:127)\n\tat org.apache.fop.layout.FontState.getAscender(FontState.java:270)\n\tat org.apache.fop.layout.LineArea.<init>(LineArea.java:843)\n\tat org.apache.fop.layout.BlockArea.getCurrentLineArea(BlockArea.java:313)\n\tat org.apache.fop.fo.FOText.addRealText(FOText.java:485)\n\tat org.apache.fop.fo.FOText.addText(FOText.java:474)\n\tat org.apache.fop.fo.FOText.layout(FOText.java:383)\n\tat org.apache.fop.fo.flow.Block.layout(Block.java:450)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat
org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat
org.apache.fop.fo.flow.Table.layout(Table.java:612)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat
org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat
org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat
org.apache.fop.fo.flow.Table.layout(Table.java:661)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat org.apache.fop.fo.flow.Table.layout(Table.java:661)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat org.apache.fop.fo.flow.Table.layout(Table.java:661)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat org.apache.fop.fo.flow.Table.layout(Table.java:661)\n\tat org.apache.fop.fo.flow.TableCell.layout(TableCell.java:545)\n\tat org.apache.fop.fo.flow.TableRow.layout(TableRow.java:693)\n\tat org.apache.fop.fo.flow.AbstractTableBody.layout(AbstractTableBody.java:380)\n\tat org.apache.fop.fo.flow.Table.layout(Table.java:661)\n\tat org.apache.fop.fo.flow.AbstractFlow.layout(AbstractFlow.java:281)\n\tat org.apache.fop.fo.flow.AbstractFlow.layout(AbstractFlow.java:237)\n\tat org.apache.fop.fo.pagination.PageSequence.makePage(PageSequence.java:555)\n\tat org.apache.fop.fo.pagination.PageSequence.format(PageSequence.java:484)\n\tat org.apache.fop.apps.StreamRenderer.render(StreamRenderer.java:265)\n\tat org.apache.fop.fo.FOTreeBuilder.endElement(FOTreeBuilder.java:233)\n\tat org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)\n\tat org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)\n\tat org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\n\tat org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)\n\tat org.apache.xerces.parsers.XMLParser.parse(Unknown Source)\n\tat org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat org.apache.fop.apps.Driver.render(Driver.java:634)\n\tat org.apache.fop.apps.Driver.render(Driver.java:619)\n\tat org.apache.fop.apps.Driver.run(Driver.java:730)\n\tat org.apache.fop.apps.Driver.renderFo(Driver.java:774)\n\tat ifs.application.reporting.rendering.PdfServices.renderFo(PdfServices.java:392)\n\tat ifs.application.reporting.rendering.PdfServices.renderPdfAfterTrasformation(PdfServices.java:102)\n\tat ifs.application.reporting.rendering.RenderingService.generatePdf(RenderingService.java:283)\n\tat ifs.application.reporting.rendering.RenderingService.generateReport(RenderingService.java:259)\n\tat ifs.application.reporting.rendering.RenderingService.getResult(RenderingService.java:173)\n\tat ifs.application.reportingservices.rendering.RenderingService.generatePdfFromPrintJob(RenderingService.java:180)\n\tat jdk.internal.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)\n\tat java.base\/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.base\/java.lang.reflect.Method.invoke(Unknown Source)\n\tat com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:656)\n\tat com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)\n\tat org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:101)\n\tat com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:273)\n\tat org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:213)\n\tat com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:444)\n\tat org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:112)\n\tat org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)\n\tat org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)\n\tat org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)\n\tat org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)\n\tat org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:277)\n\tat com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:137)\n\tat com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:146)\n\tat com.ibm.websphere.jaxrs.server.IBMRestServlet.doPost(IBMRestServlet.java:104)\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:706)\n\tat com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:96)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1258)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:746)\n\tat com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1227)\n\tat com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1011)\n\tat com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:75)\n\tat com.ibm.ws.webcontainer40.servlet.CacheServletWrapper40.handleRequest(CacheServletWrapper40.java:85)\n\tat com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:938)\n\tat com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:281)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1184)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:453)\n\tat com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:412)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:566)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:500)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:360)\n\tat com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:327)\n\tat com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:167)\n\tat com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:75)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)\n\tat com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)\n\tat com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:238)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base\/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base\/java.lang.Thread.run(Unknown Source)","ibm_threadId":"0000127f","ibm_datetime":"2023-04-25T14:46:57.192+0000","module":"SystemOut","loglevel":"SystemOut","ibm_methodName":"","ibm_className":"","ibm_sequence":"1682434017192_0000000027BE3","ext_requestID":"AARQOe2ODcO_AAAAAAAAJb\/","ext_thread":"Default Executor-thread-79"}