Exception when exporting PDF: No more DTM IDs are available

[concubicycle]

Hi!

I have some code that builds up a word document, and is supposed to save it as a pdf file. I have the following code that does the saving:

Code: Select all

// Refresh the values of DOCPROPERTY fields
FieldUpdater updater = new FieldUpdater(docxPackage);
updater.update(true);

// .. the FOSettings object
FOSettings foSettings = Docx4J.createFOSettings();
foSettings.setWmlPackage(docxPackage);

OutputStream os = new java.io.FileOutputStream(name);

// Don't care what type of exporter you use
// Docx4J.toFO(foSettings, os, Docx4J.FLAG_EXPORT_PREFER_XSL);

Docx4J.toPDF(docxPackage, os);


// Clean up, so any ObfuscatedFontPart temp files can be deleted
if (docxPackage.getMainDocumentPart().getFontTablePart() != null)
{
  docxPackage.getMainDocumentPart().getFontTablePart().deleteEmbeddedFontTempFiles();
}

This works fine for smaller pdf files (~170 pages), but gives the following exception for large pdf files:

org.docx4j.openpackaging.exceptions.Docx4JException: Exception exporting package
at org.docx4j.convert.out.common.AbstractExporter.export(AbstractExporter.java:109)
at org.docx4j.Docx4J.toFO(Docx4J.java:466)
at org.docx4j.Docx4J.toPDF(Docx4J.java:480)
at com.secdec.meercat.ui.reporting.AlertPDFTemplate$CreatePdf.run(AlertPDFTemplate.java:103)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.docx4j.openpackaging.exceptions.Docx4JException: Cannot perform the transformation
at org.docx4j.XmlUtils.transform(XmlUtils.java:1043)
at org.docx4j.XmlUtils.transform(XmlUtils.java:932)
at org.docx4j.convert.out.common.AbstractXsltExporterDelegate.process(AbstractXsltExporterDelegate.java:66)
at org.docx4j.convert.out.common.AbstractWmlExporter.process(AbstractWmlExporter.java:63)
at org.docx4j.convert.out.common.AbstractWmlExporter.process(AbstractWmlExporter.java:32)
at org.docx4j.convert.out.common.AbstractExporter.export(AbstractExporter.java:79)
... 4 more
Caused by: javax.xml.transform.TransformerException: No more DTM IDs are available
at org.apache.xpath.XPath.execute(XPath.java:365)
at org.apache.xalan.templates.ElemCopyOf.execute(ElemCopyOf.java:134)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1988)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1910)
at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:312)
at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:248)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:128)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemChoose.execute(ElemChoose.java:141)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1988)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1910)
at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:312)
at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:248)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemCopy.execute(ElemCopy.java:116)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemCopy.execute(ElemCopy.java:116)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1988)
at org.apache.xalan.transformer.TransformerImpl.transformToRTF(TransformerImpl.java:1910)
at org.apache.xalan.templates.ElemVariable.getValue(ElemVariable.java:312)
at org.apache.xalan.templates.ElemVariable.execute(ElemVariable.java:248)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:1376)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:1376)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.templates.ElemLiteralResult.execute(ElemLiteralResult.java:1376)
at org.apache.xalan.templates.ElemApplyTemplates.transformSelectedNodes(ElemApplyTemplates.java:395)
at org.apache.xalan.templates.ElemApplyTemplates.execute(ElemApplyTemplates.java:178)
at org.apache.xalan.transformer.TransformerImpl.executeChildTemplates(TransformerImpl.java:2400)
at org.apache.xalan.transformer.TransformerImpl.applyTemplateToNode(TransformerImpl.java:2270)
at org.apache.xalan.transformer.TransformerImpl.transformNode(TransformerImpl.java:1356)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:709)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1273)
at org.apache.xalan.transformer.TransformerImpl.transform(TransformerImpl.java:1251)
at org.docx4j.XmlUtils.transform(XmlUtils.java:1041)
... 9 more
Caused by: org.apache.xml.dtm.DTMException: No more DTM IDs are available
at org.apache.xml.dtm.ref.DTMManagerDefault.addDTM(DTMManagerDefault.java:148)
at org.apache.xml.dtm.ref.DTMManagerDefault.getDTM(DTMManagerDefault.java:256)
at org.apache.xml.dtm.ref.DTMManagerDefault.getDTMHandleFromNode(DTMManagerDefault.java:559)
at org.apache.xpath.XPathContext.getDTMHandleFromNode(XPathContext.java:184)
at org.apache.xpath.objects.XNodeSetForDOM.<init>(XNodeSetForDOM.java:44)
at org.apache.xpath.objects.XObjectFactory.create(XObjectFactory.java:142)
at org.apache.xpath.objects.XObject.create(XObject.java:180)
at org.apache.xpath.functions.FuncExtFunction.execute(FuncExtFunction.java:212)
at org.apache.xpath.XPath.execute(XPath.java:337)
... 62 more

Not sure how to fix this. It seems that this exception was common in older versions of xalan, but the one in my docx4j plugin is version 2.7.1. Heelp!

Thanks in advance.

[jason]

That this exception was common in older versions of xalan matches my recollection.

How many pages is your document?

If you can't make it work, I'd recommend you instead try Plutext's commercial converter, at http://converter-eval.plutext.com/
which should be a lot faster as well...

Sorry I can't be of more help right now.

[concubicycle]

Yeah, i read that it occurs for older versions of xalan and serializer. I'm using 2.7.1, which is one of the latest versions.

It begins to fail around 200 pages.

[jason]

I ran into a similar problem recently in the OpenDoPE BindingHandler step for a very large document.

I fixed it following http://www.mail-archive.com/xalan-j-use ... 04890.html

in
src/org/apache/xml/dtm/DTMManager.java, line 362, I found, read, and (I
hope) understood this comment:

/** This value, set at compile time, controls how many bits of the
* DTM node identifier numbers are used to identify a node within a
* document, and thus sets the maximum number of nodes per
* document. The remaining bits are used to identify the DTM
* document which contains this node.
*
* If you change IDENT_DTM_NODE_BITS, be sure to rebuild _ALL_ the
* files which use it... including the IDKey testcases.
*
* (FuncGenerateKey currently uses the node identifier directly and
* thus is affected when this changes. The IDKEY results will still be
* _correct_ (presuming no other breakage), but simple equality
* comparison against the previous "golden" files will probably
* complain.)
* */

I then changed the following constant's value from 16 to 12.

public static final int IDENT_DTM_NODE_BITS = 12;

Then I added a little static initializer purely to convince myself that
my webserver actually found my copy of xalan.jar

static {
System.out.println(" IDENT_DTM_NODE_BITS: "+ IDENT_DTM_NODE_BITS );
}

My understanding of this change is that I gave up four (16-12) bits
worth of IDs for identifying documents, in exchange for four more bits
of node IDs within each document. I cannot imagine in my app needing
more than 2^12 (4096) document IDs, but I certainly need more than 2^16
(65536) worth of nodes within a document. Now I can have up to 2^20
(1,048,576) nodes per document.

I got xalan from https://github.com/apache/xalan-j/tree/trunk

After making the change, build with ant clean ; ant dist

Then to use it via maven, install it to your local repo with mvn install:install-file (you'll need various args).

[jason]

docx4j 11.x will have this setting burnt in.

[mikemillerhms]

Are there any updates on this? I am hitting his on my 100+ page document when trying to generate the Table Of Contents.

Do to legal reasons I cannot send my documents to a 3rd party for PDF conversion.

[jason]

You can try https://docx4java.org/docx4j/xalan-2.7. ... E_BITS.jar

[mikemillerhms]

Cool, this worked, thanks.

FOP supports Saxon, Is there any plan to use that in the future so we don't have to rely on a hacked jar?

[jason]

docx4j can already use Saxon for XPath purposes: https://github.com/plutext/docx4j/blob/ ... l.java#L22

In forthcoming docx4j 8.0.0, Saxon can also be used for OpenDoPE content control data binding; see further https://www.docx4java.org/blog/2019/01/ ... th-2-03-0/

For XSLT (ie including PDF via XSL-FO), we plan to continue using Xalan, since we rely on its extension functions.

Looking ahead a little, docx4j for Java 11 will package its own Xalan, containing this DTM fix. See https://github.com/plutext/xalan-j/tree ... ged_Docx4j

[mikemillerhms]

Ahh, that makes sense. That should cover our needs.

Thanks!