Nov 172010
 

Suppose you imported XML data into an InDesign document. Suppose that the layout should convey the markup’s semantics: keywords in italics, proper names in small caps, block quotes indented etc.

There are several ways how to map the markup to the layout. But it is important to know: no matter how you’ve mapped it, once the mapping has taken place, markup and layout information may evolve in totally different directions. And this is dangerous. For real-world XML document types and real-world typesetting, mapping is a one-way street, leading from markup to layout, and not the other way round, as we’ll see later. If you trust that after carrying out author corrections, everything that looks like a keyword will be a keyword in the exported XML, you may be proven wrong later. Or the two paragraphs that you see in InDesign are still a single one in XML, because it has been split only visually after import and the markup hasn’t been updated accordingly.

One of the main rationales behind XML-first workflows is that the correctness of the tagging should be verified by proofreading the main rendering, which is traditionally for print. This is based on two major assumptions: that every relevant tagging will be rendered distinctively, and that everything that is rendered in a certain way is tagged in a corresponding manner. There is no guarantee that especially the latter assumption is satisfied in popular InDesign XML workflows.

Consider a publisher’s workflow where manuscripts are being normalized in Word, then converted to XML and imported into InDesign. An equivalent scenario will be a workflow where manuscripts are being edited in an online editor, saved as XML and then imported. A typesetter will refine the page makeup then, a proof (PDF or printout) will be sent to the author, and corrections are being carried out in InDesign.

Some typical author corrections:

  1. make italic words upright, or vice versa;
  2. split a paragraph into two, or join two paragraphs, or insert new paragraph;
  3. add a new image with a caption.

These corrections will be carried out meticulously, with the following results:

  1. The words in italic lack the corresponding XML markup;
  2. in terms of tagging, the split paragraph still looks as a single one;
  3. the image and its caption don’t appear at all in the XML export.

What’s going wrong here? Apparently formatting changes and newly created text frames don’t find their way to the underlying XML structures automatically. This is because mapping occurs during import, from XML to formatting, but not the other way round.

Technically it’s because layout and markup information are two independent sets of information that sit on top of the textual content. The software designers of InDesign probably thought that a typesetter’s styling doesn’t affect the markup. In database publishing, for example, this may be a valid assumption. Therefore they don’t even issue warnings when there is untagged content or when character or paragraph formatting deviates from the formatting that has been created initially according to the mapping instructions. A warning in InDesign could look like the pink backgound color that indicates missing fonts. Lack of such a warning is an omission that may be fixed easily if there was some priority to it.

To give full credit, the developers indeed conceived a way of mapping styles to element names (“tags”). This will work for named paragraph or character styles only – fair enough. But this mapping is so absurdly limited that it cannot accommodate real-life document types with all kinds of attributes that distinguish different paragraph types, or with content grouped in boxes or in a hierarchy. You could only use this mapping to create a trivial XML format that will have to be transformed into the real thing later, for example by applying XSLT on export. This in turn is limited to XSLT 1.0 where important concepts such as grouping are not as readily available as in XSLT 2. In addition, the typesetter will still be able to use explicit ad-hoc formatting that looks like there’s some semantics behind the appearance, but actually it isn’t because plain italics won’t map to the keyword tag. For these reasons, we think that this rather trivial mapping during editing, and the whole idea of carrying simplified XML with you during editing, could be dispensed with, in favor of processing an IDML export of the final document, as described below.

But returning to the case where you have mapped an article’s real-life complex XML to InDesign formatting and you’ve started applying author corrections. Even if InDesign warned about deviating XML/formatting, somebody would have to assume the responsibility of making appearence and tagging coherent. And this is extra work, vulgo costs, that these InDesign XML first workflows entail. It’s not exactly the dreaded “double data set” redundancy, but it’s a “double piggyback information” problem. Corrections to the main text’s wording don’t have to be carried out twice, fortunately, but corrections to semantic or structural meta-information have to be carried out in any of the two piggyback information sets, namely formatting and XML.

So what can be done to eliminate the extra work? There are two principal paths: either

  • change your workflows, or
  • do some scripting.

None of them are a walk in the park.

Note: Updating the underlying XML and re-importing is not an option if substantial manual makeup corrections have already been carried out. Nobody wants to redo the page makup of 400 pages for 40 corrections that affect tagging.

When going the scripting path, autocorrecting markup that deviates from layout, there are two principal alternatives:

  • native API scripting (Javascript, Applescript, Visual Basic);
  • IDML processing (XSLT 1, XSLT 2, or APIs such as IDMLLib).

In any case, there is no assurance that you export XML data that reflect all alterations made during the proof stage. But you can cover common cases such as untagged stories/ stories in unanchored frames, character styles that deviate from what would be mapped, split/joined paragraphs or tabbed tables.

When deciding whether to use native API scripting or IDML processing, there are two important things to consider: maintainability and performance.

Maintainability means that you don’t want to hand-craft and test your autocorrection script for every single layout and XML tag set. The script should introspect the layout and the mapping instructions, and autocorrect as much of the tagging as possible.

There are several different mapping methods around. We are using mostly the aid attribute approach and IDML synthesis with prior aid attribute generation. The latter is the most versatile way to generate layout from markup. In contrast to other mapping methods, it permits scriptless assignment of object styles in IDML files, where “scriptless” should be read here as “without native scripting”. And we prefer XSLT 2 processing of IDML exports for autocorrection. One of the reasons is that it is very maintainable: you can read all the style definitions in the IDML file as well as the actually applied styles and the styling that was originally requested by the XML tags’ aid attributes. If there’s deviating styling or no tagging at all, we may auto-insert the appropriate XML tagging that corresponds to the styling in place.

We are using XSLT 2.0 for this. We cannot recommend using XSLT 1.0 because of the lack of conciseness and maintainability. And we think that for XML processing, XSLT does a significantly better job than procedural, object-oriented languages such as Javascript and Java. Therefore we use the IDML format itself as an API, not the procedural/OO APIs that InDesign or IDMLLib offer.

Another important reason for us not to use InDesign’s API is performance. Given the hours that we waited for CS4 or CS5 to finish scripted postprocessing of a 700-page XML import, an issue that ultimately forced us to switch to IDML synthesis, we cannot imagine that native scripting performs any better at checking and autocorrecting the tagging of 10,000 or more text extents.

We never used IDMLlib, but I think that it compares to native scripting as the Aspose.Words library compares to Word VBA scripting. Aspose.Words is a Java lib that runs on the file format and not within the native application, as is IDMLlib, and it is really fast. But also in the .docx case, I prefer XSLT 2 processing to Java processing. This is because when you break down a complex transformation to multiple steps, you will have to design an intermediate object model in Java for each step, and the transformations are also easier to code in XSLT 2. So you have much shorter and still very comprehensible code there. So maintainability and performance are the main reasons why we are using XSLT 2 processing of IDML exports for autocorrection.

Change your workflows?

So far, we’ve only been occupied with the “do some scripting” remedy. How about changing the workflows?

We saw that if there were no author corrections after fine-tuning page makeup, we wouldn’t have to worry about syncing deviating layout and markup information.

A possible approach is to make the author carry out all corrections in the source data prior to XML conversion or import. For example, make the author correct a Word manuscript before conversion to XML, or correct XML chunks in a content management system. But many authors cannot do without the impression of the final print layout, and often this is actually important because of page count estimations or because the typesetters have messed up manuscripts before and the authors need to check the conversion quality.

A solution to this is to generate galley proofs with a rough-cut page makeup, by just importing the XML or opening a generated IDML in InDesign, and then maybe apply some (few) scripts, just for figure placement or for splitting text boxes that stretch beneath the type area. An author will discover almost all issues of missing keywords or images, overly long paragraphs that need to be split, half a page that needs to be eliminated for page count reasons, …

The corrections will be carried out in some master data (for example, Word or XML), and only then will the typesetter start to fine-tune the layout. In most cases these layout tunings will be alterations that won’t affect tagging. So, apart from the inevitable last-minute corrections that will have to be carried out in the underlying XML data and in the layouted InDesign file, there’s no necessity to export a correctly tagged XML from InDesign after the print PDF has been finished. This is still a valid XML-first workflow.

All other renderings, such as EPUB (which, btw, you still don’t want InDesign to generate) or online database content, will be derived from the same quality-assured XML that flows into InDesign, not from the quality-dubious XML export that the currently practiced XML-first or round-trip workflows naïvely spit out.

Again, Performance

Another strong case for this “rough-cut first” approach is again performance. Two recent projects have been so large (> 700 pages without hard page break, > 300k XML elements) that InDesign simply couldn’t finish XML import. It couldn’t even open an IDML file if it contained piggyback XML data. So due to the lack of imported XML, exporting XML after carrying out author corrections was a “noption” and we generated the IDML without the tagging.

Word InDesign XML round-trip workflow

Word InDesign XML round-trip workflow (sorry about the German labels, they say “Author | Copy Editor | Data Converter | Typesetter | Printer | End Customer”, “.doc with track changes”, “.doc for 2nd edition”, the others should be self-explanatory)

Finally

Does it have to be XML first at all? Our typesetting staff that executes the above-depicted XML-first workflow for relatively shallow-structured fiction books is dissatisfied with all the coordination overhead involved when another department (technical copy editors who work on the Word files) normalize the manuscript and yet another department (XSLT people) convert it to DocBook with aid attributes.  They asked whether we could replace the Word normalization, XML import/export process to an InDesign normalization, XML export process. And I think given the heterogeneity of input data and the high degree of normalization they achieve, this isn’t a bad idea. The ex-post XML generation process is conceptually the same as autocorrecting the tagging heuristically according to the actual formatting, with the starting condition that there is no tagging at all initially.

In such a workflow it’s pivotal that the typesetters only use accepted named styles and accepted custom formatting. This may be checked within the XSLT 2 autocorrection stylesheet, by a separate custom Schematron rule set or by a purpose-built IDMLLib program. These routines that check manuscript or typeset data for “structurability” are another article’s subject.

Finally finally, does it have to be InDesign at all? Consider alternative typesetting systems. In LaTeX, XSL-FO or 3B2 many layout things may be influenced by processing instructions (PIs). In LaTeX and XSL-FO, you don’t have any other means than PIs for adding custom layout instructions. This means that because this XML input file is the only source of wisdom for adjusting the layout, there is no room for layout data that will evolve differently than the tagging. On the other hand, for example in LaTeX, you could do all kinds of nasty things in processing instructions, up to replacing the entirety of the default rendering with custom layout and content that is included in a processing instruction. These close-to-criminal offences happen in InDesign, too, and in our view, there’s no other way to suppress them than to visually compare a genuine rendering of the actual unspoiled XML content against the print rendering.

So why does it have to be InDesign so often? The kind of books I’ve been talking about rarely use the full range of layout tweaks that InDesign offers. So a classic XML-aware book typesetting system should be acceptable to the publishers. The choice of the renderer for print output shouldn’t be that important, provided that there’s a choice of service providers that support this typesetting system.

I think the preference towards InDesign that I perceive among many production editing executives is mainly a cultural phenomenon, as many people’s preference for MS Word has been for a long time.

This post is also available in: German

  20 Responses to “XML-first Workflows in InDesign: Ropes of Sand?”

  1. […] This post was mentioned on Twitter by Matthew Diener and Julieta Lionetti, Araman Consulting. Araman Consulting said: Interesting article about using #XML & InDesign: "XML-first Workflows in InDesign: Ropes of Sand?" http://bit.ly/fSIzmf #eprdctn #idml #xslt […]

  2. Very interesting post. InDesign is not a patch on the old FrameMaker functionality, especially when it comes to round-tripping changes. Our solution to this has been to see InDesign as an output path only (subject to the same caveats you list above) and to try to make the process of making changes upstream and re-flowing the layout as easy as possible. One helpful thing here is generating ICML instead of IDML, which is a much lighter content type, and it allows you to leverage Adobe’s InCopy workflow tricks (like updating copy in place) to a certain extent.

  3. I completely agree on what you describe in this very interesting post, but I would respond in a different way to your concluding question: “So why does it have to be InDesign so often?”

    Answer #1 (technical)
    Because XSL-FO cannot compete with InDesign in terms of typographical quality: even for a simple book (one column/flow) with footnotes and figures you cannot have (even using high quality FO processors like Antenna House) decent horizontal justification (no track control, hyphenation nightmare if full justification enabled) or vertical justification (poor baseline or facing pages alignment).

    Answer #2 (organizational, as usual more important than technology)
    Because print production people are not used (or do not want to switch, probably rightfully) to a batch pagination paradigm. If you need to move a figure to a top recto page you want to do it by yourself using the mouse, not by inserting a PI of some sort in the xml source and regenerating the pdf (and repeating the whole process for every change you want in the layout).

    My conclusion (from direct experience) is that if you want high quality print output you cannot have also xml-roundtripping, which leaves only the xml-last option (and therefore I hope to read soon your article on the “structurability of manuscript or typeset data”).

    This blog seems very promising, please keep it alive.

    • I think XSL-FO could compete with with InDesign in terms of typographical quality if someone whould write some software that uses InDesign as XSL-FO formatter. I think this is the missing link.
      Did anyone try that?

      • Yes, it has to be InDesign for print work, and that’s not a cultural bias. Even its closest competitor for print layout, Quark XPress, is no competition.

        There are InDesign plugins that (for example) update source data when a price in a catalogue is altered in layout. I don’t see why the same kind of two-way communication couldn’t happen with XML source data.

        Of course, then you’d have to trust your InDesign operators…

  4. This is a great post that brings to light many of the issues that Typefi Systems has been working to solve for the last ten years. We believe the necessity of publishers to generate multiple outputs at the same time and the challenging economic climate demand an XML first workflow. With an XML first workflow and automated composition with something like Typefi Publish publishers can gain the benefits of XML first (content re-use, single-source publishing, and content fidelity) and still create highly designed print output. With new tools do come new workflows, that really should accelerate the entire process. After all, you would not use a hydraulic hammer the same way you use a manual one to build a house.

    Our customers leverage editorial and markup tools such as Inera eXtyles or our Writer (both of which work in the ubiquitous MS Word) to create XML. This content is then passed through a Typefi template created in InDesign. The document is dynamically laid out to create an InDesign and PDF file, which can be automatically returned to the editor for review within seconds or minutes. The editor does not even need InDesign on the desktop. Many of our customers achieve 85%-95% of the final output at first pass. If there are content changes, they can be made in the Word document and re-run at no additional “cost.” Editors can also provide immediate feedback on changes to design as well which can be applied to the template without programming.

    This is often the most challenging part for new customers to grasp. So many people in InDesign workflows hold on to the concept that once in InDesign they need to stay there, which makes sense in a manual workflow. “I just waited two days to six weeks for a first pass, I don’t want to wait that long again.” But if you can get that first pass back in seconds I can make corrections right there in Word and run it again. I can also provide instant feedback to the designer for structural changes, which promotes better collaboration.

    Finally, changes can be made to the InDesign file. We break down these changes into two kinds, design changes that do not impact the XML (such as moving tables and figures) or content changes, such as adding, changing or deleting text. Design changes are either made to the template or the final InDesign file with no problems. Our customers try to avoid as much as possible making content changes in InDesign which is much easier since it only takes them seconds to minutes to create a new InDesign file. But if minor last-minute changes need to be made to the content, that can be round-tripped, but you do have to be careful as InDesign is not a validating environment.

    So our customers gain the benefits of InDesign, phenomenal typesetting and the ability to hand-touch files with the speed of batch-pagination. The cost is a little more planning and discipline up front. But as my old boss used to say, “Nobody has time to do it right the first time, but they always have time to do it twice.”

    • Chris, thanks for your insightful feedback.

      Our product le-tex aspect operates in a way similar to what you describe: editing in plain MS Word, saving/uploading to aspect, rule-based checking, conversion to from OOXML to IDML and optionally from there to PDF for a rough page makeup.

      As you say, the main challenge is to convince production editors and authors to make content changes in a document format that they can actually edit. And to accept galley proofs without the last 5% of layout refinement before the content is finalized.

      Unfortunately, many of our customers and their authors are loath to accept this paradigm shift. They expect to receive a thoroughly typeset proof, and then they start submitting corrections.

      Round-tripping these content/markup corrections is generally tough to set up, as you confirmed. I recently saw presentation slides by one of your colleagues, about a journal workflow. They carried out last-minute corrections in two data sets in parallel, the Word and the InDesign file.

      This is only acceptable if you can get authors either to use such a system or to accept roughly layouted galley proofs. Otherwise there will be too many redundant operations which will make the whole think costly and error-prone.

      Using such a system requires strict adherence to the Word template and its checking rules. This is a thing you cannot expect every author to accept, or they are not able to install or use add-ons or even Word templates. Of course if the balance of power tilts towards the journal publishers, they can force their rules upon the authors.

      As regards rough galley proofs: in our experience, neither authors nor production editors like them. You cannot make them understand which of the rough edges just result from batch processing and lack of refinement, and that everything will look smooth after final typesetting adjustments.

      So: lucky are the publishers that could convince their authors to either upload Word themselves or accept rough proofs. And poor are the typesetting service providers who have to deal with the rest of them and with publishers that want InDesign, XML and “traditional” proofing.

      • Yes, the rough proof is a great idea. As long as it doesn’t LOOK like a real galley (eg, it’s in Courier 11 without any ornaments or flourishes), it strikes me as an effective tactic.

  5. […] Imsieke von le-tex hat unter XML-first Workflows in InDesign: Ropes of Sand? eine lesenswerte Zusammenfassung über mögliche Herangehensweisen von strukturierten Daten und […]

  6. This post has confirmed a lot of what I’ve been thinking about XML with InDesign. It feels like Adobe’s support for XML is half-assed – in some places it’s really powerful, but there are a lot of things they just haven’t thought about.

    I recently posted on the InDesign Secrets forum about my problems getting XML *out* of InDesign – trying to get a complex document to match a complex DTD is a lot of work. I don’t know what the best solution for this is though.

    I’ve subscribed to your blog – looking forward to a lot more interesting posts :)

    • @Nick Morgan: I tried to check your post at InDesign Secrets, but looks link is not working. Anyway, we’re extracting the XML files from InDesign even after doing many rounds of corrections in InDesign. We have been educated the InDesign users as well as by deploying the scripts to tackle any type of corrections without disturbing the XML coding which is embedded in the InDesign. If required, you can write your actual problems, so that I can explain you how we’re approaching the same in InDesign with XML.

  7. Thanks for the great post, and for the informative comments.

    What are your thoughts on moving away from InDesign entirely and using XHTML (strict) and CSS instead; this seems the right trajectory in this digital age. Is this feasible for print? (Notwithstanding the fact that print designers would have to learn web design and development. Alternatively, reskill web designers.] Can intelligent use of the Class attribute give sufficient semantic granularity for multiple formats and channels?

    River Valley Technologies, Richard Pipe and the Fraser University prototype all appear to be doing interesting work in this general area.

    • Well, I think, plain XHTML+CSS is semantically underdeveloped for typical publishing constructs, such as footnotes, index entries, citations, and figure captions. HTML5 is improving a bit in certain regards, and EPUB3 puts an RDFa layer on top of HTML5 in order to be able to mark up some publishing terms semantically. And if there is a recommended semantic markup, you need to enforce its use, for example, by disallowing some layout-only style attributes in HTML/CSS, or by requiring hierarchical nesting instead of plain h1/h2/… headings. One tool to enforce this might be Schematron, where you can define specific rules for your workflow. A combination of configurable online editor, server-side Schematron validation, and server-side preview rendering will be a good choice for many workflows. But it will of course work with other XML vocabularies. An advantage of XHTML over arbitrary XML is that there are no good generic open-source XML editors around. It’s an advantage if you can include TinyMCE or FCK Editor in your workflows, without the need to convert the data back and forth from XML to HTML. The automatically generated print layout might suffice for many purposes, but if a really neat page makeup is required, a professional typesetter might import the XHTML/XML data to a typesetting system. But then, still, there might be last-minute corrections that need to be retrofitted into the XHTML/XML data.

  8. As Anna has raised the question of XHTML and CSS, it is worth looking at the real reason that InDesign was developed: for beautiful designs in ink on paper. At present there is NO alternative for creating CMYK ink separations than the “desktop publishing applications”, if we assume the publisher requires print-oriented output controls for fine gradients, shadows, transparencies, etc. Then there is the extremely wide range of typographic controls at the fingertips of the InDesign layout person. There is such sophistication in the output from InDesign that it would be murderously complex to replicate with other technologies. It would make an XSL-FO developer weep in their beer.
    So that begs the question, why put ink on paper? In an all-digital delivery situation, there is no need for InDesign. The key problem is we are working across two different worlds of content creation, design and output and there is no simple bridge between them. XML is supposed to be “presentation agnostic”, which means nothing to a publisher.
    Everything that bridged the gap so far is some kind of workaround to deal with InDesign’s difficulty with hierarchical content. The IDD programming object model is simply too limited. We have only master page, spread, table, graphic, text and some limited media object types such as video and audio that can be “called by name” in InDesign. Styles and fonts can be related to these objects, but the text objects are a very flat “structure” of text frame, paragraphs and inlines in X-Y coordinate relationship to the page boundaries. There was never any need in print design to have “n” levels of text structure in parent-child or sibling relationships.
    Working between the “flat” InDesign and the “deep” XML runs up against the classic problems of “upcasting” IDD to XML (adding structure that is missing) and “downcasting” XML to IDD (removing unnecessary levels of structure). It is much easier to “downcast” than to “upcast”. Upcasting requires some type of coding assumptions about the desired hierarchical relationship of the XML that can only be inferred from the flat structure of the IDD content.
    So when you import and map XML to IDD styles, you are writing a rule from a deep structure to a shallow one. When you edit text in IDD to add a “new paragraph”, it is only in the shallow structure of the IDD objects. To correctly write the text edit back to XML, IDD would have to be able to know what XML element is permitted at the location where you put in a paragraph return. Going from IDD to XML presents too many possible structures at any given point for a single set of algorithms to work.
    In short, don’t expect miracles and you won’t be disappointed. The XML structure in IDD is a mirage, not a real, working dynamic part of the IDD file.

  9. At River Valley, we have taken a different approach, which is to use 30-year old technology (TeX) as our pagination engine. We convert any files that we get into “structured TeX”, i.e. use TeX as a tagging tool to start with, so

    \firstname{John}
    \surname{Smith}

    (TeX is mark-up, but the simplest mark-up, so designed to be readable. So operators can work direct on “tagged” text, not via a front end.)

    Then we run this through TeX (with TeX4ht) to get XML with any DTD. (We need to configure TeX4ht for this.) From that, we create another temporary TeX file, then a PDF.

    Any changes to the text are put in the Structured TeX file and the file rerun through system. So:

    Word etc > Structured TeX > XML > TeX > PDF, ePub etc

    Any visual commands, like spacing to correct an anomaly, are put in the Structured TeX and saved as Processing Instruction in XML, then pop out in the second TeX file.

    Advantages:

    — Free, open software, so not dependent on the whims of Adobe, MS, etc.
    — *True* XML-first
    — Handles the most complex math and tables (I think no other solution to this one)
    — Unparalleled typography — InDesign copied the TeX line break algorithm. ;-)

    Disadvantages:

    — Bit geeky
    — No wysiwyg control of output
    — Not suitable for “coffee-table” books or design needing page by page visual tweaking.

    • Dear Kaveh,

      Thanks for your remarks. It’s always nice to see TeX being used for publishing production.

      For many workflows, we also use TeX. It depends on the standardization / “coffe-table-ness” of the layout and on a cultural phnomenon: many production editing executives stipulate that InDesign is being used, even though there is no other reason than that they’re comfortable with that tool.

      We have different TeX workflows. If fulltext XML is a deliverable, most of our workflows are XML first, either

      1. Word → XML (with XSLT 2)
      2. xmltex

      or

      1. LaTeX → XML (using latexml; we also used tex4ht but found latexml more useful, even though it is slightly more complex)
      2. xmltex

      I noticed River Valley before and thought: “Oh, we seem to share the same mindset in several regards.”

      • Yes, many similarities, Gerrit. I have met your colleagues several times. Last time I went past your booth at the Frankfurt book fair. I don’t know what they were showing but they caused a traffic jam, so I couldn’t talk to them. ;-)

  10. I am hoping someone can dumb this down for me because I am not familiar with most of these terms. I am trying to put a word document of around 100 pgs, already formatted, into an InDesign file. however, when I do, it changes its format and jumbles up the words and placement. I am wondering how I can place a word document into InDesign while maintaining its format. Anyone think they can help me? It is much appreciated.

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>