Prince doesn't have a quirks mode at the moment, so the only distinction is between XML/XHTML documents and HTML documents. By default, Prince will auto-detect the document format by seeing if it has an <xml> declaration, or uses namespaces, but you can override this using the --input option on the command-line or the setHTML() method in the API.