[QDOX-183] Parser Errors when processing jdk-src - jira.codehaus.org

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.11
Component/s: None
Labels:
None
Environment:
QDox 1.10
Windows XP
Java 1.6

Number of attachments :
0

Description

When I try to parse the JDK sources with qdox-1.10 I get a bunch of errors. I have no idea where they come from?

1. get http://www.java.net/download/jdk6/6u16/promoted/b01/jdk-6u16-fcs-src-b01-jrl-31_jul_2009.jar (132MB)
2. Run the following Java Class (make sure to give enough memory to the JVM: "-Xmx1024M"):

import java.io.File;

import com.thoughtworks.qdox.JavaDocBuilder;
import com.thoughtworks.qdox.directorywalker.FileVisitor;
import com.thoughtworks.qdox.parser.ParseException;

public class MainTest {

    /**
     * @param args
     */
    public static void main(String[] args) {
       JavaDocBuilder builder = new JavaDocBuilder();
       builder.setErrorHandler(new JavaDocBuilder.ErrorHandler() {
           @Override
           public void handle(ParseException parseException) {
               System.err.println("Exception while parsing file: " + parseException.getMessage());               
           }
       });
       builder.addSourceTree(new File("C:\\apps\\jdk-1_6-src\\j2se\\src\\share\\classes"));
    }
}

3. I get the following errors: (some of the nio-files are not valid java as they contain pre-processing directives but others like the mbeans-stuff seems to be fine)

Exception while parsing file: syntax error @[145,21] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/jmx/mbeanserver/MBeanSupport.java
Exception while parsing file: syntax error @[67,29] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/jmx/mbeanserver/MXBeanSupport.java
Exception while parsing file: syntax error @[100,36] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/jmx/mbeanserver/StandardMBeanSupport.java
Exception while parsing file: syntax error @[37,5] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/tools/javac/code/Source.java
Exception while parsing file: syntax error @[280,1] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/xml/internal/stream/writers/XMLWriter.java
Exception while parsing file: syntax error @[782,39] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/xml/internal/ws/api/addressing/WSEndpointReference.java
Exception while parsing file: syntax error @[170,46] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/com/sun/xml/internal/ws/server/AbstractInstanceResolver.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/ByteBufferAs-X-Buffer.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/charset/Charset-X-Coder.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/Direct-X-Buffer-bin.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/Direct-X-Buffer.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/Heap-X-Buffer.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/X-Buffer-bin.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/nio/X-Buffer.java
Exception while parsing file: syntax error @[6581,1] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/java/util/Formatter.java
Exception while parsing file: syntax error @[209,53] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/javax/management/monitor/Monitor.java
Exception while parsing file: syntax error @[342,45] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/javax/management/openmbean/OpenMBeanAttributeInfoSupport.java
Exception while parsing file: syntax error @[280,45] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/javax/management/openmbean/OpenMBeanParameterInfoSupport.java
Exception while parsing file: syntax error @[146,36] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/javax/management/StandardEmitterMBean.java
Exception while parsing file: syntax error @[317,29] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/javax/management/StandardMBean.java
Exception while parsing file: syntax error @[72,1] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/org/omg/CORBA/CompletionStatus.java
Exception while parsing file: syntax error @[54,1] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/org/omg/CORBA/DefinitionKind.java
Exception while parsing file: syntax error @[120,39] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/sun/rmi/transport/tcp/TCPTransport.java
Exception while parsing file: syntax error @[42,38] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/sun/tools/jconsole/inspector/Utils.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/sun/util/CoreResourceBundleControl-XLocales.java
Exception while parsing file: syntax error @[8,2] in file:/C:/apps/jdk-1_6-src/j2se/src/share/classes/sun/util/LocaleDataMetaInfo-XLocales.java

Options
- Show All
- Show Open

Sub-Tasks

1.	Exception #1	Closed	Unassigned
2.	Exception #2	Closed	Robert Scholte
3.	Exception #3	Closed	Unassigned
4.	Parser exception due to 'final' before an annotation used as methodparameter	Resolved	Robert Scholte
5.	Parser exception due to double javadoc comment	Resolved	Robert Scholte
6.	Parsing exception due to operator in constructor of a new Object	Resolved	Robert Scholte
7.	Parse exception for Generic constructors	Resolved	Robert Scholte

Activity

Hide

Permalink

Marco Soldati added a comment - 22/Nov/09 4:28 PM

the failures are caused by 6 different exceptions.
I created one sub task per exception and listed all files that are affected.
I know it is not exactly what you wanted from me, but I'm afraid I don't have time for a more detailed analysis.

Show

Marco Soldati added a comment - 22/Nov/09 4:28 PM the failures are caused by 6 different exceptions. I created one sub task per exception and listed all files that are affected. I know it is not exactly what you wanted from me, but I'm afraid I don't have time for a more detailed analysis.

Hide

Permalink

Robert Scholte added a comment - 23/Nov/09 11:57 AM

You guessed right, like this there's not much we can do

Although a superb QDox would we able to parse all these files, building a model with the complete java-api sources looks kind of weird to me. So maybe you could give some background info why you need it, maybe we can offer a better solution.

But apart from that: All you subtasks refer to file on your machine. I've tried to install it, but already with the first file the line in which the error seemed to occur didn't exist. The @[line, column] should give you a hint where QDox fails.

And it's not that hard to create a testcase. Let me provide you a simple template:

public void testTemplate() throws Exception {
  JavaDocBuilder builder = new JavaDocBuilder();
  String source = "public class Foo {}";
  builder.addSource( new StringReader(source) );
}

With the syntax errors you found this should be enough, because an exception will be thrown and the test fails.

Hopefuly you can find some time to specify your cases.

Show

Robert Scholte added a comment - 23/Nov/09 11:57 AM You guessed right, like this there's not much we can do Although a superb QDox would we able to parse all these files, building a model with the complete java-api sources looks kind of weird to me. So maybe you could give some background info why you need it, maybe we can offer a better solution. But apart from that: All you subtasks refer to file on your machine. I've tried to install it, but already with the first file the line in which the error seemed to occur didn't exist. The @ [line, column] should give you a hint where QDox fails. And it's not that hard to create a testcase. Let me provide you a simple template: public void testTemplate() throws Exception { JavaDocBuilder builder = new JavaDocBuilder(); String source = " public class Foo {}" ; builder.addSource( new StringReader(source) ); } With the syntax errors you found this should be enough, because an exception will be thrown and the test fails. Hopefuly you can find some time to specify your cases.

Hide

Permalink

Robert Scholte added a comment - 06/Dec/09 4:59 AM

I´m thinking of closing this issue (and it's subtasks) as incomplete by the end of the year since there are still no valid code examples.

Show

Robert Scholte added a comment - 06/Dec/09 4:59 AM I´m thinking of closing this issue (and it's subtasks) as incomplete by the end of the year since there are still no valid code examples.

Hide

Permalink

Marco Soldati added a comment - 06/Dec/09 1:39 PM

no. wait please. I'll come back to you asap.
there is the usual end-of-year pressure, hence i'm a bit short of time. sorry for the delay.

Show

Marco Soldati added a comment - 06/Dec/09 1:39 PM no. wait please. I'll come back to you asap. there is the usual end-of-year pressure, hence i'm a bit short of time. sorry for the delay.

Hide

Permalink

Robert Scholte added a comment - 06/Dec/09 2:27 PM

Ok, I'll give it some more time. I guess the example is enough for you reproduce the cases. If not, just let us know.

Show

Robert Scholte added a comment - 06/Dec/09 2:27 PM Ok, I'll give it some more time. I guess the example is enough for you reproduce the cases. If not, just let us know.

Hide

Permalink

Robert Scholte added a comment - 10/Jan/10 6:41 AM

All sub-issues have been resolved. I've deployed a SNAPSHOT as well, so if you have a repository-entry to http://snapshots.repository.codehaus.org/ you could give it a try.
But still I have this big question: why-o-why parse the complete jdk-source?

Show

Robert Scholte added a comment - 10/Jan/10 6:41 AM All sub-issues have been resolved. I've deployed a SNAPSHOT as well, so if you have a repository-entry to http://snapshots.repository.codehaus.org/ you could give it a try. But still I have this big question: why-o-why parse the complete jdk-source?

Robert Scholte made changes - 10/Jan/10 6:41 AM

Field	Original Value	New Value
Fix Version/s		1.11 [ 16104 ]
Assignee		Robert Scholte [ rfscholte ]
Status	Open [ 1 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Hide

Permalink

Marco Soldati added a comment - 10/Jan/10 7:39 AM

Wow, that was quick! Thanks a lot!
I reran my tests and everything works smoothly now.
I also ran the latest snapshot against some other java libraries that I had installed on my system: spring-framework-2.5.6, saxon9-2-0-2j, apache-fop-0.95. Works all like a charm. Many thanks again.

Now to your question why I do all of this:
I'm working on a JavaScript-based API viewer. It bases on JSON-files that store the content of the API. I need a JavaDoc parser to create these JSON files. Of course I could use Sun's JavaDoc tool but QDox seems to be more lightweight. I guess QDox is also faster which is important as the JSON files are built frequently. I plan to do some rudimentary performance tests soon.

While developing my viewer I use the JDK sources as my most complex test case. They contain almost any "hack" that is possible with JavaDoc.

Show

Marco Soldati added a comment - 10/Jan/10 7:39 AM Wow, that was quick! Thanks a lot! I reran my tests and everything works smoothly now. I also ran the latest snapshot against some other java libraries that I had installed on my system: spring-framework-2.5.6, saxon9-2-0-2j, apache-fop-0.95. Works all like a charm. Many thanks again. Now to your question why I do all of this: I'm working on a JavaScript-based API viewer. It bases on JSON-files that store the content of the API. I need a JavaDoc parser to create these JSON files. Of course I could use Sun's JavaDoc tool but QDox seems to be more lightweight. I guess QDox is also faster which is important as the JSON files are built frequently. I plan to do some rudimentary performance tests soon. While developing my viewer I use the JDK sources as my most complex test case. They contain almost any "hack" that is possible with JavaDoc.

Hide

Permalink

Robert Scholte added a comment - 10/Jan/10 10:07 AM

I already kind of expected this brute force test answer. But be aware that this type of method won't cover every possible bit of specification. And you might discover things you would never use (think of ~~QDOX-198~~). Also think of who is responsible for which part of the code. It looks like these tests were specially focussed on testing QDox instead of your own code.
Anyhow, I'm glad you're not using QDox to create just a model of the complete JDK for further production usage, that just wouldn't make sense. And with your issues we were able to improve QDox. So thanks for taking the time to provide the testcases.

Show

Robert Scholte added a comment - 10/Jan/10 10:07 AM I already kind of expected this brute force test answer. But be aware that this type of method won't cover every possible bit of specification. And you might discover things you would never use (think of QDOX-198 ). Also think of who is responsible for which part of the code. It looks like these tests were specially focussed on testing QDox instead of your own code. Anyhow, I'm glad you're not using QDox to create just a model of the complete JDK for further production usage, that just wouldn't make sense. And with your issues we were able to improve QDox. So thanks for taking the time to provide the testcases.

Robert Scholte made changes - 10/Jan/10 10:07 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:

Robert Scholte

Reporter:

Marco Soldati

Votes:

0 Vote for this issue

Watchers:

0 Start watching this issue

Dates

Created:

22/Nov/09 6:53 AM

Updated:

10/Jan/10 10:07 AM

Resolved:

10/Jan/10 6:41 AM