![\begin{table}\begin{center}
\includegraphics[width=0.9\linewidth]{figs/overallData.eps}
\end{center}\end{table}](Timg74.png) |
Table 8.4 summarizes the responses from subjects when asked to rate the usefulness of the system on a scale from 1 (totally useless) to 10 (extremely useful), and if they would use the system as their daily programming environment. Although those evaluations are subjective, they are indications of the subjects' desire to use the system.
As both the quantitative data in Table 8.3 and the subjective evaluations in Table 8.4 show, CodeBroker has been quite effective in supporting programmers to locate and reuse components during the experiments.
``I would have never looked the roll function by myself, I would have done a lot of stuff by hand. Just because it showed up in the list, I saw the Calendar provided the roll feature that allowed me to do the task.''
``I did not know the isDigit thing. I would have wasted time to design that thing.''
The delivery mechanism not only supported subjects to reuse components right off the deliveries, it also created a snowball effect that triggered them to reuse other unknown components that were not directly delivered but were needed to reuse those delivered components. Components in the libraries of object-oriented programming languages are often coupled through parameter passing or accessing the common class variables. To reuse one component often requires the reuse of other components tightly coupled. In the experiments, when those coupled components were not known, programmers used the deliveries of CodeBroker as the starting point and then followed the existing hyperlinks of the documentation system to learn and reuse them.
The delivery mechanism also created latent reuse opportunities. Sometimes, the delivered components were not immediately reused because, although they were related to the task to some extent, they could not be directly reused right away for the immediate programming task in which the subjects were engaged. However, as programming continued, subjects realized that something delivered before could now be reused. For example, in one experiment, the subject was first concerned with finding how to read the contents of a file. Among the delivered components was the isDirectory method, which could not be reused right away for the task of ``reading the contents of a file'' but somehow caught the attention of the subject. Later, when the subject moved to the task of ``creating a new directory if it does not exist,'' he thought of something he had seen before, but he could not remember the name. So he asked if the system had a mechanism allowing him to go back to previous deliveries. When told not, he inserted a temporary comment to find the isDirectory method.
``It beats browsing. Because the way that I normally would have done the task, I would do a lot of browsing and then write the code alongside. So this reduced the browsing and searching.''
``Yes. First, I did not have to start browsing and go through the packages, and I did not have to go through the index of methods. I could just go to the short list [RCI-display], found it and clicked it.''
``I thought there might be a parse method, but I also was not sure whether it is called parse or something else. I also wasn't sure if it was in the Format class. Maybe it is in a different class like Integer or number or something else. It's helpful that I saw parse [in the RCI-display] and went through to see that it was in the Format class. ''
``It seems to me the key benefit of this [CodeBroker] is that it gives you methods for every class, not like this one [the API Documentation Browser] that you have to first find which class it is in then go to the class. Although it has index of methods, but it is hard to find here [the API Documentation Browser].''
Subjects also acknowledged that the reduced cost of locating components motivated them more to explore the possibility of reuse. As one subject said:
``Having this system, I would try to explore more, I would spend more time to see whether this thing exists or not.''
The more knowledge subjects had about the repository, the better were their doc comments used to retrieve relevant components. One subject described why he wrote one particular comment as:
``I knew there should be a class called NumberFormat or DecimalFormat having the method format...That's why I wrote the word `format' because I knew it would catch those.''As a result, he found what he expected from the deliveries of CodeBroker.
Probably because most subjects had a fairly good working knowledge of programming in general and Java in particular, they were able to describe, in comments, the functionality of programs in a similar way that was used to describe components in the repository.
Different subjects had different ways of writing comments. Some wrote very long and elaborate comments to describe everything they wanted to do in the method or the class. Others wrote very concise and short comments focused on the major task of the program. Because descriptions of components in the repository are short and concise, the short and focused comments made the system deliver more task-relevant components.
Comments are essential for CodeBroker to deliver task-relevant components. Therefore, in the interviews, subjects were asked if they wrote comments in their daily programming activities. Two subjects answered they always wrote comments before the implementation for most classes and methods, one subject said he always wrote comments for classes but not always before the implementation, one subject said he mostly wrote comments for classes but not always before the implementation, and one subject said he usually did not write comments. However, two subjects indicated that they probably would write more comments before implementation if they were going to use CodeBroker because they could benefit from the comments. Two subjects changed their styles of comments within methods from C++-style (which begins with ``//'' and continues until the end of the line) to the doc comment style in order to take advantage of the system delivery. That was an unexpected use of the system,8.4 and showed that subjects expected and valued the help provided by the delivery mechanism of the CodeBroker system.
The signature matching mechanism of CodeBroker (Section 6.1.2) did not play too much of a role in the experiments. In fact, only one subject tried once to look at the change of delivery when he finished the signature declaration of a method, but the system failed to improve the task-relevance of the delivery because there was not any component in the repository that was both similar in concept and compatible in constraint with the task of the subject. In all other experiments, subjects shifted their attention to the RCI-display immediately after they had written the comments and started browsing. When they found components they needed, they moved back to programming and did not pay any attention to the RCI-display until they wrote the next doc comment. The original design goal of adopting the signature matching mechanism in CodeBroker was to help programmers find components that can be reused to replace the module under development. However, in the experiments, all subjects used the system to look for components that could be reused as parts of the module implementation instead of components to replace their intended implementation. The system was more effective in delivering implementation parts than delivering replacement components.
The fact that subjects did not pay attention to the change of delivery caused by signature definitions provided a clue to speculate on the boundary of the action-present (see Section 5.1.2) period of programming. When programmers are writing comments, they are still at the stage of planning, thinking of how to implement the program. At that stage, they are still willing to explore alternative solutions. When they start to define the signature, they have already committed to one chosen solution and have shifted into the stage of execution, at which they are less inclined to explore alternatives.
Although user models did not remove too many components, all subjects said, in the interviews, that they did not notice many well-known components were delivered. A careful examination of those components removed by user models in the experiments found that none of them could have been reused in those tasks. The perception of subjects and available, although quite limited, data still pointed to the need and effectiveness of personalizing component delivery based on user models.
As shown in Table 8.5, neither the system nor the users added too many components to the user models. More on this will be discussed in Section 8.5.4.
The query refinement interface was invoked by three subjects (S1, S3, and S5), and interestingly, they used three different features. S1 used the interface to modify a query and the modification led to locating a needed component. S3 did not change the query; instead he filled the field of Interested Components with a package name because he was sure that all his needed components were from that package. He did, in fact, find all of his needed components from that package. Later in the interview, he commented:
``It worked great since I knew everything was going to be in java.text. That is a nice feature, that little refinement thing.''In addition to query modifications, S5 also used the interface to specify packages in which he was not interested by filling the field of Filtered Components. However, that still did not help him find what he wanted because the terms he chose appear in many other packages he did not specify in the Filtered Components field.
Instead of using the query refinement interface, some subjects activated the delivery mechanism by directly modifying comments in the editor when they realized their previous comments did not help them find what they wanted.
All of these observations confirmed that locating reusable components is an iterative process and the support of retrieval-by-reformulation is necessary (Section 5.4). However, the CodeBroker system did not provide a mechanism that could guide programmers in refining their queries (for discussions, see Section 8.5.3).
Subjects were observed to use the following procedure to choose a component. They first looked at names and descriptions in RCI-display. If they found one promising component, they clicked on it and went to the Browser for detailed information. If they did not find anything interesting in RCI-display, they either moved back to the editor to modify their doc comments or invoked the query refinement interface. If they found several similar components in RCI-display, they moved the mouse over the names and looked at the signatures, trying to find the best one. This process, called information discernment in Section 3.4.3, was often very short, and rarely took more than a minute; in contrast, the detailed evaluation of chosen components often took several minutes.
