November 1997 in Boston
ITS Authoring Systems
Generating Case Oriented Intelligent Tutoring Systems
University of Würzburg
Department of Computer Science (Informatik VI)
Am Hubland 16
97074 Würzburg, Germany
There are many ways to use expert systems for teaching problem solving. The use of existing knowledge bases can help to reduce the cost of building new systems. We introduce an approach to use knowledge bases for classification problems as a domain model for intelligent tutoring systems. Most of the existing tutoring systems, that are really used in practice, are based on hypertext techniques with predefined links, that makes it hard to change or expand the presented knowledge or even to change the whole domain. Expert system techniques allow to formulate different knowledge bases for classification problems very comfortable. Here we focus on the didactic elements of case based tutoring systems generated from existing knowledge bases. The biggest advantage of these systems is the possibility to add new cases in a very short time without changing the systems itself or the knowledge base.
Computer based training provides special possibilities for the presentation of learning objectives by using the new media opportunities and so support their graphic presentation. With the varied techniques can for example the success oriented learner be motivated with immediate feedback while the more cautious learner can reduce his fear through the anonymity of the medium and so develop some skills doing experiments with the system.
Many programs try to develop flexible reference books with hypermedia techniques, while other programs are used exclusive to assess the student by practice possible questions and answers. Case oriented training systems are developed to help the student use their already existing knowledge on new typical or unusual problem situations and so get more self confidence in problem solving.
Intelligent tutoring systems are based on the idea that having a declarative knowledge representation helps understanding the actions of the student better. So, the student can order examinations or suggest diagnoses that are totally senseless from the view of a learning system author. So these actions would not be possible in systems with static links while the student can learn a lot from selecting these actions.
A base for these dynamic linked systems have to provide at least basic knowledge about the underlying problem solving method itself, what shows clearly the relation to expert systems. An ideal intelligent tutoring system has four components [Wenger87, Puppe92]:
By a closer look at these components, we see that at least the domain model can be supported when not even replaced by a knowledge base of an expert system. Additional the inference machine of the expert system can be used to assess the students actions and so find its place in the didactic component. By this view expert systems "only" have to be expanded with a tutorial environment to provide the skeleton of an intelligent tutoring system.
The first of these systems was GUIDON, a system based on the expert system MYCIN [Clancey93]. But also the simulation based system of [Woolf95] must be count to this area. In [Jacobsen&Smith91], [Lippert90] and [Thornberg et.al.90] some approaches are described how to use existing expert systems in learning environments. The last mentioned systems were used as an addition to the conventional classes in school, but never were used without the advising "human" teacher.
Fig.1:Architecture of the Trainer integrated in the D3 architecture. The Trainer is a component of D3 that works only with the student but uses the static knowledge from the knowledge and the case base und the dynamic knowledge of the problem solver and the explanation tool. The knowledge and the case based is authored by the domain expert totally graphic with the help of an acquisition system and multiple dialogs in D3.
In the following, we want to describe the approach that a tutoring system present problem situations without human advise or explanation, so that the student can practice problem solving, supported and assessed by the system. The expert system toolbox D3 [Puppe96] has such a tutorial component, called the Trainer, that can criticize the students problem solving actions based on the rules of the knowledge base. The basic architecture with the combination to the expert system toolbox is shown in figure 1.
Fig. 2:Case presentation of the guided mode.
The Trainer ([Puppe&Reinhardt95], [Reinhardt96]) works case oriented with the student, what means the system presents a case (for example in medical domains a patient), the student has to classify. In the easiest level, the guided mode (see figure 2), the student gets the case data in special groups (like history, examinations, laboratory tests, technical tests in medical domains). The student now has to state some suggested diagnoses after the presentation of a new group by selecting one or more diagnoses from a dynamic hierarchy of all possible diagnoses.
Fig 3:Assessement of the selected diagnoses.
The system assesses this selection based on the diagnoses the expert systems has derived (see figure 3). At this level the student learns how to conclude a diagnosis from given data.
In the other level, the free mode, only the basic case data is presented to the student and she has to select the next tests to be done in the case (see figure 4). The selection of the new data can also be assessed by the system in comparison with the tests the expert system would do next. At the end of a case the guided test or at any time at the free test the student can justify her chosen diagnosis by selecting observations (case data) that point to the diagnosis. This action can also be assessed on base of the rules of the expert system.
Fig. 4:Selection of new data in the free test. The student chooses some new examinations and the results are integrated in the case presentation.
The third task for classification problem solving after conclusion of diagnoses and selection of new data is the recognition of raw (visual) case data (for an example see figure 5). A hypertext document showing the raw data of the case supports this student action. There must be a link from the document to the knowledge base and the case to make an assessment of the recognition possible. The student enters the recognized data in the normal case presentation and the derived diagnoses are based on these unsure data as well.
Fig. 5:Part of a hypertext document to visualize some observations that should be recognized by the student. The formalized data should be entered in the case presentation.
In a case oriented training system it is very important how realistic the case, for example the patient in medical domains, can be presented to the student and how realistic the interactions are. The constraints can be divided in three major groups:
constraints of the medium (how many human senses are reached)
level of detail (how detailed is the case presented)
interaction (how can the student interact with the system and how does it react to the student)
A specific tutorial factor is the way, motivation and the scope of the feedback the systems gives to the student actions. Finally the total amount of time for a learning session must be considered.
Since it is not possible to simulate a case in full detail in the moment (for every domain) the presentation of the case focus on diagnostic relevant aspects and is so a serious simplification in comparison to the real situation. Nevertheless there are some differences in the level of detail that can be considered. In the training system of D3 the base if the case presentation is a hierarchical and standardized case description that is generated directly from the knowledge base. This is a simplification in two major ways: In the real situation the observations are rarely in a standard form and usually not presented all at once. One way to get more of the realistic situation can be reached by the hypertext document, where the observation are presented in non-standard, mostly visual form. The student has to recognize them and transform them into the standard form in the hierarchy. An approximation to the sequential nature of real data is the stepwise presentation of the case data or the force to ask for new data by the student.
As in the case presentation the interaction possibilities are constrained to the diagnostic relevant aspects. So important capabilities like emotionally open to a patient, the art of communication or the manual ability of take off blood cannot be caught in a computer based system. The relevant diagnostic actions in this case are the indication of new data and the statement of diagnostic hypotheses. In both cases the selection should not be restricted by multiple choice alternatives, while the problem of natural language processing should not be ignored also. So the selection of diagnoses or new data also happens in a hierarchy of all known alternatives. The consequences of the student actions are also presented to her in a standard form. When she ordered new tests the data is integrated in the existing case presentation.
There are three forms of feedback to student actions provided:
Since the feedback and the assessment of those major tasks in different situations are very variable it is not possible to "can" these feedback as static text. The feedback has to be generated from the assessment in comparison to the knowledge base. The important and difficult part is here to recognize convenient alternatives to the system solution and assess these situation very positive. Hence this is dependent on the knowledge base a generated system can never guarantee for totally correct feedback.
A difficult problem is also how to enable the student to justify her decisions. A justifications as a exact replication of the system relations is in general not very useful, because the same correlation can be formulated in many different ways. That is why an abstraction has to be carried out. The biggest abstraction for the justification of a hypothesis is the restriction to the selection of some observations from the case presentation, that point to the hypothesis. This corresponds directly to the causal set-covering diagnostic while in the heuristic diagnostic these relations have to be extracted from the mostly (at least in our knowledge bases) more complex rule structures. The big difficulty here is that mainly symptom combinations are used in the rules, sometimes united and renamed in syndromes or symptom interpretations. The problem is that these names cannot be presupposed.
Looking at the three major points case presentation, interaction and feedback it is obvious that there will never be an optimal solution regarding all points in a generated system. It is more realistic to offer different compromises that can be found in the settings. These compromises are influenced by the basic criteria time, level of reality, demand of knowledge and specificity of the case. Regarding this a case presentation that is close to reality needs a lot more time and may be unnecessary for the demand of knowledge of the student.
The Trainer offers many possibilities to vary the case presentation that can be set by the student or the expert who developed the knowledge base to find an optimal compromise for the special learning situation. So the case data can be presented in groups (like history, physical examination, laboratory and technical examinations). This separation helps the student to save time and teaches the reasonable sequence of examinations in general, but lacks on reality. More realistic but also more time intense and more difficult is the setting, that the student only gets the basic data and has to order every additionally test by herself. The Trainer integrates the new data in the existing case presentation and the results are shown in the hierarchy. This option is more realistic than the first solution but still cannot simulate the real situation totally (like the interview with a real patient in medical domains). Another possibility is to show the reachable facts in the hierarchy but not show the answers to them unless the student really clicks on them to "ask" for the answer. This last option is the most realistic one the Trainer offers but also needs the most time to get through the case.
Another simplification for the student is the possibility to filter normal and abnormal data. The case presentation than only shows the abnormal facts so she can focus on the relevant ones. Of course there is also a trade off between level of reality, amount of time and demand of knowledge.
The integration of the already mentioned diagnostic middle level like symptom interpretation in the case presentation as an alternative to the pure symptom presentation of the moment could be another convenient option. But in the moment this possibility is not included in the Trainer version.
Last there is the documentation with hypermedia to mentioned, where the case can be presented very realistic with additional text elements, pictures, animation and sound.
The interaction in the Trainer is always based on the selection from a hierarchy of diagnoses or examinations and tests. These hierarchies are dynamic extended, when the student clicks on one level what guarantees the clarity of the given facts. This is a very important point because in normal knowledge bases there are up to 3oo diagnoses in average, so to show all diagnoses from the beginning would require too much from the student. So the facts are always ordered in groups and the student can navigate through the hierarchy with the helps of the dynamic nodes (figure 6). Since it is not guaranteed that all examinations were done in the case what sometimes leads to the message for the student that this test could not be integrated in the case presentation. Here one could think of generating normal test results to provide the student with a complete case description.
Fig. 6:Dynamic hierarchy of diagnoses. If the student knows that it is an inflammantory desease, she has not to open the other groups of diagnoses. Now she can weight the suspected diagnoses with the two categories "suggested" or "confirmed".
The possibility to divide diagnoses in different assessment classes allows the expert to focus on special diagnoses of the case even when other not so interesting diagnoses are also in the case (figure 7).
Fig. 7:Unimportant diagnoses are shown to the student if she ask for it, but a case can be totally examined by the student without recognizing multiple diagnoses if they are unimportant.
On the other hand this focus on relevant diagnoses has its drawbacks. It could be that during a case the student never sees all diagnoses and so loose the sensibility for multiple diagnoses. Recommendable would be a devaluation of the unimportant diagnoses and a revaluation of the relevant diagnoses, so that the student gets positive feedback if she found the relevant ones but also the system points to the not so important ones.
For the justification of hypotheses in the moment only symptoms or confirmed diagnoses can be used, while it was desirable to integrate symptom abstractions and syndromes. For the justification of new examinations suspected diagnoses are used that could be confirmed with these tests. Here a more detailed technique would increase the justification ability enormous.
The explanation to the assessment are right now generated very detailed and good out of the knowledge base but the possibility to provide additional informal knowledge can lead to a deeper understanding of the field.
A choice between the immediate feedback that is implemented in the moment and a more delayed feedback could help the unsure learner to get more motivated for experimental based learning.
To add costs for test and examinations and resulting from that a final "bill" for the case would lead to a responsible usage in the real life situation. These calculations have to take into regard the financial cost as well as the cost like radiation exposure for a patient.
There are different approaches for evaluating a tutorial system that base on existing methods in evaluation of general systems. A survey can be found in (Legree, Gillis and Orey 93) and (Mark and Greer 93).
Coming from the four basic components in a tutoring system that should harmonize with each other, different start points must be considered. In all this discussion one should never forget that the student never learns only from one source but the combination of the whole learning environment and the different media leads to the success.
Through the graphical knowledge acquisition (Gappa 95) the Trainer has a comfortable authoring system for the diagnostic rules. The new integrated hypermedia document can be linked together and combined with the elements (symptoms, diagnoses and cases) of the knowledge base completely graphical so that no knowledge engineer is needed for that also.
In the training system there are complex heuristic rules used that the system transfers into simple symptom diagnoses relations for assessment and explanations. An example of such a rule is shown in figure 8.
For the student the following presentation would be extracted (in dependence to the actual case) like:
Symptoms that point to Sjoegren-Syndrome:
Fig.8:Complex rule in the rheumatology knowledge base. The deepness of such rules is not restricted and the symptoms can be combined with and- Nodes, Or-Nodes or n-of-m-nodes. The n-of-m-Nodes means that n of the m given symptoms have to be true.
This simplification is chosen because the real rules are often too complex to teach to the students directly. The transformed and reduced rule is easier to understand for the student in the problem solving situation.
The discussed transformation of the rules makes it also hard to model the knowledge level of the student. A student model in general should answer the following questions :
While in the Trainer a history is integrated and also a kind of personality can be represented in the settings, the modeling of the students knowledge is not considered in any way.
In the future an overlay model of the knowledge base in combination with a text book like presentation will be implemented, that will be used for the selection of new cases and a more individual feedback that point to former solved cases.
The Trainer offers different didactic concepts and decisions that can be set by the system user or by the expert who developed the knowledge base. The most important didactic principles were discussed in the last section and are related to the compromise between level of reality, amount of time and demand of knowledge.
Altogether there are multiple didactic methods that can be combined by the expert or the student. The integration of these methods was done in close cooperation with several domain experts. This cooperation helps to integrate the system to the real learning environment.
The best way for evaluation of user interfaces are pilot testing and experimental application. The design of the Trainer user interface was developed in close cooperation with the domain experts (Schewe, Quak, Reinhardt, Puppe 96). During the whole development phase the system was tested by user from different groups (students, doctors in different stages of education). This leads to a practice oriented interface that is adapted to the real needs without loosing the shell character.
The first success of the generated training systems comes with the improved user interface in comparison to the first system of D3 TUDIS (Poeck and Tins 93). First of all the dynamic hierarchies made it possible for the students to handle the huge amount of symptoms. In TUDIS the symptoms were in a scrollable window where the students totally lost the overall view and so their motivation.
Another important improvement was the organization of the assessment in feedback categories instead of detailed points. The categories were characterized through pictures what makes it easy for the student to classify themselves.
The system generated with the Trainer that are in practice right now showed that they are reasonable additions to conventional teaching environments. So there are systems connected to university classes (rheumatology, flower classification) and systems connected to a text book (neurology). The biggest advantage is the high flexibility regarding to changes of knowledge and case base. So in the rheumatology class the students can examine the "same" patient the saw the first day in real the next day as a Trainer case.
Some problems raise from the generated user interface that cannot be totally adjusted to a special domain. There are training systems not only in medical domains but also in flower classification (Reinhardt 96) or technical domains like error diagnostic (Puppe, Seidel and Daniel 95). Since there is a different terminology in every domain there cannot be a prefect terminology in the Trainer. There will be a new concept where the expert can adjust some key words to get closer to the specific terminology.
With the mentioned approaches to improve the Trainer most of the problems can be solved or at least reduced. The next huge development will be the combination with an electronic text book and the coupling with different problem solving processes.
Clancey, W.J. 1993. Guidon-Manage revisited: A socop-technical system approach. Journal of Artificial Intelligence in Education 4(1):5-34.
Euler, D. 1992. Didaktik des Computergesteuerten Lernens - Praktische Gestaltung und Theoretische Grundlagen. Multimediales Lernen in der Berufsausbildung 3.
Gappa, U. 1995. Graphische Wissensacquisitionssysteme. Ph.D. diss., Dept. of Computer Science, University of Karlsruhe, Germany.
Jacobsen, R., and Smith, G. 1991. Expert Systems: Using artificial intelligence to support students doing algebra homework. Journal of Artificial Intelligence in Education 2(1).
Legree, P., Gillis, P., and Orey, M. 1993. The quantitative evaluation of intelligent tutoring systems applications: product and process criteria. Journal of Artificial Intelligence in Education 4(2/3).
Lippert, R. 1990. Teaching problem solving in mathematics and science with expert systems. Journal of Artificial Intelligence in Education 1(2/3).
Mark, M., and Greer, J. 1993. Evaluation methodologies for intelligent tutoring systems. Journal of Artificial Intelligence in Education 4(2/3).
Poeck, K., and Tins, M. 1993. An intelligent tutoring system for classification problem solving. In Proceedings of the GWAI-92. Springer.
Puppe, F. 1992. Intelligente Tutorsysteme. Informatik Spektrum 14:195-207.
Puppe, F., Gappa, U., Poeck, K., and Bamberger S. eds. 1996. Wissensbasierte Diagnose- und Informationssysteme. Spinger.
Puppe, F., and Reinhardt, B. 1995. Generating Case-Oriented Training from Diagnostic Expert Systems. Machine-Mediated Learning 5(3&4).
Puppe, F., Seidel, G., and Daniel, M. 1995. Qualifizierende Arbeitsgestaltung mit tutoriellen Expertensystemen für technische Diagnoseaufgaben. Technical Report, University of Würzburg.
Reinhardt, B. Expert systems and hypertext for teaching diagnostics. 1996. In Proceedings of the european conference of artificial intelligence in education in Lisbon.
Schewe, S., Quak, T., Reinhardt, B., and Puppe, F. 1996. Evaluation of a knowledge based tutorial program in rheumatology - a part of a mandatory course in internal medicine. In Proceedings of the third international conference on intelligent tutoring systems ITS96 in Montreal. Springer.
Thornberg, M., Baer, R., Ferrara, J., and Althopuse, B. 1990. Using expert systems to teach concepts associated with special eligibility decisions. Journal of Artificial Intelligence in Education 1(3).
Wenger, E. 1987. Artificial Intelligence and Tutoring Systems. Morgan Kaufman.
Woolf, B., and Hall, W. 1995. Multimedia Pedagogues - Interactive Systems for Teaching and Learning. IEEE Computer, May, 74-80.