Tekstin tallennus- ja hakumenetelmien kehittäminen suomen kielen tulkintaohjelmien avulla: FULLTEXT-projektin loppuraportti

Transkriptio

1 Tekstin tallennus- ja hakumenetelmien kehittäminen suomen kielen tulkintaohjelmien avulla: FULLTEXT-projektin loppuraportti Riitta Alkula & Timo Honkela ABSTRACT The project, Linguistic processing and retrieval techniques in Finnish fulltext databases (FULLTEXT), dealt with the special problems of fulltext databases in the Finnish language. Finnish has a rich inflectional and derivational morphology. Another typical characteristic is the use of compounds; in the English language these compunds would be multi-word terms. The characteristics of Finnish result in poor system performance when commercial information retrieval systems developed for English are used. To decrease the size of the inverted file and to improve retrieval efficiency, it is reasonable to normalize the inflectional variants of a word to the basic form. In the FULLTEXT project, natural language analysis modeules for Finnish were incorporated into the BASIS and APL-MINTTU retrieval systems and severeal test databases were produced. When word forms were normalized to their basic form, the memory size of the index file was smaller than the a traditional index, where the words are saved in their inflectional form. Even when the components of the compound words were added to the basic form index it still remained smaller than the traditional index. In the retrieval tests, best recall was achieved in the index that contained the basic word forms and components of compound words. It was found that good recall did not result in poor precision. The precision ratio was about as good as in other indexes. Queries had best precision in a database where the automatically truncated terms were searched in a traditional index and then the retrievd index terms were analyzed and filtered with natural language analysis modules. Unfortunately, in this case, the recall ration was lower than in other test databases. Problems in the use of natural language modules were also investigated. When the search terms are given in their basic form, the searcher must be more conscious with derivatives and compounds than when using truncated search terms in traditional indexes. Methods to transform the search terms to their correct basic form should be further developed. Remarks The scanned original full text report starts at the 4th page of this document.

2 References Bain, Malcolm, Richard Bland, Lou Burnard, Jon Duke, Colin Edwards, David Lindsey, Nicholas Rossiter, and Peter Willett. Free text retrieval systems: a review and evaluation. Taylor Graham Publishing, Blair, David C. Language and representation in information retrieval. Elsevier North-Holland, Inc., Doszkocs, Tamas E., James Reggia, and Xia Lin. "Connectionist models and information retrieval." Annual review of information science and technology 25 (1990): Lehti, Merja, and Pirkko Eskola. Suorakäyttöisten tiedonhakujärjestelmien käyttö Suomessa Valtion teknillinen tutkimuskeskus. Informaatiopalvelulaitos, Harter, Stephen P. Online information retrieval: concepts, principles, and techniques. Academic Press Professional, Inc., Heimbürger, Anneli, Riitta Alkula, and Taru Kuhanen. Hyperteksti ja hypermedia. Valtion teknillinen tutkimuskeskus, informaatiopalvelulaitos, Honkela, Timo, and Ari M. Vepsäläinen. "Interpreting imprecise expressions: Experiments with Kohonen s selforganizing maps and associative memory." In Proceedings of ICANN 2011, vol. 1, pp Jäppinen, Harri, Aarno Lehtola, Esa Nelimarkka, and Matti Ylilammi. "Knowledge engineering approach to morphological analysis." In Proceedings of the first conference on European chapter of the Association for Computational Linguistics, pp Association for Computational Linguistics, Karetnyk, David, Fred Karlsson, and Godfrey Smart. "Knowledge-based indexing of morpho-syntactically analysed language." International Journal of Applied Expert Systems 4, no. 1 (1991): Karlsson, Fred. "Morphological tagging of Finnish." Computational Morphosyntax, Publica 13 (1985): Koskenniemi, Kimmo. "An application of the two-level model to Finnish." Computational morphosyntax: Report on research 1984 (1981): Koskenniemi, Kimmo. "FINSTEMS: a module for information retrieval." Computational Morphosyntax: Report on Research 84 (1981): Kotzias, Klaus. "How to respond to different language particularities by indexing texts using automatic text analysis." In International online information meeting, pp Laalo, Klaus. Säkeistä patoihin: suomen kielen monitulkintaiset sananmuodot. Suomalaisen kirjallisuuden seura, Lin, Xia, Dagobert Soergel, and Gary Marchionini. "A self-organizing semantic map for information retrieval." In Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, pp ACM, Newton, Steve J. Text filing and retrieval systems: a practical evaluation guide. National computing centre, Peters, Thomas A. "When Smart People Fail: An Analysis of the Transaction Log of an Online Public Access Catalog." Journal of academic librarianship 15, no. 5 (1989): Ritter, Helge, and Teuvo Kohonen. "Self-organizing semantic maps." Biological cybernetics 61, no. 4 (1989): Saffady, William. Text storage and retrieval systems: A technology survey and product directory. Meckler, Salton, Gerard. Automatic Text Processing: The Transformation, Analysis, and Retrieval of. Addison-Wesley, Tenopir, Carol, and Jung Soon Ro. Full text databases. Greenwood Press, Thönssen, Barbara. "Automatische Indexierung und Schnittstellen zu Thesauri.[Interfaces Between Automatic Indexing and Thesauri]." Nachrichten fur Dokumentation (West Germany) 39, no. 4 (1988): [The list of references has been reproduced to support search system operations. Errors are possible. Please check the original.]

3 Keywords and search terms: Named entities: VTT, Valtion teknillinen tutkimuskeskus, TEKES, VTKK, KTA-Papyrus, Aamulehti, Länsiväylä-lehti, Tampereen yliopisto, Eeva Palosuo, Juhani Virtanen, Matti Sihto, Kimmo Koskenniemi, Mika Herpiö, Pekka Vuorio, Harri Arnola, Sauli Laitinen, Eero Sormunen, Taru Kuhanen, Sanna Hätönen, Raili Salminen, Markku Kuokkala, Markku Ylinen, Tarja Hjorth, Kaarina Nazarenko, Jaakko Anttila, Kari Martiskainen, Irma Salovaara, Pirjo Valpas, Tarja Heinivaho, Klaus Nurmi, Tuija Tuominen, Kalervo Järvelin, Olli Paavola. Finnish terms: Tiedonhakujärjestelmä, hakujärjestelmä, suomen kieli, taivutusmuoto, johdos, yhdyssana, homografia, sanaliitto, hakusana, taivutusvartalo, perusmuoto, MINTTU, BASIS, testikysely, hakemisto, TWOL, hakutulos, käyttäjä, perusmuotohaku, automaattinen katkaisu English terms: Information retrieval system, database, free-text retrieval, inverted index, index term, stop word, query, Finnish language, inflectional word forms, compound word, automatic truncation, morphological analysis, APL language, C language The full report follows in a scanned form.

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102