WP4: Media content access, production processes and tools Pirkko Oittinen
WP4 Case: Cross Media Solutions Archiving and Access Planning and Design Cross Media Content Management Multimediality Topicality Visuality Publishing and Use 2
Challenges of content management Large volumes of content of different types Legislative requirements for cross modality presentation are increasing Complex processes with hundreds of media objects of different types in the process chain at any given time The processes are formed as an interplay between technical affordances, and journalistic, design and production needs Archiving and access take place at every stage 3
Crossmedia metadata and search needs Pekka Kauranen, Yle/TV2 Eero Sormunen, Tay/INFIM Next Media Result Seminar, 25 January 2011 Helsinki
Objectives and methods Analysis and comparison of (1) crossmedia metadata workflows and (2) search and selection criteria of users of media materials Case organisations STT Lehtikuva (1) YLE (1&2) Methods Workflows: thematic interviews and observation (Sanna Olkkonen/Aalto) User criteria: time line interviews (Marjo Markkula/TaU) 5
Main results Metadata production Metadata serves several functions in the processes Models of media and metadata flows Not fully integrated in planning and editorial processes Bottlenecks and development needs are pointed out User criteria A framework for analysis of search and selection criteria was developed The search criteria are mainly related to media content and can usually only be described as textual metadata The selection criteria are related to media content and context Contextual criteria > users read textual metadata Evaluation of details, impressions and quality > Requires viewing/listening of media Report WP4 CROSSMEDIA SOLUTIONS D4.1.1.3 http://tivit.dicole.net/presentations/attachment_original/657/17661/nm D4.1.1.3_20110112.pdf 6
Observations 1 Automated and manual methods in use to produce metadata of television programs: During production using the information systems for planning, delivery control, editing and archival (Tilsu, Plasma, inews, Avid Media Composer, Metro) Stored in video files captured for a given program Most of the metadata produced during program production are stored automatically in the CMS Metro After delivery broadcast assistants add content related metadata in Metro 7
Observations 2 Some of the useful metadata produced during the production process is not automatically copied in the Metro system. Due to restrictions and priority reasons integration of Metro and the other information systems not yet completed (e.g. inews Metro). A part of the metadata is currently discarded in the production process (Video Mixdown). Some useful metadata not available for search in Metro Examples include manuscripts Direct access to archived content gives rise to changes in content description rules So far, however, the changes have been minor and concerned mainly technical descriptions (e.g. Zooming) 8
Observations 3 Use of video content analysis and speech recognition The volumes of television programs and related materials to be archived increases The resources available for manual content description in the editorial and archival departments decrease Well functioning automated video content analysis and speech recognition would help in Search of sparsely described video and video content free of captions and graphics Search of audio content 9
Päätuloksia Metadatan tuotanto Metadata palvelee useita toiminnallisia funktioita Metadatan ja median kulun mallinnus ei täydellisesti integroitunut suunnittelu ja toimitusprosesseihin osoitetaan mahdollisia integrointi ja kehittämiskohteita Käyttäjien kriteerit Kehitettiin analyysikehikko haku ja valintakriteereille Hakukriteerit liittyvät valtaosin median sisältöön, yleensä kuvattavissa vain tekstimuotoisena metadatana Valintakriteerit liittyvät sekä sisältöön että kontekstiin Kontekstikriteerit > luetaan tekstimuotoista metadataa Yksityiskohtien, vaikutelmien ja laadun arviointi > Edellyttää median katselua/kuuntelua Raportti WP4 CROSSMEDIA SOLUTIONS D4.1.1.3 http://tivit.dicole.net/presentations/attachment_original/657/17661/nm D4.1.1.3_20110112.pdf 10
Tulosten kommentointia 1 Televisio ohjelmaa koskevat metatiedot tuotetaan automaattisesti tai manuaalisesti: vaiheittain ohjelmantekoprosessin suunnittelu, lähetyksenhallinta, editointi ja arkistointivaiheiden tietojärjestelmissä (Tilsu, Plasma, inews, Avid Media Composer, Metro) ohjelmaa varten kuvattuihin videotiedostoihin Suurin osa eri järjestelmissä ohjelmantekoprosessin aikana tuotetuista metatiedoista arkistoituu automaattisesti medianhallintajärjestelmä Metroon. Ohjelman lähetyksen jälkeen kuvaussihteeri lisää Metroon arkistoituun ohjelmaan kuvasisältöjä koskevat metatiedot. 11
Tulosten kommentointia 2 Osa ohjelmantekoprosessin aikana tuotetuista hyödyllisistä metatiedoista ei kopioidu automaattisesti Metroon. Resurssi ja/tai priorisointisyistä tiettyjen tietojärjestelmien ja Metron välille ei ole (vielä) rakennettu integraatiota (esim. inews Metro). Osa metatiedoista häviää tuotantoprosessissa (Video Mixdown). Osa ohjelmantekoprosessin aikana tuotetuista hyödyllisistä metatiedoista ei ole haettavissa Metrosta. Esim. käsikirjoitusten sisällöt Suora pääsy arkistoituihin videoaineistoihin muuttaa sisällönkuvailusääntöjä Tähän mennessä ohjelmien sisällönkuvailusäännöt muuttuneet melko vähän. Luovuttu lähinnä kuvan teknisten lisämääreiden kirjaamisesta (esim. zoomaus, aukiveto, panorointi jne.). 12
Tulosten kommentointia 3 Videon sisällön analyysi ja puheentunnistusmenetelmien käyttöönotto Metroon arkistoitavien televisio ohjelmien ja työmateriaalien määrä lisääntyy Resurssit videoaineistojen manuaaliseen sisällönkuvailuun toimituksissa ja tv arkistossa vähenevät Toimivien automaattisten videon sisällön analyysi ja puheentunnistusmenetelmien käyttö helpottaisi mm.: vähän sisällönkuvailtujen videoaineistojen hakua plansseista tai grafiikasta puhtaiden videoaineistojen hakua audiosisältöjen hakua 13
Crossmedia Archive Multimodal analysis of broadcasting content Jouni Frilander, Yle Jorma Laaksonen, Aalto SCI / AIRC
Presentation contents Business goals Research topics Results so far Goals for year 2011 15
Business goals Future business opportunities. Automated content analysis and annotation tools for audio, video and still images. Finnish language specific solutions. Partners increase their understanding of possibilities and limitations of technology, and create ideas for functionality of next generation media production, archiving and recommendation products Research results will enable creation of new products and enterprises for media industry 16
Yle s collections Yle has notable historic archives. 1.000.000 still images 200.000 hours of audio 200.000 hours of video. Archives are being digitised, thus, their content can be utilised in digital form in the future. 17
Yle s collections images 18
Yle s collections audio 190 vuotta sanomalehdistöä 15.1.1961 Kylmyyden maailmanennätys 11.4.1970 19
Yle s collections video 20
Research topics What kind of automated content analysis can be applied to large amounts of media content? How present methods for automated content analysis could be enhanced? How to automatically create metadata that serves user needs at information retrieval stage? Is it possible to combine automated and human made content description. What methods are used to search, browse and choose different types of media content in the future. How the developed content analysis methods can be applied in media recommendation systems? 21
Results so far video summarisation Sample video summarises ten minutes of content in 30 seconds. Useful tool when looking for right kind of archive material from video archive. The applied summarisation method has been evaluated quantitatively with good results in NIST's yearly TRECVID video retrieval evaluations in 2007 and 2008. 22
Results so far concept detection Analysed video shows identified shots along with concepts detected from the shots. Could provide means for finding objects and humans within video and still image content. Needs enhancing in order to become useful in real life. Sample pictures of successful and unsuccessful detection The concept detection system has been developed and evaluated with good results in TRECVID evaluations since 2005. 23
Results so far automatic speech recognition 1/2 Sample video shows recognised speech as subtitles in the lower part of screen. Quite good automatic creation of keywords for audio and video content. Good automatic creation of content description of audio and video content if speech is clear. Also makes it possible to automatically subtitle television programmes, but recognition quality needs to be enhanced. Sample of good and modest accuracy. 24
Results so far automatic speech recognition 2/2 Recognition accuracy in Word Error Rate (WER) News anchor / voice over: excellent accuracy 12% WER. News interviews: good to fair accuracy 33% 53% WER Informal chat: modest accuracy 57% WER Sufficient for search purposes Almost all keywords recognized correctly 25
Goals for year 2011 The methods and tools will be developed further and evaluated with more material. The video content analysis method will be applied in media recommendation. Plans exist for using the speech recognition system for automatic subtitling in a business pilot. 26
Cross Media ideas, needs and solutions Harri Taskinen, Anygraaf Pauli Tölli, STT Lehtikuva Riku Makkonen, Sanoma Magazines
The goal Electronic archive How are the search for and use of resources going to change? What kind of requirements will this place on applications, metadata descrption and user interface development? Integrated media management system 28 28
Tasks Archiving solutions for multimedia environments Development of the multimedia production of news Content analysis and search methods in multimedia archives Planning tools for editorial processes Multichannel publishing in magazines 29 29
Starting point ideas, needs Texts Texts Images Images Events Articles to print Articles to web Images to print Images to web Event information Cross Media Texts Images Events etc. 30 30
OTHER ELEMENTS 31
Helsingin keskustan hyvä Jokereiden ja HIFK:n pysäköintikuri yllätti Ajokeli huono Lumilapioita koko maassa myyty talviklassikko ei hätkähdä lumipyrystä ennätystahtia. johtuen. Suomessa lumisateesta myyty tänä vuonna satojatuhansia lumityövälineitä 32
Theme1 Theme 2 Press releases, emails etc. Topic 1 Topic 2 Stories in archive Image 1 Attachments(i.e. PDF) Event 1 Event 2 Topic 3 Planning data Image 2 Story Version 1.1 Version 1.2 Version 3 Version 4 Video/audio clips Graphics Story Version 1.1 Version 1.2 Version 1 Story (in Swedish) Version 2 Version 1 Story (in English) Story Sport results Contact information Companies WWW links Places NEXT NEXT MEDIA MEDIA A TIVIT A PROGRAMME 2/6/2011 33
Requirements for data model Adaptation to diverse, still unknown future needs Facilitation of inheritance of once input data Support for shorter time scales towards the end of the process Metadata to facilitate design of different kinds of services without unnecessary time consumption Support for clustering and packining of content for different services 34
Observations 1 The activities should be clustered around topics and content collected around these which allows assembling of different types of service packages Often a topic is based on an event which can be characterised by rich metadata early on in the process. This supports inheritance of the metadata Alternatively a topic/story, associated with metadata may be used as a starting point; inference mechanisms are used for inheritance of metadata In news production topics have a life span of a day whereas themes have no time limit 35
Observations 2 Topic based structure helps collection of background material to a single location (place names, contact info etc. are stored in separate data bases) The theme concept allows packaging of larger wholes and search at the theme level ( find all images used in theme x ) The metadata model designed can be mapped to IPTC and NewsML G2 standards 36