DaLAJ V.1.0

I. IDENTIFYING INFORMATION

Title

DaLAJ v1.0

Subtitle

A proof-of-concept version of a DAtaset for Linguistic Acceptability Judgments, based on the correction-annotated version of Swedish learner essays (SweLL corpus)

Created by

Elena Volodina & Yousuf Ali Mohammed (elena.volodina@svenska.gu.se)

Publisher(s)

Språkbanken Text (sb-info@svenska.gu.se)

Link(s) / permanent identifier(s)

https://spraakbanken.gu.se/en/resources/dalaj

License(s)

CC BY 4.0

Abstract

DaLAJ 1.0 is a Dataset for Linguistic Acceptability Judgments for Swedish, comprising 9 596 sentences in its first version; The baseline for the dataset is 58% accuracy on a binary classification task (i.e. sentence either is correct or not) using BERT embeddings.

DaLAJ is based on the SweLL second language learner data [^3], consisting of essays at different levels of proficiency. To make sure the dataset can be freely available despite the GDPR regulations, we have sentence-scrambled learner essays and removed part of the metadata about learners, keeping for each sentence only information about the mother tongue and the level of the course where the essay has been written. We use the normalized version of learner language as the basis for the DaLAJ sentences, and keep only one error per sentence. We repeat the same sentence for each individual correction tag used in the sentence. For DaLAJ 1.0 we have used four error categories (out of 35 available in SweLL), all connected to lexical or word-building choices, namely errors with compounding, word choices, use of foreign words, use of wrong derivation pattern. Our baseline results for the binary classification show an accuracy of 58% for DaLAJ 1.0. The dataset is included in the SuperLim benchmark. Description of the DaLAJ format, first experiments, our insights and the motivation for the chosen approach to data sharing are provided in [^1]

Funded by

This work has been supported by Nationella Språkbanken – jointly funded by its 10 partner institutions and the Swedish Research Council (dnr 2017-00626), as well as partly supported by a grant from the Swedish Riksbankens Jubileumsfond (SweLL - research infrastructure for Swedish as a second language, dnr IN16-0464:1).

Cite as

Consider citing [^1] [^2]

Related datasets

Coming: full DaLAJ (v.2.0 or later); part of the SuperLim collection (https://spraakbanken.gu.se/en/resources/superlim)

II. USAGE

Key applications

Machine Learning, Neural networks; Linguistic Acceptability Judgments, Error detection; Error classification; First language identification; Classification by proficiency level; etc.

Intended task(s)/usage(s)

(1) Given a sentence, decide whether the sentence is correct or not; (2) Given a sentence (containing an error), find a string that is incorrect; (3) Given a sentence (containing an error), classify the error type; (4) Given a (group of?) sentence(s) (both correct and incorrect), identify the first language of the writer; (5) Given a (group of?) sentence(s) (both correct and incorrect), classify the level of proficiency of the writer (beginner, intermediate, advanced);

Recommended evaluation measures

(1) Accuracy (2) F0.5-score (3) F0.5-score

Dataset function(s)

Training, evaluation, testing

Recommended split(s)

Recommended split of the sentences is provided in the data. Note that there are duplicates in the correct sentences.

III. DATA

Primary data

Text

Language

Swedish

Dataset in numbers

4730 incorrect sentences; 4730 correct sentences (including duplicates); more statistics in [^1] [^2]

Nature of the content

see [^1] [^2]

Format

csv format; 10 columns:

• running number
• original sentence (that contains an error)
• correction sentence
• error indices, i.e. start character index - end character index
• correction indices, i.e. start character index - end character index
• error-correction pair (two strings)
• correction tag
• mother tongue of the writer
• approximate level of the writer - information derived from the course the learner has been taking when writing the essay
• split used for the experiment: train, valid, test

Data source(s)

SweLL learner essays, see [^3]

Data collection method(s)

SweLL learner written essays are collected in the SweLL project, from the test situations (2017-2020) provided learners signed consents, and filled in demographic metadata. Teacheers filled in Task metadata sheets, and where appropriate, grades for the essays. All metadata is registered in the SweLL portal.

Data selection and filtering

Essays were manually pseudonymized, normalized and correct-annotated, see [3]. The selection of essays for manual annotation was subject to balancing the essays, where possible, after mother tongues (10 most frequent in Sweden), gender balance, course level balance, prior education level balance (which in the end was impossible to achieve)

Data preprocessing

Manual transcription, pseudonymizarion, normalization, correction annotation, see [3]

Data labeling

see [^3]

Annotator characteristics

Krippendorff’s alpha on correction annotation task is 0.85 on the basis of 10% of essays that have been double-annotated.

IV. ETHICS AND CAVEATS

Ethical considerations

SweLL dataset is subject to GDPR and Ethical Review Authority restrictions. The DaLAJ format allows to use the dataset openly, since (1) the demographic metadata is practically removed (apart from the information on mother tongue and level of the course), (2) the essays sentences are randomly scrambled and (3) all personal information in the sentences is pseudonymized (e.g. city names replaced with other city names)

Things to watch out for

V. ABOUT DOCUMENTATION

Data last updated

2021-05-26, v1.0

Which changes have been made, compared to the previous version*

This is the first official version

Access to previous versions

This document created

2021-05-27, Elena Volodina

This document last updated

2021-05-27, Elena Volodina

Where to look for further details

[^1], [^2]

Documentation template version

v1.0

VI. OTHER

Related projects

SweLL - research infrastructure for Swedish as a Second Language. https://spraakbanken.gu.se/en/projects/swell

References

[^1] Volodina, Elena, Yousuf Ali Mohammed, and Julia Klezl. (2021). DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing. arXiv preprint arXiv:2105.06681.
https://arxiv.org/pdf/2105.06681.pdf

[^2] Volodina, Elena, Yousuf Ali Mohammed, and Julia Klezl. (2021). DaLAJ - a dataset for linguistic acceptability judgments for Swedish. In Proceedings of the 10th Workshop on Natural Language Processing for Computer Assisted Language Learning (NLP4CALL 2021). Linköping Electronic Conference Proceedings 177:3, s. 28-37. https://ep.liu.se/ecp/177/003/ecp2021177003.pdf

[^3] Elena Volodina, Lena Granstedt, Arild Matsson, Beáta Megyesi, Ildikó Pilán, Julia Prentice, Dan Rosén, Lisa Rudebeck, Carl-Johan Schenström, Gunlög Sundberg and Mats Wirén (2019). The SweLL Language Learner Corpus: From Design to Annotation. Northern European Journal of Language Technology, Special Issue. https://nejlt.ep.liu.se/article/view/1374/1010