If you like to learn languages ​​(or you teach them), then you have probably come across such a way of mastering a language as parallel reading. It helps you immerse yourself in context, increases vocabulary and makes learning fun. In my opinion, it is worth reading the texts in the original in parallel with the Russian ones, when the basics of grammar and phonetics have already been mastered, so that no one has canceled the textbooks and teachers. But when it comes to reading, you want to choose something to your liking, or something already familiar or beloved, and this is often impossible, because no one has released such a version of a parallel book. And if you are not learning English, but conditional Japanese or Hungarian, then it is difficult to find at least some interesting material with parallel translation.
Today we will take a decisive step towards rectifying this situation.
. " " .
( , ):
TO KILL A MOCKINGBIRD by Harper Lee DEDICATION for Mr. Lee and Alice in consideration of Love & Affection Lawyers, I suppose, were children once. Charles Lamb PART ONE 1 When he was nearly thirteen, my brother Jem got his arm badly broken at the elbow. When it healed, and Jem’s fears of never being able to play football were assuaged, he was seldom self-conscious about his injury. His left arm was somewhat shorter than his right; when he stood or walked, the back of his hand was at right angles to his body, his thumb parallel to his thigh. He couldn’t have cared less, so long as he could pass and punt.
, , - . 1 , , . , , . ; , . - .
, :
, .
, . :
- . (, ..), . .
- , - .
- , (, , ), , .
lingtrain-aligner, python, . , . .
, . . , 50- , — . , , . , , .
, :
- .
- .
- razdel .
- , .
, .
.
, . , . .
| %%%%%title. | ||
| %%%%%author. | ||
| %%%%%h1. %%%%%h2. %%%%%h3. %%%%%h4. %%%%%h5. | ||
| %%%%%divider. | ||
| %%%%%. |
: [.,:,!?] , .
- ( , , , ).
- .
- (H1 , H5 ). , .
- , , ( ).
, . , :
TO KILL A MOCKINGBIRD%%%%%title. by Harper Lee%%%%%author. %%%%%divider. PART ONE%%%%%h1. 1%%%%%h2. When he was nearly thirteen, my brother Jem got his arm badly broken at the elbow. When it healed, and Jem’s fears of never being able to play football were assuaged, he was seldom self-conscious about his injury. His left arm was somewhat shorter than his right; when he stood or walked, the back of his hand was at right angles to his body, his thumb parallel to his thigh. He couldn’t have cared less, so long as he could pass and punt. ...
%%%%%author. %%%%%title. %%%%%divider. %%%%%h1. 1%%%%%h2. , , . , , . ; , . - . ...
"" (" ", " " ..) h1, h2. .
Colab
Colab . , . . html .
, .
:
pip install lingtrain-aligner
:
from lingtrain_aligner import preprocessor, splitter, aligner, resolver, reader, vis_helper
:
text1_input = "harper_lee_ru.txt" text2_input = "harper_lee_en.txt" with open(text1_input, "r", encoding="utf8") as input1: text1 = input1.readlines() with open(text2_input, "r", encoding="utf8") as input2: text2 = input2.readlines()
SQLite ( ) lang_from lang_to. , :
db_path = "db/book.db" lang_from = "ru" lang_to = "en" models = ["sentence_transformer_multilingual", "sentence_transformer_multilingual_labse"] model_name = models[0]
:
splitter.get_supported_languages()
, , xx, . sentence_transformer_multilingual 50+ , sentence_transformer_multilingual_labse 100+ .
:
text1_prepared = preprocessor.mark_paragraphs(text1) text2_prepared = preprocessor.mark_paragraphs(text2)
:
splitted_from = splitter.split_by_sentences_wrapper(text1_prepared , lang_from, leave_marks=True) splitted_to = splitter.split_by_sentences_wrapper(text2_prepared , lang_to, leave_marks=True)
aligner.fill_db(db_path, splitted_from, splitted_to)
. batch_size, window, . , . . , , .
batch_ids = [0,1,2,3] aligner.align_db(db_path, \ model_name, \ batch_size=100, \ window=30, \ batch_ids=batch_ids, \ save_pic=False, embed_batch_size=50, \ normalize_embeddings=True, \ show_progress_bar=True )
! , . vis_helper. 400, , batch_size=400. , , batch_size=50, 4 -.
vis_helper.visualize_alignment_by_db(db_path, output_path="alignment_vis.png", \ lang_name_from=lang_from, \ lang_name_to=lang_to, \ batch_size=400, \ size=(800,800), \ plt_show=True)
. , . :
- .
- , . , , , .
- .
- . " " , . , , . .
. , .
. , , — , , .
. .
. , , , . , 10,11,12 15,16,17 . . , . . resolver.
:
conflicts_to_solve, rest = resolver.get_all_conflicts(db_path, min_chain_length=2, max_conflicts_len=6)
conflicts to solve: 46 total conflicts: 47
conflicts_to_solve , , rest .
:
resolver.get_statistics(conflicts_to_solve) resolver.get_statistics(rest)
('2:3', 11) ('3:2', 10) ('3:3', 8) ('2:1', 5) ('4:3', 3) ('3:5', 2) ('6:4', 2) ('5:4', 1) ('5:3', 1) ('2:4', 1) ('5:6', 1) ('4:5', 1) ('8:7', 1)
, 2:3 3:2, , , .
:
resolver.show_conflict(db_path, conflicts_to_solve[10])
124 , . 125 , , — . 126 . 122 The Radley Place jutted into a sharp curve beyond our house. 123 Walking south, one faced its porch; the sidewalk turned and ran beside the lot.
, 125 126 , [124]-[122] [125,126]-[123]. ? , . , , , . :
- [124,125]-[122] // [126]-[123]
- [124]-[122] // [125,126]-[123]
, , — 2 ( ) 6. , . , , .
:
steps = 3
batch_id = -1 #
for i in range(steps):
conflicts, rest = resolver.get_all_conflicts(db_path, min_chain_length=2+i, max_conflicts_len=6*(i+1), batch_id=batch_id)
resolver.resolve_all_conflicts(db_path, conflicts, model_name, show_logs=False)
vis_helper.visualize_alignment_by_db(db_path, output_path="img_test1.png", batch_size=400, size=(800,800), plt_show=True)
if len(rest) == 0:
break
:
:
book.db. .
, , . :
resolver.fix_start(db_path, model_name, max_conflicts_len=20)
resolver.fix_end(db_path, model_name, max_conflicts_len=20)
reader.
from lingtrain_aligner import reader
, , :
paragraphs_from, paragraphs_to, meta = reader.get_paragraphs(db_path, direction="from")
direction ["from", "to"] . (, ) .
create_book():
reader.create_book(paragraphs_from, paragraphs_to, meta, output_path = f"lingtrain.html")
:
html . , pdf, .
. , . template.
reader.create_book(paragraphs_from, paragraphs_to, meta, output_path = f"lingtrain.html", template="pastel_fill")
reader.create_book(paragraphs_from, paragraphs_to, meta, output_path = f"lingtrain.html", template="pastel_start")
, , .
template="custom" styles. CSS , .
, :
my_style = [ '{}', '{"background": "#fafad2"}', ] reader.create_book(paragraphs_from, paragraphs_to, meta, output_path = f"lingtrain.html", template="custom", styles=my_style)
span' :
my_style = [ '{"background": "linear-gradient(90deg, #FDEB71 0px, #fff 150px)", "border-radius": "15px"}', '{"background": "linear-gradient(90deg, #ABDCFF 0px, #fff 150px)", "border-radius": "15px"}', '{"background": "linear-gradient(90deg, #FEB692 0px, #fff 150px)", "border-radius": "15px"}', '{"background": "linear-gradient(90deg, #CE9FFC 0px, #fff 150px)", "border-radius": "15px"}', '{"background": "linear-gradient(90deg, #81FBB8 0px, #fff 150px)", "border-radius": "15px"}' ] reader.create_book(paragraphs_from, paragraphs_to, meta, output_path = f"lingtrain.html", template="custom", styles=my_style)
[2] Google Colab.
[3] Sentence Transformers .
[4] Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
[5] Language Agnostic BERT Sentence Encoder .