Sentencepiece calculation results

Evaluation against standard

VocSizeComb.上/下ScoreComm.
2万Comb.prop30/52194
3万Comb.prop30/52174
2万Position3/12166
1.5万Count3/12156
1.5万Position3/12155
2万Comb.prop3/12126
2万Count3/12095
3万Position3/12077
3万Comb.prop3/12045
1.5万Comb.prop30/52033
1.5万Comb.prop3/12026
3万Count3/11906
1.5万Position30/51814
2万Position30/51804
3万Position30/51714
1.5万Count30/51615
2万Count30/51525
1万Count3/11526
1万Position3/11495
3万Count30/51484
1万Comb.prop3/11325
1万Position30/51273
1万Count30/51143
1万Comb.prop30/51143
3万Comb.prop70/35733
2万Comb.prop70/35543
1.5万Comb.prop70/35463
1.5万Count70/35375
1万Count70/35272
2万Count70/35104
3万Count70/3593
2万Position70/3584
3万Position70/3505
1.5万Position70/3503
1万Comb.prop70/3503
1万Position70/3503

Self(=same 部) / Other (=other 部) results

Results for different values of the upper and lower limits (上下) and vocabulary size (3万, 2万).

For all subcategories 部類 in a category 部, for the 10 most similar subcategories, a same/other calculation is performed, the maximal score is thus 10 times the number of the subcategories for a given category.

KR1 經部 (10部類)

Results for KR1
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/353.00 1.11 1.11 7.78 35.00 37.78 35.56 8.89 10.00 10.00 8.89 8.89
30/531.00 36.67 42.22 42.22 59.00 62.22 66.67 62.22 48.00 53.33 58.89 53.33
3/163.00 65.56 66.67 64.44 61.00 62.22 64.44 61.11 62.00 64.44 65.56 64.44

KR2 史部 (15部類)

Results for KR2
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/3529.33 24.00 18.67 55.00 52.00 52.67 54.00 37.14 16.67 11.33 12.00 40.00
30/561.33 64.00 64.67 72.86 77.33 81.33 86.00 79.29 71.33 73.33 78.67 79.29
3/166.67 68.00 64.00 65.00 60.67 62.67 62.67 61.43 67.33 66.00 64.00 61.43

KR3 子部 (12部類)

Results for KR3
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/3528.33 30.83 33.33 26.00 22.50 19.17 13.33 21.00 33.33 30.00 28.33 23.00
30/524.17 27.50 22.50 18.00 24.17 20.83 18.33 17.00 26.67 25.83 20.83 19.00
3/120.83 27.50 25.00 26.00 26.67 30.00 26.67 23.00 25.00 25.00 23.33 24.00

KR4 集部 (9部類)

Results for KR4
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/3545.56 45.56 45.56 16.00 35.56 38.89 42.22 24.00 47.78 54.44 54.44 22.00
30/547.78 51.11 52.22 26.00 51.11 54.44 52.22 14.00 50.00 52.22 52.22 26.00
3/134.44 45.56 40.00 36.00 35.56 38.89 33.33 28.00 34.44 38.89 38.89 30.00

KR5 道部 (9部類)

Results for KR5
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/352.22 0.00 0.00 21.11 8.89 2.22 2.22 24.44 14.44 11.11 8.89 5.56
30/553.33 58.89 63.33 67.78 68.89 70.00 71.11 65.56 64.44 67.78 70.00 72.22
3/163.33 60.00 60.00 54.44 60.00 55.56 54.44 53.33 61.11 56.67 57.78 55.56

KR6 佛部 (20部類)

Results for KR6
LimitCountProb.Pos
上下3万2万1.5万1万3万2万1.5万1万3万2万1.5万1万
70/3527.00 22.00 18.00 32.00 93.00 91.00 90.00 0.00 28.50 15.00 14.00 12.00
30/595.50 99.00 99.00 95.50 100.00 100.00 100.00 100.00 99.50 100.00 100.00 100.00
3/196.50 96.50 97.50 95.00 94.00 96.50 97.00 93.00 96.00 96.50 97.50 95.50