The processing of results

Message boards : Number crunching : The processing of results
Message board moderation

To post messages, you must log in.

Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 8 · Next

AuthorMessage
Tomas Brada

Send message
Joined: 22 Jan 18
Posts: 63
Credit: 373,915
RAC: 0
Message 681 - Posted: 21 Jan 2019, 12:22:19 UTC - in response to Message 680.  

The report shows first which file it is processing and how many squares were in the file, then if duplicate is found, it will show the square as in the file and if that square is not diagonal cf, then also kf of the duplicate.
ID: 681 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 682 - Posted: 21 Jan 2019, 12:27:09 UTC
Last modified: 21 Jan 2019, 12:29:20 UTC

Are these duplicates in the rez_ODLK1_Oct_F2.txt part???

Processing rez_ODLK1_Oct_F2.txt : 404478
duplicate (orig) :
0 2 3 4 5 7 8 6 9 1
6 1 5 8 3 9 7 4 0 2
5 4 2 6 0 1 9 3 7 8
7 6 4 3 9 8 2 1 5 0
3 8 0 7 4 6 1 9 2 5
9 3 1 2 7 5 0 8 4 6
8 7 9 5 1 2 6 0 3 4
2 0 8 9 6 4 5 7 1 3
1 9 6 0 2 3 4 5 8 7
4 5 7 1 8 0 3 2 6 9

duplicate (orig) :
0 3 4 8 5 2 9 6 7 1
3 1 5 4 6 9 7 2 0 8
5 0 2 6 9 8 1 3 4 7
7 8 1 3 0 4 2 9 6 5
2 5 3 7 4 6 8 1 9 0
1 6 0 9 7 5 4 8 3 2
8 7 9 5 3 1 6 0 2 4
9 4 8 1 2 3 0 7 5 6
6 9 7 2 1 0 5 4 8 3
4 2 6 0 8 7 3 5 1 9

rez_ODLK1_Oct_F2.txt -: in: 404478 not lk: 0 not dlk: 0 dup: 2 new uniqe kf dlk: 404476

But in the rez_ODLK1_Oct_F2.txt part there are no two such DLS!

In addition, the kanonizator_dlk_1.03 program reports that in this part there are 404478 CFs.

Канонизатор ДЛК10

Загружена хеш-таблица
Время загрузки:         0.015 сек
Введено ДЛК:            404478
Время загрузки:         4.758 сек
Найдено КФ:             404478
Время поиска:           0.546 сек
КФ записаны в файл:     output.txt
Время записи:           0.375 сек
Общее время работы:     5.694 сек
ID: 682 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 683 - Posted: 21 Jan 2019, 12:40:33 UTC

rez_ODLK1_Oct_F2.txt -: in: 404478 not lk: 0 not dlk: 0 dup: 2 new uniqe kf dlk: 404476

Maybe these “2 new uniqe kf dlk” are contained in the previous parts?

In which parts?
ID: 683 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 684 - Posted: 21 Jan 2019, 12:52:31 UTC
Last modified: 21 Jan 2019, 12:55:11 UTC

Tomas Brada
I understand that you should not solve this problem.
Excuse me.

I wanted to understand where the error is.
But I do not understand this :(

The problem is that I cannot merge all the parts into one file.
Therefore, I check each part separately.
ID: 684 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomas Brada

Send message
Joined: 22 Jan 18
Posts: 63
Credit: 373,915
RAC: 0
Message 685 - Posted: 21 Jan 2019, 12:56:30 UTC - in response to Message 683.  

The program works across all files. It goes like this: first file is read, the cf dls are added to memory set. Then the second file is read, each square is converted to cf dlk and checked against the memory set if it already exists there, if not that means it is unique and is added to memory set, if it already is in the memory set, this duplicate is reported and not added again.

rez_ODLK1_Oct_F2.txt -: in: 404478 not lk: 0 not dlk: 0 dup: 2 new uniqe kf dlk: 404476

rez_ODLK1_Oct_F2.txt - input file
in: 404478 - 404478 squares in input file
not lk: 0 - no squares failed the is_lk() check
not dlk: 0 - no squares failed is_dlk() check
dup: 2 - two duplicates
new uniqe kf dlk: 404476 - 404476 cf dls added to memory set

That means there are two KF DLK which already are in at least one of the previous files.
ID: 685 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 686 - Posted: 21 Jan 2019, 13:11:56 UTC - in response to Message 685.  
Last modified: 21 Jan 2019, 13:15:58 UTC

That means there are two KF DLK which already are in at least one of the previous files.

The fact of the matter is that I did not find these two DLS in previous files!

Can you tell exactly in which part are these two DLS?

I see these two DLS only in the rez_ODLK1_Oct_F2.txt part.
ID: 686 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 687 - Posted: 21 Jan 2019, 13:22:40 UTC

I found!

These two DLS are contained in the part addition_results_67879.txt.
No more questions.

Thanks.
ID: 687 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 691 - Posted: 22 Jan 2019, 9:46:54 UTC
Last modified: 22 Jan 2019, 9:49:17 UTC

I published the results of the project ODLK for May - December 2018
https://cloud.mail.ru/public/6igF/E4JFS4ZYy

385041 unique CFs ODLS

I checked the results for uniqueness regarding the results of the project ODLK1.

Tomas Brada
can you check these results for uniqueness with respect to the database of 5,329,884 GFs?
ID: 691 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomas Brada

Send message
Joined: 22 Jan 18
Posts: 63
Credit: 373,915
RAC: 0
Message 693 - Posted: 22 Jan 2019, 10:12:53 UTC - in response to Message 691.  

db unique kf dlk: 5329884
Processing rez_ODLK_May-dec_F2.txt : 385041
rez_ODLK_May-dec_F2.txt -: in: 385041, not lk: 0, not dlk: 0, dup: 0, new uniqe kf dlk: 385041                           
db unique kf dlk: 5714925

No duplicates with respect to the 5329884 database. 385041 new. Now 5714925 in the database.
ID: 693 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 695 - Posted: 22 Jan 2019, 10:28:20 UTC - in response to Message 693.  

db unique kf dlk: 5329884
Processing rez_ODLK_May-dec_F2.txt : 385041
rez_ODLK_May-dec_F2.txt -: in: 385041, not lk: 0, not dlk: 0, dup: 0, new uniqe kf dlk: 385041                           
db unique kf dlk: 5714925

No duplicates with respect to the 5329884 database. 385041 new. Now 5714925 in the database.

Thank you very much!

Fine!
ID: 695 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 697 - Posted: 22 Jan 2019, 11:46:32 UTC
Last modified: 22 Jan 2019, 14:32:03 UTC

I introducing the new DLS recording format by Belyshev
http://forum.boinc.ru/default.aspx?g=posts&m=90094#post90094

Download the namer.zip archive
Further you watch the redme.txt file

Example

0 2 4 5 7 3 9 8 6 1
8 1 6 4 5 2 3 9 0 7
7 8 2 6 0 4 1 3 9 5
1 9 7 3 2 8 4 6 5 0
9 5 3 0 4 6 2 1 7 8
4 6 1 9 8 5 7 0 3 2
3 0 8 2 9 7 6 5 1 4
6 4 9 8 1 0 5 7 2 3
2 7 5 1 3 9 0 4 8 6
5 3 0 7 6 1 8 2 4 9

0 3 7 6 5 9 2 8 4 1
3 1 6 2 9 8 7 5 0 4
9 6 2 1 0 4 8 3 5 7
6 2 1 3 8 0 4 9 7 5
5 8 0 7 4 2 9 1 6 3
2 7 3 0 6 5 1 4 9 8
4 9 5 8 3 7 6 0 1 2
8 4 9 5 1 3 0 7 2 6
1 5 4 9 7 6 3 2 8 0
7 0 8 4 2 1 5 6 3 9

0 4 5 7 8 6 9 2 3 1
5 1 3 8 7 4 2 9 0 6
8 7 2 9 0 1 5 3 6 4
1 9 8 3 5 2 4 6 7 0
3 5 0 6 4 8 7 1 9 2
6 8 7 2 9 5 0 4 1 3
2 3 9 5 1 7 6 0 4 8
4 0 6 1 3 9 8 7 2 5
9 2 1 4 6 0 3 5 8 7
7 6 4 0 2 3 1 8 5 9

0 4 5 8 6 9 7 2 3 1
8 1 6 4 9 2 3 5 0 7
9 6 2 7 0 1 5 3 4 8
2 5 7 3 1 8 4 0 9 6
7 3 0 9 4 6 8 1 2 5
6 9 1 0 8 5 2 4 7 3
3 0 8 2 5 7 6 9 1 4
4 8 9 6 3 0 1 7 5 2
1 7 4 5 2 3 9 6 8 0
5 2 3 1 7 4 0 8 6 9

0 4 8 7 5 6 3 2 9 1
8 1 7 4 9 3 5 6 0 2
9 8 2 5 7 0 1 3 6 4
5 6 0 3 8 1 4 9 2 7
6 9 5 1 4 2 7 0 3 8
4 2 1 9 6 5 0 8 7 3
2 7 4 8 3 9 6 5 1 0
3 0 9 6 1 8 2 7 4 5
1 5 3 2 0 7 9 4 8 6
7 3 6 0 2 4 8 1 5 9

0 4 9 8 5 7 2 6 3 1
9 1 8 4 6 2 3 5 0 7
8 5 2 6 0 1 9 3 7 4
5 7 1 3 8 0 4 2 9 6
3 9 0 7 4 8 5 1 6 2
6 8 3 2 9 5 7 4 1 0
2 3 7 5 1 9 6 0 4 8
1 0 6 9 3 4 8 7 2 5
4 2 5 0 7 6 1 9 8 3
7 6 4 1 2 3 0 8 5 9

0 5 4 8 2 3 7 6 9 1
3 1 7 6 9 8 2 5 0 4
4 8 2 7 0 1 9 3 5 6
1 6 8 3 5 7 4 9 2 0
2 9 0 5 4 6 8 1 7 3
7 0 3 9 8 5 1 4 6 2
8 3 5 2 1 9 6 0 4 7
6 2 9 0 3 4 5 7 1 8
9 7 1 4 6 0 3 2 8 5
5 4 6 1 7 2 0 8 3 9

0 5 4 8 6 7 3 9 2 1
8 1 9 4 7 3 2 5 0 6
9 0 2 1 8 4 7 3 6 5
6 9 8 3 0 2 4 1 5 7
5 6 3 9 4 8 1 2 7 0
1 4 7 6 9 5 0 8 3 2
7 2 0 5 3 1 6 4 9 8
4 8 6 0 2 9 5 7 1 3
3 7 1 2 5 6 9 0 8 4
2 3 5 7 1 0 8 6 4 9

0 6 7 4 8 9 3 2 5 1
9 1 4 7 0 3 5 6 2 8
4 5 2 0 9 8 1 3 6 7
2 8 0 3 5 7 4 1 9 6
7 2 3 8 4 6 0 9 1 5
6 3 1 2 7 5 9 8 4 0
1 9 8 5 3 2 6 0 7 4
5 4 9 6 2 1 8 7 0 3
3 0 6 9 1 4 7 5 8 2
8 7 5 1 6 0 2 4 3 9

0 8 4 7 2 3 5 6 9 1
9 1 3 6 7 4 8 5 2 0
4 5 2 1 0 9 7 3 6 8
1 9 7 3 6 8 4 2 0 5
2 7 0 8 4 6 9 1 5 3
3 4 6 2 8 5 0 9 1 7
5 2 1 9 3 0 6 8 7 4
8 3 5 0 9 1 2 7 4 6
6 0 9 5 1 7 3 4 8 2
7 6 8 4 5 2 1 0 3 9

These DLSs in new format

7TQf7wSBLMRgpXjfS72KiPwQ3
7c1cX663mMc6FYAPvGsBDVBHB
C2q8vwLHiRe3Cr2aZM2ziWbmh
C8VTvQa4q74CgWQTHDdZgJDmj
CDwdqJLJyM17qm6UoGEbQBibi
GxUE4weihQzxenLNTJDaEHVUC
HDhB2Vk3s5zzvXoJg9sq1f3SQ
HniXR5hQT5sRfULymSTk4gCmz
TuFMHpbuxQBVbcS8g5yt74FHw
UaaA2GbMf74zeVM67NkYou6yi

Important note!
The program namer.exe works only with DLS in format 2.
ID: 697 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Tomas Brada

Send message
Joined: 22 Jan 18
Posts: 63
Credit: 373,915
RAC: 0
Message 702 - Posted: 22 Jan 2019, 13:43:22 UTC - in response to Message 697.  

Do you think it is worth the hassle to introduce new format?

Btw: good compression ratio archived! To encode arbitrary (non-latin) 10x10 square of 0-9 it would take 56 characters. This is only 25.
ID: 702 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 703 - Posted: 22 Jan 2019, 13:52:32 UTC - in response to Message 702.  
Last modified: 22 Jan 2019, 14:16:54 UTC

Do you think it is worth the hassle to introduce new format?

Yes, in the future, necessarily, because the database is increasing.

Can you transfer to me or ice00 DB from 5714925 CFs in a new format?
How it is MB?
ID: 703 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 710 - Posted: 26 Jan 2019, 5:34:39 UTC
Last modified: 26 Jan 2019, 5:35:19 UTC

ice00
January ends.
Are you ready to process results with software developed by Tomas Brada?
Can you get the entire database from 5714925 unique CFs ODLS to check for new results for uniqueness?
Or did you not do anything to solve this problem?

Will you still publish raw results?
ID: 710 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 713 - Posted: 27 Jan 2019, 9:21:16 UTC - in response to Message 710.  

I'm actually too busy with working issues to being able to code/test something in free time due to stringent deadlines I have.

Providing raw not processed result is temporary a workaround that with the script already done it takes only 1 minute.
ID: 713 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 714 - Posted: 27 Jan 2019, 10:15:11 UTC - in response to Message 713.  
Last modified: 27 Jan 2019, 10:20:49 UTC

I'm actually too busy with working issues to being able to code/test something in free time due to stringent deadlines I have.

In this case, you do not need to administer the BOINC project!
I suggest you stop it.

Providing raw not processed result is temporary a workaround that with the script already done it takes only 1 minute.

Temporary a workaround? How long? For 2 years? For 5 years?
You already for 4 months published the raw results.
January is the fifth month.
Who will process the results?
I can't do it on my PC.

There are two administrators in the ODLK1 project.
The results are not processed! New Applications do not run!

Where is the work of administrators???
ID: 714 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 734 - Posted: 16 Apr 2019, 18:21:27 UTC - in response to Message 358.  

I had the complete sources and I'm still creating and testing the scripts needed to automate the process (that involve two servers).
Do not expect (automatic processed) result published until Christmas due to my working deadlines and two intensive courses I'm making every days (70+ hours of courses to make in two months, to be followed not during working time).

I can instead publish unprocessed data tomorrow.

ice00
by next Christmas, will you automatically processing the results? :)
ID: 734 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 737 - Posted: 21 Apr 2019, 19:08:00 UTC - in response to Message 734.  

It is two weeks that I'm working on it. Being testing some new stuff to be sure it process all in the right way.
ID: 737 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 739 - Posted: 22 Apr 2019, 18:34:33 UTC - in response to Message 737.  

It is two weeks that I'm working on it.

And what are your successes?
Can you process the results for January - March 2019 with the script of Belyshev zamyk.bat?
So far you have posted only the raw results for this period.
ID: 739 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 741 - Posted: 25 Apr 2019, 17:25:41 UTC - in response to Message 739.  

First step achieved: having only one file from ODLK1 with unique squares by automate the process of adding new month result of the server.

For the periods of 2018-05 .. 2019-03 ODLK1 has: 2086518 unique CFs ODLS.

Step 2 with Belyshev zamyk.bat equivalent processing will follow as soon as it works correctly.
ID: 741 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Previous · 1 · 2 · 3 · 4 · 5 · 6 . . . 8 · Next

Message boards : Number crunching : The processing of results


©2024 Progger & Stefano Tognon (ice00)