The processing of results

Message boards : Number crunching : The processing of results
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 . . . 8 · Next

AuthorMessage
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 328 - Posted: 4 Oct 2018, 3:54:33 UTC
Last modified: 4 Oct 2018, 4:43:42 UTC

I return to the issue of processing the results of ODLK and ODLK1 projects.
I have repeatedly raised this issue.
I raised this question in personal correspondence and on the forum.
The problem is still not solved.

ODLK project launched May 19, 2017.
ODLK1 project launched November 5, 2017.
All this time I have been processing the results of both projects manually.
This is very bad!

The procedure for processing the results must be automated!

I processed the results of the ODLK project for September. There are few results in this project and less time is required for processing than for results from the ODLK1 project.
I will no longer manually process the results. This is nonsense!
We have two operational BOINC projects and have not automated the processing of results.

Now about the primary results in projects.
As I noted earlier, the results in the ODLK project are open and accessible to all.
In the project ODLK1 results are closed.
If the administrators of the project ODLK1 Progger and ice00 do not want to automate the processing of results, let them open the results.
Then the primary results will be available to everyone. Anyone can handle them yourself.

But opening the results, remember the story
https://boinc.progger.info/odlk/forum_thread.php?id=69

Recently, Belyshev made a wonderful script for primary and secondary processing of results.
I can tell about this script, if administrators are interested.
If they are still not interested in automatic processing of the results, then you do not need to talk about the script.

So, I wait for the answer of administrators.
ID: 328 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 330 - Posted: 4 Oct 2018, 18:23:45 UTC - in response to Message 328.  

If you have the algorithm or script for processing the result, then I'm interested for making the automation.
ID: 330 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 333 - Posted: 4 Oct 2018, 18:31:55 UTC - in response to Message 330.  

If you have the algorithm or script for processing the result, then I'm interested for making the automation.

Good.
Tomorrow I will talk in detail about the script by Belyshev.
ID: 333 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 334 - Posted: 5 Oct 2018, 0:29:50 UTC
Last modified: 5 Oct 2018, 0:47:26 UTC

Progger, ice00

I start the story about the script by Belyshev.

whitefox;91190 wrote:
Для автоматизации выполнения замыкания, в архив программы Канонизатор_ЛК_по_ДЛК помещён скрипт zamyk.bat.

See
http://forum.boinc.ru/default.aspx?g=posts&m=91190#post91190

Download link archive
https://yadi.sk/d/jga_oPqjuhuYC

Now you have to write the results to the file input.txt and run the zamyk.bat script.
All CF ODLS will be written to the file out_klpmd.txt.

You can process the results every day. Or you can process the results for the month.

The software made for Windows.

We already have a large database CF ODLS.
Some CF ODLS from the out_klpmd.txt file may be contained in an existing database.
Therefore, CF ODLS needs to be checked for uniqueness.
For this, I use the utility by a colleague V. Chirkov.
This is a simple utility. A similar utility is available by Belyshev.

You can write this utility yourself.

Good luck!
ID: 334 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 336 - Posted: 25 Oct 2018, 9:39:17 UTC
Last modified: 26 Oct 2018, 1:04:11 UTC

Hello, ice00!

Did you process the results from the ODLK1 project for September?
Or do you have a problem?

I am showing an example.
I took the results for October 1, 2018 from the project ODLK1 (Application odlk3).
There are 3940 CF ODLS.
I write these solutions to the file input.txt and run the zamyk.bat script.

Script work protocol

Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 3940

Проверено ЛК: 82 скорость 4644 ДЛК в сек
Проверено ЛК: 163 скорость 4738 ДЛК в сек
Проверено ЛК: 244 скорость 4698 ДЛК в сек
Проверено ЛК: 324 скорость 4707 ДЛК в сек
Проверено ЛК: 405 скорость 4746 ДЛК в сек
. . . . . 
Проверено ЛК: 3673 скорость 4715 ДЛК в сек
Проверено ЛК: 3754 скорость 4705 ДЛК в сек
Проверено ЛК: 3834 скорость 4701 ДЛК в сек
Проверено ЛК: 3914 скорость 4710 ДЛК в сек

Проверено ДЛК           : 1156893
Найдено КФ марьяжных ДЛК: 4179
Время работы в сек      : 248.249

*************************************************
Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 4186

Проверено ЛК: 82 скорость 4674 ДЛК в сек
Проверено ЛК: 162 скорость 4699 ДЛК в сек
. . . . 
Проверено ЛК: 3856 скорость 4091 ДЛК в сек
Проверено ЛК: 3937 скорость 4715 ДЛК в сек
Проверено ЛК: 4005 скорость 3998 ДЛК в сек
Проверено ЛК: 4072 скорость 3920 ДЛК в сек
Проверено ЛК: 4142 скорость 4079 ДЛК в сек

Проверено ДЛК           : 1225324
Найдено КФ марьяжных ДЛК: 4193
Время работы в сек      : 285.403

. . . . 

Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 8148

Проверено ЛК: 59 скорость 3380 ДЛК в сек
Проверено ЛК: 123 скорость 3620 ДЛК в сек
Проверено ЛК: 192 скорость 3978 ДЛК в сек
Проверено ЛК: 244 скорость 3055 ДЛК в сек
Проверено ЛК: 309 скорость 3804 ДЛК в сек
. . . . . 
Проверено ЛК: 7906 скорость 4673 ДЛК в сек
Проверено ЛК: 7987 скорость 4694 ДЛК в сек
Проверено ЛК: 8067 скорость 4689 ДЛК в сек
Проверено ЛК: 8148 скорость 4680 ДЛК в сек

Проверено ДЛК           : 2388615
Найдено КФ марьяжных ДЛК: 8396
Время работы в сек      : 579.438

Найдено марьяжных КФ:
count[1] = 8376
count[2] = 20
Всего: 8396
Найдено соквадратов: 8416
КФ соквадратов: 8396

Для продолжения нажмите любую клавишу . . .

Found 8396 CF ODLS. They are written to the out_klpmd.txt file.

Please note: CF ODLS is written to the out_klpmd.txt file in format 2.
Our database CF ODLS is in format 1.
To check the CF ODLS for uniqueness, you must first convert them to format 1.
But you can continue the CF ODLS database in format 2.

This is an example of processing the results for the day.
The processing of results for the month lasts a very long time.
Therefore, it is necessary to automate this processing.
But as I see it has not been done yet (?)
ID: 336 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 337 - Posted: 25 Oct 2018, 10:55:18 UTC
Last modified: 25 Oct 2018, 10:57:45 UTC

Now I am setting an example for everyone.
I took the results for October 1, 2018 from the ODLK project at the following link
https://boinc.progger.info/odlk_results/odlk3/2018-10-01.txt

This is 950 CF ODLS.
I wrote these CF ODLS in the input.txt file and ran the zamyk.txt script.

Script work protocol

Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 950

Проверено ЛК: 78 скорость 4606 ДЛК в сек
Проверено ЛК: 158 скорость 4619 ДЛК в сек
Проверено ЛК: 239 скорость 4692 ДЛК в сек
Проверено ЛК: 319 скорость 4685 ДЛК в сек
Проверено ЛК: 400 скорость 4707 ДЛК в сек
Проверено ЛК: 479 скорость 4700 ДЛК в сек
Проверено ЛК: 560 скорость 4702 ДЛК в сек
Проверено ЛК: 640 скорость 4649 ДЛК в сек
Проверено ЛК: 721 скорость 4658 ДЛК в сек
Проверено ЛК: 801 скорость 4682 ДЛК в сек
Проверено ЛК: 815 скорость 801 ДЛК в сек
Проверено ЛК: 896 скорость 4690 ДЛК в сек

Проверено ДЛК           : 278285
Найдено КФ марьяжных ДЛК: 950
Время работы в сек      : 63.742

*************************************************
Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 950

Проверено ЛК: 77 скорость 4543 ДЛК в сек
Проверено ЛК: 156 скорость 4667 ДЛК в сек
Проверено ЛК: 237 скорость 4760 ДЛК в сек
Проверено ЛК: 318 скорость 4698 ДЛК в сек
Проверено ЛК: 399 скорость 4686 ДЛК в сек
. . . . . 
Проверено ЛК: 633 скорость 4217 ДЛК в сек
Проверено ЛК: 711 скорость 4678 ДЛК в сек
Проверено ЛК: 793 скорость 4720 ДЛК в сек
Проверено ЛК: 875 скорость 4717 ДЛК в сек

Проверено ДЛК           : 279372
Найдено КФ марьяжных ДЛК: 953
Время работы в сек      : 60.138

*************************************************
Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 7

Проверено ДЛК           : 2095
Найдено КФ марьяжных ДЛК: 7
Время работы в сек      : 0.468

*************************************************
╚чтыхўхэшх шэЇюЁьрЎшш

=================================================
Канонизатор ЛК10

Введено ЛК   : 1910
Найдено КФ ЛК: 1907
Время работы : 0.124 сек

Поиск всех КФ марьяжных ДЛК для ЛК

Введено ЛК: 1907

Проверено ЛК: 76 скорость 4471 ДЛК в сек
Проверено ЛК: 155 скорость 4673 ДЛК в сек
Проверено ЛК: 202 скорость 2740 ДЛК в сек
Проверено ЛК: 277 скорость 4468 ДЛК в сек
. . . . . 
Проверено ЛК: 1552 скорость 4712 ДЛК в сек
Проверено ЛК: 1632 скорость 4700 ДЛК в сек
Проверено ЛК: 1714 скорость 4756 ДЛК в сек
Проверено ЛК: 1796 скорость 4723 ДЛК в сек
Проверено ЛК: 1876 скорость 4722 ДЛК в сек

Проверено ДЛК           : 559752
Найдено КФ марьяжных ДЛК: 1910
Время работы в сек      : 122.725

Найдено марьяжных КФ:
count[1] = 1898
count[2] = 12
Всего: 1910
Найдено соквадратов: 1922
КФ соквадратов: 1910

Для продолжения нажмите любую клавишу . . .

Found 1910 CF ODLS. They are written to the out_klpmd.txt file in format 2.

What is not clear in this example?
ID: 337 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 338 - Posted: 30 Oct 2018, 20:21:28 UTC - in response to Message 337.  

Hi,

having problem in translate Russian text in source not coded in UTF-8 (so Google translate did not process it) into English.
ID: 338 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 339 - Posted: 31 Oct 2018, 3:39:10 UTC - in response to Message 338.  


having problem in translate Russian text in source not coded in UTF-8 (so Google translate did not process it) into English.

Unfortunately, I can not help you with this.

Ask for help to the software developer Belyshev.
You can also ask for help from Progger.
https://yadi.sk/i/S1HhJBwp3Pr9jX
ID: 339 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 344 - Posted: 3 Nov 2018, 15:15:36 UTC - in response to Message 339.  


having problem in translate Russian text in source not coded in UTF-8 (so Google translate did not process it) into English.

Unfortunately, I can not help you with this.

Ask for help to the software developer Belyshev.
You can also ask for help from Progger.


Get the translation in English.

Source code did not provide all programs that the bat script are calling (like IZVL, SORT_COUNT and TYPE_COUNT).
Are this available or should be created from scratch?

Thanks
ID: 344 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 345 - Posted: 3 Nov 2018, 15:44:27 UTC - in response to Message 344.  
Last modified: 3 Nov 2018, 15:47:42 UTC


Source code did not provide all programs that the bat script are calling (like IZVL, SORT_COUNT and TYPE_COUNT).
Are this available or should be created from scratch?

The fact is that I use script executables in the Windows system.
I am not compiling anything again.
So I don't need the source codes.

The script developer Belyshev can answer on this question.
https://yadi.sk/i/S1HhJBwp3Pr9jX
ID: 345 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 356 - Posted: 19 Nov 2018, 18:51:39 UTC
Last modified: 19 Nov 2018, 18:57:04 UTC

ice00
You did not solve the issue of processing the results?

Results in the ODLK1 project are not processed since September 1, 2018.
The situation is critical.
Results in the project are closed.
Project participants do not see results.

This is an abnormal situation.
If you are not able to process the results, you must open them.

Progger ignores my calls altogether.
Results must be made available to participants.
Do you understand this ???

Please indicate what you intend to do in this situation.
ID: 356 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 357 - Posted: 30 Nov 2018, 5:42:49 UTC
Last modified: 30 Nov 2018, 5:43:15 UTC

ice00
I am waiting for an answer to my question.

What did you do to process the results?
What do you intend to do?
Three months results in the ODLK1 project are not processed and not published.
In such a situation the results may be lost.

I suggest you publish unprocessed results for each month if you are not able to do automatic processing of the results.
ID: 357 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 358 - Posted: 1 Dec 2018, 12:40:28 UTC - in response to Message 357.  

ice00
I am waiting for an answer to my question.

What did you do to process the results?
What do you intend to do?
Three months results in the ODLK1 project are not processed and not published.
In such a situation the results may be lost.

I suggest you publish unprocessed results for each month if you are not able to do automatic processing of the results.


hi,

I had the complete sources and I'm still creating and testing the scripts needed to automate the process (that involve two servers).
Do not expect (automatic processed) result published until Christmas due to my working deadlines and two intensive courses I'm making every days (70+ hours of courses to make in two months, to be followed not during working time).

I can instead publish unprocessed data tomorrow.
ID: 358 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 359 - Posted: 1 Dec 2018, 13:32:51 UTC - in response to Message 358.  

I can instead publish unprocessed data tomorrow.

OK. Publishe unprocessed results tomorrow.

Merry Christmas!
I hope that by Christmas 2019 you will do automatic processing of the results :)
ID: 359 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 360 - Posted: 1 Dec 2018, 17:10:10 UTC

Unprocessed result should appears onto those link (It is not necessary to have a Google account to download it)

https://drive.google.com/open?id=1wIf97_bLDvnb1WalOsjU9vODZqRZ_WNw
ID: 360 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 361 - Posted: 2 Dec 2018, 2:50:22 UTC - in response to Message 360.  
Last modified: 2 Dec 2018, 2:51:53 UTC

ice00
Thanks.
I downloaded unprocessed results for October and November.
However, you forgot to publish the results for September.

The last processed results for August posted here
https://boinc.multi-pool.info/latinsquares/forum_thread.php?id=46&postid=287#287

Please publish the results for September.
Then I will publish a short report.
ID: 361 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 362 - Posted: 2 Dec 2018, 9:54:34 UTC - in response to Message 361.  

September added.

Thanks
ID: 362 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 363 - Posted: 3 Dec 2018, 0:55:46 UTC

I previewed the results, only the initial processing.
Unique ODLS pair groups were found that are published in the following topics

https://boinc.progger.info/odlk/forum_thread.php?id=42
https://boinc.progger.info/odlk/forum_thread.php?id=40

Full processing of the results requires a lot of time and computational resources.
I hope that ice00 will finish the work on preparing the automatic processing of the results.

Dear participants!
Now you can complete processing of the results yourself.
The above is how to do this.

I hope that soon ice00 will automatically process the results.
ID: 363 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Profile Natalia Makarova
Project scientist
Avatar

Send message
Joined: 22 Oct 17
Posts: 3083
Credit: 0
RAC: 0
Message 532 - Posted: 2 Jan 2019, 7:17:24 UTC

ice00
please publish the results from the ODLK1 project for December.
ID: 532 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
ice00
Project administrator
Project developer

Send message
Joined: 28 Oct 17
Posts: 189
Credit: 57,700
RAC: 4
Message 584 - Posted: 10 Jan 2019, 17:26:30 UTC - in response to Message 532.  

ice00
please publish the results from the ODLK1 project for December.


Hi,
already published on 1/1/2019.
The files are all in the folder for ODLK1 result: https://drive.google.com/drive/u/0/folders/1wIf97_bLDvnb1WalOsjU9vODZqRZ_WNw (it should be open to everyone)
ID: 584 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · 3 · 4 . . . 8 · Next

Message boards : Number crunching : The processing of results


©2024 Progger & Stefano Tognon (ice00)