Solver Statistics - Aymeric du Peloux

2007-03-26T14:33:26Z

Minglw: Updated the summary with new results from YASS (using v2.66)

This is a summary of running what I called the Aymeric collection (282 levels total) using these solvers:<br>
• BoxSearch v5.1 beta2 <br>
• Takaken v7.0<br>
• YASS v2.66<br>
<br>
The Aymeric collection is a collection of level sets by Aymeric Du Peloux (282 levels total).
It consists of these level sets (and in this order):<br>
• COSMONOTES ( 20 levels)<br>
• COSMOPOLY ( 22 levels)<br>
• LOMA (100 levels) -- Note: this is a collection by many authors.<br>
• MICROCOSMOS ( 40 levels)<br>
• MINICOSMOS ( 40 levels)<br>
• NABOKOSMOS ( 40 levels)<br>
• PICOKOSMOS ( 20 levels)<br>
<br>
For copyrights, credits and redistribution of these statistics, please read the [[Solvers#Disclaimer | disclaimer]].<br>
<br>
Though some people might be interested in how long it takes to solve a given level and/or how good the solution is (optimal or not), this summary disregards all timing associated with solving the levels and solution statistics.
These tests are mainly for gauging solvability, not timing nor how good the solution is.

<div class="note-2">
<u>Not all Sokoban solvers are created equal:</u><br>
*BoxSearch v5.1 beta2 allows setting for max time limit and max memory.
*Takaken v7.0 allows setting for max time but lacks max memory option.
*YASS v2.66 allows setting for max time and max memory option.

<u>Test machine info:</u><br>
*Machine: ThinkPad T42P
*CPU: x86 Family 6 Model 13 Stepping 6 Genuine Intel
*Physical RAM: 2 GB
*OS Name & Version: Microsoft Windows XP Professional v5.12600 Service Pack 2 Build 2600.

<u>Initial tests:</u><br>
*BoxSearch is set with max memory limit of 1500 MB and 600 seconds. (using quick mode)
*Takaken is set with 600 seconds time limit
*YASS is set with 1000 MB memory and 600 second limit.

<u>Further tests:</u><br>
*(none so far)
</div>

<div class="box-2">
<b>Note:</b>
Each solver is tested separately to ensure that they don't compete for system resources.
</div>

The following table shows the levels where each of the solvers failed:
======================================================================

BoxSearch Takaken YASS
============== ============== ==============
275
============== ============== ==============
Total: 0 1 0

Total (unique level count): 1

All solvers failed on these levels:
===================================================================================

===================================================================================
Total count: 0

"BoxSearch" failed on these levels where at least one solver solved:
===================================================================================

===================================================================================
Total count: 0

"Takaken" failed on these levels where at least one solver solved:
===================================================================================
275
===================================================================================
Total count: 1

"YASS" failed on these levels where at least one solver solved:
===================================================================================

===================================================================================
Total count: 0

"BoxSearch" solved these levels where all other solvers failed:
===================================================================================

===================================================================================
Total count: 0

"Takaken" solved these levels where all other solvers failed:
===================================================================================

===================================================================================
Total count: 0

"YASS" solved these levels where all other solvers failed:
===================================================================================

===================================================================================
Total count: 0

"BoxSearch" failed on these levels where all other solvers solved:
===================================================================================

===================================================================================
Total count: 0

"Takaken" failed on these levels where all other solvers solved:
===================================================================================
275
===================================================================================
Total count: 1

"YASS" failed on these levels where all other solvers solved:
===================================================================================

===================================================================================
Total count: 0

<div class="box margin-2"><div style="width:500px;">
BoxSearch solved 282 of 282 levels.
<div class="box-progress">
<div class="small" style="width: 100%;">BoxSearch 100%</div>
</div>

Takaken solved 281 of 282 levels.
<div class="box-progress">
<div class="small" style="width: 99.6%;">Takaken 99.6%</div>
</div>

YASS solved 282 of 282 levels.
<div class="box-progress">
<div class="small" style="width: 100%;">YASS 100%</div>
</div>

All solvers combined solved 282 of 282 levels.
<div class="box-progress">
<div class="small" style="width: 100%;background:rgb(125,190,255);">All solvers 100%</div>
</div>

</div></div>

Talk:Solver Statistics - XSokoban - Thinking Rabbit & Various Authors

2007-03-08T04:10:12Z

Minglw:

Hallo Brian, hallo Matthias, hi Minglw!

(sorry, only in german at time)

Ich würde mir wünschen, dass wir die Levels, die im Wiki unter dem Namen XSokoban laufen, auseinandernehmen. Dafür sollten wir Minglw bitten, entsprechende Listen anzufertigen, sofern entsprechende Levels im Test fehlen, was ja bei den echten Original der Fall sein wird, sollten wir Minglw diese Levels nachliefern.
Ich möchte der Tendenz in der Szene entgegenwirken Originallevels und XSokoban in einem Atemzug zu benutzen (zu nennen).

Ich würde vorschlagen, dass die Levels aufgeteilt werden, in die 50 Originals von Thinking Rabbit (aus Spectrum Holobyte's -> Soko-Ban), nachträglich angefügt die 6 wirklichen Abweichungen (demolished levels by XSokoban, also 12, 20, 25, 37, 42, 50).
Ich will betonen, dass die 6 Levels von XSokoban für mich keine Variationen im besten Wortsinne sind, sondern eigentlich nur verunstaltete...

Desweiteren würde ich gern die szeneintern "Extra levels" genannten XSokoban Levels, also Level 51 bis 90 gerne von den Originals separieren.

Somit würde auch an dieser Stelle klar hervortreten, und das sehe ich als Mission, dass XSokoban Levels KEINE Originals sind.

Beste Grüße

Sokobano

The XSokoban/90 level set is THE established test set for Sokoban solver benchmarking, and there is no way to change that. Many academic papers dealing with Sokoban solvers refer to these levels, so despite the awkward background (copyrighted levels, wrongly called "XSokoban levels" instead of giving proper credits to "Thinking Rabbit" for 50 of the levels, modified levels, and not to the better), this set is eminently qualified for a statistical survey like the one here on the Wiki.

-bd

<div class="box-3">
It's maybe a good idea to have a second column on that page for the Original levels. This way everybody can see that there are some levels where the two collections differ. There the "40 extra" levels can also be separated from the 50 Original levels.<br>
I also think that we shouldn't separate the 90 XSokoban levels because there are many comparisons using these levels (Rolling Stone, Takaken, ...) Hence, it's easier to compare the result on the Wiki with those pages when the "90 XSokoban" levels are listed. <br>
Matthias
</div>

<div style="background-color:#F0FFF0;padding:5px;border:1px solid green;">
Well, the purpose of these statistics is to show what the Sokoban Solvers can solve.
We are not here trying to infringe on copyrights or whether the set's content is proper or not.
With that, I don't see there's further need to discuss about the content of the sets.

So far, I have purposely chosen level sets that are publicly available.
The only exception is BoxWorld, but the program is freely available for everyone to download, and there is a program freely available out there for people to extract the levels from BoxWorld. Bottom line is, people have access to the levels freely.
That's not to say that I won't release statistics on sets that are commercially available.
I am not here to release the content/design of the level sets anyway, just the statistics on whether the solvers can solve them or not.

The main reason that I've chosen level sets that are freely available is so that people can
verify the results if they want. Better yet, people with a faster computer and more RAM can
help to test and update the results here. If people do that, I would appreciate it if they can send
me the output file(s) from the solvers proving that the levels are indeed solved. (it's just a way to verify the stats are good). It's not good enough to send me just the solution for the level(s), that didn't prove anything other than saying that you have the solution to a given level.

Let's get back on topic with XSokoban statistics....
The current presentation of the XSokoban Statistics are nice.
However, it's difficult to make any updates to them (unless a script/program is created to convert from the existing plain text format into some nice looking format.

Recently, I've some updated statistics for the YASS solver (using version 2.66).
It would be a time consuming process to manually merge the new statistics into the existing format, and in the end, one would probably still wonder if the updates are indeed correct.

So, I am going to just post the plain text format.
If you like to see the previous colorful table format showing which of the 90 levels were solved or failed, feel free to make the change.

-- Ming

</div>

Sokoban Wiki - User contributions [en]

Solver Statistics - Aymeric du Peloux

Talk:Solver Statistics - XSokoban - Thinking Rabbit & Various Authors