[Gnash-commit] /srv/bzr/gnash/trunk r10788: Use to_int() for fetching color value passed to MovieClip.lineStyle(). Fixes the UB and still works fine for pollock.swf. Thanks bwy for pointing out.

[Sandro Santilli]

------------------------------------------------------------
revno: 10788
committer: Sandro Santilli <address@hidden>
branch nick: trunk
timestamp: Tue 2009-04-14 10:51:26 +0200
message:
  Use to_int() for fetching color value passed to MovieClip.lineStyle(). Fixes 
the UB and still works fine for pollock.swf. Thanks bwy for pointing out.
modified:
  libcore/MovieClip.cpp

=== modified file 'libcore/MovieClip.cpp'
--- a/libcore/MovieClip.cpp     2009-04-13 09:30:55 +0000
+++ b/libcore/MovieClip.cpp     2009-04-14 08:51:26 +0000
@@ -4516,20 +4516,10 @@
         }
         case 2:
         {
-            boost::uint32_t rgbval = boost::uint32_t(
-                // See pollock.swf:
-                // it gets too many black colors with the clamp;
-                // it sets color to a random number from
-                // 0 to 160000000 (about 10 times more then the max).
-                // I guess the correct behaviour is a different handling
-                // of the overflow. Letting the compiler decide on that
-                // gives better results. None of the existing testcases
-                // change results based on this, which calls to adding
-                // a new test. Probably DrawingApiTest.as would be a good
-                // place for that.
-                fn.arg(1).to_number()
-                //clamp<float>(fn.arg(1).to_number(), 0, 16777216)
-            );
+            // See pollock.swf for eventual regressions.
+            // It sets color to a random number from
+            // 0 to 160000000 (about 10 times more then the max).
+            boost::uint32_t rgbval = fn.arg(1).to_int();
             r = boost::uint8_t((rgbval & 0xFF0000) >> 16);
             g = boost::uint8_t((rgbval & 0x00FF00) >> 8);
             b = boost::uint8_t((rgbval & 0x0000FF) );