Skip to content

Ruby 1.9 style hashes broken in > 0.9.5 #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sethvargo opened this issue Jan 5, 2013 · 6 comments · Fixed by #114
Closed

Ruby 1.9 style hashes broken in > 0.9.5 #106

sethvargo opened this issue Jan 5, 2013 · 6 comments · Fixed by #114
Assignees
Milestone

Comments

@sethvargo
Copy link

In my testing, the Ruby 1.9 hash style syntax was supported in 0.9.5, but has not worked since then. Is there any way to bring back that syntax in 1.x?

@sethvargo
Copy link
Author

Here's a diff of 0.9.5 and 0.9.6. It worked in 0.9.5, and was broken in 0.9.6 and up.

diff --git acoderay-0.9.5/lib/README bcoderay-0.9.6/lib/README
index 38f89bb..e08ca59 100644
--- acoderay-0.9.5/lib/README
+++ bcoderay-0.9.6/lib/README
@@ -18,7 +18,7 @@ And with line numbers.
 * is what everybody should have on their website
 * solves all your problems and makes the girls run after you

-Version: 0.9.5
+Version: 0.9.6
 Author:: murphy (Kornelius Kalnbach)
 Contact:: murphy rubychan de
 Website:: coderay.rubychan.de[http://coderay.rubychan.de]
diff --git acoderay-0.9.5/lib/coderay/.DS_Store bcoderay-0.9.5/lib/coderay/.DS_Store
deleted file mode 100644
index fb583d8..0000000
Binary files acoderay-0.9.5/lib/coderay/.DS_Store and /dev/null differ
diff --git acoderay-0.9.5/lib/coderay/scanner.rb bcoderay-0.9.6/lib/coderay/scanner.rb
index 74079a8..b12c865 100644
--- acoderay-0.9.5/lib/coderay/scanner.rb
+++ bcoderay-0.9.6/lib/coderay/scanner.rb
@@ -69,6 +69,7 @@ module CodeRay
         def normify code
           code = code.to_s
           if code.respond_to?(:encoding) && (code.encoding.name != 'UTF-8' || !code.valid_encoding?)
+            code = code.dup
             original_encoding = code.encoding
             code.force_encoding 'Windows-1252'
             unless code.valid_encoding?
diff --git acoderay-0.9.5/lib/coderay/scanners/ruby/patterns.rb bcoderay-0.9.6/lib/coderay/scanners/ruby/patterns.rb
index 69ae012..ba3ac8a 100644
--- acoderay-0.9.5/lib/coderay/scanners/ruby/patterns.rb
+++ bcoderay-0.9.6/lib/coderay/scanners/ruby/patterns.rb
@@ -16,7 +16,7 @@ module Scanners
     DEF_KEYWORDS = %w[ def ]
     UNDEF_KEYWORDS = %w[ undef ]
     ALIAS_KEYWORDS = %w[ alias ]
-    MODULE_KEYWORDS = %w[class module]
+    MODULE_KEYWORDS = %w[ class module ]
     DEF_NEW_STATE = WordList.new(:initial).
       add(DEF_KEYWORDS, :def_expected).
       add(UNDEF_KEYWORDS, :undef_expected).
@@ -25,7 +25,8 @@ module Scanners

     PREDEFINED_CONSTANTS = %w[
       nil true false self
-      DATA ARGV ARGF __FILE__ __LINE__
+      DATA ARGV ARGF
+      __FILE__ __LINE__ __ENCODING__
     ]

     IDENT_KIND = WordList.new(:ident).
diff --git acoderay-0.9.5/lib/coderay/scanners/ruby.rb bcoderay-0.9.6/lib/coderay/scanners/ruby.rb
index cef8d83..3cadc64 100644
--- acoderay-0.9.5/lib/coderay/scanners/ruby.rb
+++ bcoderay-0.9.6/lib/coderay/scanners/ruby.rb
@@ -181,17 +181,20 @@ module Scanners
               if last_token_dot
                 kind = if match[/^[A-Z]/] and not match?(/\(/) then :constant else :ident end
               else
-                kind = patterns::IDENT_KIND[match]
-                if kind == :ident
-                  if match[/^[A-Z]/] and not match[/[!?]$/] and not match?(/\(/)
-                    kind = :constant
-                  elsif scan(/:(?= )/)
-                    match << ':'
-                    kind = :symbol
+                if value_expected != :expect_colon && scan(/:(?= )/)
+                  tokens << [match, :key]
+                  match = ':'
+                  kind = :operator
+                else
+                  kind = patterns::IDENT_KIND[match]
+                  if kind == :ident
+                    if match[/\A[A-Z]/] and not match[/[!?]$/] and not match?(/\(/)
+                      kind = :constant
+                    end
+                  elsif kind == :reserved
+                    state = patterns::DEF_NEW_STATE[match]
+                    value_expected = :set if patterns::KEYWORDS_EXPECTING_VALUE[match]
                   end
-                elsif kind == :reserved
-                  state = patterns::DEF_NEW_STATE[match]
-                  value_expected = :set if patterns::KEYWORDS_EXPECTING_VALUE[match]
                 end
               end
               value_expected = :set if check(/#{patterns::VALUE_FOLLOWS}/o)
@@ -401,7 +404,9 @@ module Scanners
 # }}}

           unless kind == :error
-            value_expected = value_expected == :set
+            if value_expected = value_expected == :set
+              value_expected = :expect_colon if match == '?' || match == 'when'
+            end
             last_token_dot = last_token_dot == :set
           end

diff --git acoderay-0.9.5/lib/coderay.rb bcoderay-0.9.6/lib/coderay.rb
index f54e463..b439bf9 100644
--- acoderay-0.9.5/lib/coderay.rb
+++ bcoderay-0.9.6/lib/coderay.rb
@@ -134,7 +134,7 @@ module CodeRay
   # Minor: feature milestone
   # Teeny: development state, 0 for pre-release
   # Revision: Subversion Revision number (generated on rake gem:make)
-  VERSION = '0.9.5'
+  VERSION = '0.9.6'

   require 'coderay/tokens'
   require 'coderay/token_classes'
diff --git acoderay-0.9.5/test/functional/basic.rb bcoderay-0.9.6/test/functional/basic.rb
index 150089e..8ecd3d3 100755
--- acoderay-0.9.5/test/functional/basic.rb
+++ bcoderay-0.9.6/test/functional/basic.rb
@@ -115,4 +115,8 @@ more code  # and another comment, in-line.
     assert CodeRay::Scanners.list.include?('plaintext')
   end

+  def test_scan_a_frozen_string
+    CodeRay.scan RUBY_VERSION, :ruby
+  end
+  
 end

@ghost ghost assigned korny Jan 6, 2013
@korny
Copy link
Member

korny commented Jan 6, 2013

Wow, this is unfortunate. I thought it was working, strange that it didn't come up until now. Thank you for reporting!

I guess the feature was lost in the transition to 1.0 somehow. It relies on an older variant of the value_expected flag, so it's not easy to put back in to the new Ruby scanner. I will think of something.

In the meantime, please take a look at this test case I prepared:

new = { hash: :syntax }
call(with, hash: :syntax)
this_crazy(hash: :syntax)
test \
  hash: :syntax,
  multi: :line

Can you think of more examples that should be supported?

@sethvargo
Copy link
Author

It broke in 0.9.6 actually.

And I would test the case without a space in between (foo:"bar" and foo:bar)

Thanks for looking into this.

Seth

@korny
Copy link
Member

korny commented Jan 6, 2013

It broke in 0.9.6 actually.

Actually, it only changed: 0.9.6 introduced the :key token type and used it for the hash keys, so it should be a
different color:

screenshot_01

Maybe your stylesheet doesn't include a color for this class, but it's there in 0.9.6+.

1.0.0 fails to highlight it as anything meaningful and just falls back to :ident.

@korny
Copy link
Member

korny commented Jan 6, 2013

Just for reference, the updated test:

new = { hash: :syntax }
call(with, hash: :syntax)
this_crazy(hash: :syntax)
test \
  hash: :syntax,
  multi: :line

more hash::tests
more hash:"tests"
more hash:42
more hash:??
more hash:{}
more hash:nil
more hash:foo
more hash://
_ _:_

@sethvargo
Copy link
Author

Anything I can do to help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants