Skip to content

Commit 5fdb493

Browse files
kornykorny
committed
Merge branch 'master' into escape-utils
2 parents d79c6ba + 0a1f500 commit 5fdb493

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+126
-3093
lines changed

‎.travis.yml

+5-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ rvm:
22
- 1.8.7
33
- ree
44
- 1.9.3
5-
- 2.0.0
5+
- 2.0
6+
- 2.1
7+
- 2.2
8+
- 2.3.0
69
- ruby-head
710
- jruby-18mode
811
- jruby-19mode
@@ -19,3 +22,4 @@ matrix:
1922
- rvm: rbx-18mode
2023
- rvm: rbx-19mode
2124
script: "rake test" # test:scanners"
25+
sudo: false

‎Changes.textile

+69-34
Original file line numberDiff line numberDiff line change
@@ -2,42 +2,80 @@ h1=. CodeRay Version History
22

33
p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
44

5+
h2. Changes in 1.1.1
6+
7+
* SQL scanner: Allow @$@ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
8+
* SQL scanner: Fix open strings [#163, thanks to Adam]
9+
* Ruby scanner: Accept number literal suffixes @r@ and @i@ (Ruby 2.1)
10+
* Ruby scanner: Accept quoted hash keys like @{ "a": boss }@ (Ruby 2.2)
11+
* Ruby scanner: Accept save navigation operator @&.@ (Ruby 2.3)
12+
* Ruby scanner: Accept squiggly heredoc @<<~@ (Ruby 2.3)
13+
* Diff scanner: Prevent running out of regexp stack.
14+
* HTML encoder: You can keep tabs intact now by setting @tab_width: false@.
15+
516
h2. Changes in 1.1
617

7-
* New scanner: Lua [#21, #22, thanks to Quintus]
8-
* New scanner: Sass [#93]
9-
* New scanner: Go [#28, thanks to Eric Guo and Nathan Youngman]
10-
* New scanner: Taskpaper [#39, thanks to shimomura]
18+
New scanners:
19+
20+
* Go [#28, thanks to Eric Guo and Nathan Youngman]
21+
* Lua [#21, #22, thanks to Quintus]
22+
* Sass [#93]
23+
* Taskpaper [#39, thanks to shimomura]
24+
25+
More new stuff:
26+
27+
* @.xaml@ file type [#121, thanks to Kozman Bálint]
28+
* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
29+
* new token kind @:id@ for CSS/Sass [#27]
30+
* new token kind @:done@ for Taskpaper [#39]
31+
* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
32+
* new token kind @:unknown@ for Debug scanner
33+
* new DebugLint encoder that checks for empty tokens and correct nesting
34+
35+
Improvements:
36+
37+
* CSS scanner uses @:id@ and @:tag@ now [#27]
1138
* Diff scanner: Highlight inline changes in multi-line changes [#99]
1239
* JavaScript scanner: Highlight multi-line comments in diff correctly
13-
* Ruby scanner: Accept %i and %I symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
14-
* Ruby scanner: Accept keywords as Ruby 1.9 hash keys [#126]
40+
* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
1541
* HTML scanner displays style tags and attributes now [#145]
16-
* Remove double-click toggle handler from HTML table output
17-
* Fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
18-
* Fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
19-
* Added @:keep_state@ functionality to more scanners [#116]
20-
* CSS scanner uses @:id@ and @:tag@ now [#27]
21-
* Removed @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency. Nobody was using this, right?
22-
* Add .xaml file type [#121, thanks to Kozman Bálint]
23-
* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
24-
* New token type @:id@ for CSS/Sass [#27]
25-
* New token type @:done@ for Taskpaper [#39]
26-
* New token type @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
27-
* New token type @:unknown@ for Debug scanner
28-
* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
29-
* Override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
30-
* Fixed @:docstring@ token type style
31-
* Fixed several problems related to Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
32-
* @PluginHost@ now works with Strings instead of Symbols internally (to avoid using @#to_sym@)
33-
* The @Debug@ scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input)
34-
* The @Raydebug@ scanner highlights unknown token kinds as @:plain@
35-
* @Plugin@ does not warn about fallback when default is defined
36-
* Fixed @HTML@ encoder when output is a StringIO (eg. when using @-HTML@)
37-
* @HTML@ encoder will not warn about unclosed token groups at the end of the stream
38-
* @Debug@ encoder refactored; use @DebugLint@ if you want strict checking now
39-
* @Debug@ encoder will not warn about errors in the token stream
40-
* New @DebugLint@ encoder that checks for empty tokens and correct nesting
42+
* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
43+
* Ruby scanner: Accept keywords as Ruby hash keys [#126]
44+
* performance improvements to several scanners and encoders, especially Terminal and HTML
45+
* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
46+
* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
47+
* updated and cleaned up the documentation,
48+
* documented list of TokenKinds
49+
* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
50+
* @rake generate@ supports Git now instead of Subversion
51+
52+
Removed:
53+
54+
* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
55+
* double-click toggle handler from HTML table output
56+
* @rake_helpers@, @sample@ directories and several other ancient garbage
57+
58+
Fixes:
59+
60+
* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
61+
* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
62+
* fixed @:docstring@ token type style
63+
* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
64+
* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
65+
* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
66+
* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
67+
* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
68+
* HTML encoder will not warn about unclosed token groups at the end of the stream
69+
* fixed problem with coderay/version.rb being loaded twice
70+
71+
Internals:
72+
73+
* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
74+
* The Raydebug scanner highlights unknown token kinds as @:plain@.
75+
* The Debug encoder refactored; use DebugLint if you want strict checking now..
76+
* The Debug encoder will not warn about errors in the token stream.
77+
* Plugin does not warn about fallback when default is defined.
78+
* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
4179

4280
h2. Changes in 1.0.9
4381

@@ -447,6 +485,3 @@ The helper classes were cleaned up; see above for details.
447485

448486
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
449487
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
450-
451-
452-

‎Gemfile

+5-3
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ gemspec
66
# Add dependencies to develop your gem here.
77
# Include everything needed to run rake, tests, features, etc.
88
group :development do
9-
gem "bundler", ">= 1.0.0"
9+
gem "bundler"
1010
gem "rake"
1111
gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
12-
gem "term-ansicolor", '~> 1.2.2'
13-
gem "shoulda-context", "~> 1.1.2"
12+
gem "term-ansicolor"
13+
gem 'tins', '~> 1.6.0'
14+
gem "shoulda-context"
15+
gem "test-unit"
1416
gem "json" if RUBY_VERSION < '1.9'
1517
gem "rdoc"
1618
end

‎README.markdown

+2-2
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ You put your code in, and you get it back colored; Keywords, strings, floats, co
1616

1717
### Dependencies
1818

19-
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0. It also runs on JRuby.
19+
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0+. It also runs on JRuby.
2020

2121
## Example Usage
2222

@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
2828

2929
## Documentation
3030

31-
See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
31+
See [rubydoc](http://rubydoc.info/gems/coderay).

‎bin/coderay

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ defaults:
3535
3636
common:
3737
coderay file.rb # highlight file to terminal
38-
coderay file.rb > file.html # highlight file to HTML page
38+
coderay file.rb -page > file.html # highlight file to HTML page
3939
coderay file.rb -div > file.html # highlight file to HTML snippet
4040
4141
configure output:

‎lib/coderay/encoders/html.rb

+3-2
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ module Encoders
2626
# == Options
2727
#
2828
# === :tab_width
29-
# Convert \t characters to +n+ spaces (a number.)
29+
# Convert \t characters to +n+ spaces (a number or false.)
30+
# false will keep tab characters untouched.
3031
#
3132
# Default: 8
3233
#
@@ -165,7 +166,7 @@ def setup options
165166

166167
@break_lines = (options[:break_lines] == true)
167168

168-
@expand_tab = ' ' * options[:tab_width]
169+
@expand_tab = options[:tab_width] ? ' ' * options[:tab_width] : "\t"
169170

170171
@opened = []
171172
@last_opened = nil

‎lib/coderay/scanners/diff.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ def scan_tokens encoder, options
100100
next
101101
elsif match = scan(/-/)
102102
deleted_lines_count += 1
103-
if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
103+
if options[:inline_diff] && deleted_lines_count == 1 && (changed_lines_count = 1 + check(/.*(?:\n\-.*)*/).count("\n")) && changed_lines_count <= 100_000 && match?(/(?>.*(?:\n\-.*){#{changed_lines_count - 1}}(?:\n\+.*){#{changed_lines_count}})$(?!\n\+)/)
104104
deleted_lines = Array.new(changed_lines_count) { |i| skip(/\n\-/) if i > 0; scan(/.*/) }
105105
inserted_lines = Array.new(changed_lines_count) { |i| skip(/\n\+/) ; scan(/.*/) }
106106

‎lib/coderay/scanners/go.rb

-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
module CodeRay
22
module Scanners
33

4-
# Scanner for Go, copy from c
54
class Go < Scanner
65

76
register_for :go

‎lib/coderay/scanners/ruby.rb

+13-6
Original file line numberDiff line numberDiff line change
@@ -164,15 +164,19 @@ def scan_tokens encoder, options
164164
end
165165

166166
elsif match = scan(/ ' (?:(?>[^'\\]*) ')? | " (?:(?>[^"\\\#]*) ")? /mx)
167-
encoder.begin_group :string
168167
if match.size == 1
168+
kind = check(self.class::StringState.simple_key_pattern(match)) ? :key : :string
169+
encoder.begin_group kind
169170
encoder.text_token match, :delimiter
170-
state = self.class::StringState.new :string, match == '"', match # important for streaming
171+
state = self.class::StringState.new kind, match == '"', match # important for streaming
171172
else
173+
kind = value_expected == true && scan(/:/) ? :key : :string
174+
encoder.begin_group kind
172175
encoder.text_token match[0,1], :delimiter
173176
encoder.text_token match[1..-2], :content if match.size > 2
174177
encoder.text_token match[-1,1], :delimiter
175-
encoder.end_group :string
178+
encoder.end_group kind
179+
encoder.text_token ':', :operator if kind == :key
176180
value_expected = false
177181
end
178182

@@ -191,11 +195,14 @@ def scan_tokens encoder, options
191195
encoder.text_token match, :error
192196
method_call_expected = false
193197
else
194-
encoder.text_token match, self[1] ? :float : :integer # TODO: send :hex/:octal/:binary
198+
kind = self[1] ? :float : :integer # TODO: send :hex/:octal/:binary
199+
match << 'r' if match !~ /e/i && scan(/r/)
200+
match << 'i' if scan(/i/)
201+
encoder.text_token match, kind
195202
end
196203
value_expected = false
197204

198-
elsif match = scan(/ [-+!~^\/]=? | [:;] | [*|&]{1,2}=? | >>? /x)
205+
elsif match = scan(/ [-+!~^\/]=? | [:;] | &\. | [*|&]{1,2}=? | >>? /x)
199206
value_expected = true
200207
encoder.text_token match, :operator
201208

@@ -208,7 +215,7 @@ def scan_tokens encoder, options
208215
encoder.end_group kind
209216
heredocs ||= [] # create heredocs if empty
210217
heredocs << self.class::StringState.new(kind, quote != "'", delim,
211-
self[1] == '-' ? :indented : :linestart)
218+
self[1] ? :indented : :linestart)
212219
value_expected = false
213220

214221
elsif value_expected && match = scan(/#{patterns::FANCY_STRING_START}/o)

‎lib/coderay/scanners/ruby/patterns.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,7 @@ module Ruby::Patterns # :nodoc: all
114114
# NOTE: This is not completely correct, but
115115
# nobody needs heredoc delimiters ending with \n.
116116
HEREDOC_OPEN = /
117-
<< (-)? # $1 = float
117+
<< ([-~])? # $1 = float
118118
(?:
119119
( [A-Za-z_0-9]+ ) # $2 = delim
120120
|

‎lib/coderay/scanners/ruby/string_state.rb

+8
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,14 @@ class StringState < Struct.new :type, :interpreted, :delim, :heredoc,
3737
end
3838
end
3939

40+
def self.simple_key_pattern delim
41+
if delim == "'"
42+
/ (?> (?: [^\\']+ | \\. )* ) ' : /mx
43+
else
44+
/ (?> (?: [^\\"\#]+ | \\. | \#\$[\\"] | \#\{[^\{\}]+\} | \#(?!\{) )* ) " : /mx
45+
end
46+
end
47+
4048
def initialize kind, interpreted, delim, heredoc = false
4149
if heredoc
4250
pattern = heredoc_pattern delim, interpreted, heredoc == :indented

‎lib/coderay/scanners/sql.rb

+17-25
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,12 @@ class SQL < Scanner
5757

5858
STRING_PREFIXES = /[xnb]|_\w+/i
5959

60+
STRING_CONTENT_PATTERN = {
61+
'"' => / (?: [^\\"] | "" )+ /x,
62+
"'" => / (?: [^\\'] | '' )+ /x,
63+
'`' => / (?: [^\\`] | `` )+ /x,
64+
}
65+
6066
def scan_tokens encoder, options
6167

6268
state = :initial
@@ -90,7 +96,7 @@ def scan_tokens encoder, options
9096
state = :string
9197
encoder.text_token match, :delimiter
9298

93-
elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9]* /x)
99+
elsif match = scan(/ @? [A-Za-z_][A-Za-z_0-9\$]* /x)
94100
encoder.text_token match, name_expected ? :ident : (match[0] == ?@ ? :variable : IDENT_KIND[match])
95101
name_expected = false
96102

@@ -115,40 +121,26 @@ def scan_tokens encoder, options
115121
end
116122

117123
elsif state == :string
118-
if match = scan(/[^\\"'`]+/)
119-
string_content << match
120-
next
124+
if match = scan(STRING_CONTENT_PATTERN[string_type])
125+
encoder.text_token match, :content
121126
elsif match = scan(/["'`]/)
122127
if string_type == match
123128
if peek(1) == string_type # doubling means escape
124-
string_content << string_type << getch
125-
next
126-
end
127-
unless string_content.empty?
128-
encoder.text_token string_content, :content
129-
string_content = ''
129+
encoder.text_token match + getch, :content
130+
else
131+
encoder.text_token match, :delimiter
132+
encoder.end_group :string
133+
state = :initial
134+
string_type = nil
130135
end
131-
encoder.text_token match, :delimiter
132-
encoder.end_group :string
133-
state = :initial
134-
string_type = nil
135136
else
136-
string_content << match
137+
encoder.text_token match, :content
137138
end
138139
elsif match = scan(/ \\ (?: #{ESCAPE} | #{UNICODE_ESCAPE} ) /mox)
139-
unless string_content.empty?
140-
encoder.text_token string_content, :content
141-
string_content = ''
142-
end
143140
encoder.text_token match, :char
144141
elsif match = scan(/ \\ . /mox)
145-
string_content << match
146-
next
142+
encoder.text_token match, :content
147143
elsif match = scan(/ \\ | $ /x)
148-
unless string_content.empty?
149-
encoder.text_token string_content, :content
150-
string_content = ''
151-
end
152144
encoder.text_token match, :error unless match.empty?
153145
encoder.end_group :string
154146
state = :initial

‎lib/coderay/version.rb

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
module CodeRay
2-
VERSION = '1.1.0'
2+
VERSION = '1.1.1'
33
end

‎rake_tasks/test.rake

-6
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,4 @@
11
namespace :test do
2-
desc 'run all sample tests'
3-
task :samples do
4-
ruby './test/samples/suite.rb'
5-
end
6-
72
desc 'run functional tests'
83
task :functional do
94
ruby './test/functional/suite.rb'
@@ -85,4 +80,3 @@ Please rename or remove it and run again to use the GitHub repository:
8580
end
8681

8782
task :test => %w(test:functional test:units test:exe)
88-
task :samples => 'test:samples'

‎test/samples/README

-1
This file was deleted.

‎test/samples/cache.actual

-2
This file was deleted.

‎test/samples/cache.expected

-2
This file was deleted.

0 commit comments

Comments
 (0)