Skip to content

Commit 8d90847

Browse files
kornykorny
committed
Merge branch 'master' into cleanup-output
2 parents bf395bd + 0a1f500 commit 8d90847

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

71 files changed

+216
-3150
lines changed

‎.travis.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ rvm:
22
- 1.8.7
33
- ree
44
- 1.9.3
5-
- 2.0.0
5+
- 2.0
6+
- 2.1
7+
- 2.2
8+
- 2.3.0
69
- ruby-head
710
- jruby-18mode
811
- jruby-19mode
@@ -19,3 +22,4 @@ matrix:
1922
- rvm: rbx-18mode
2023
- rvm: rbx-19mode
2124
script: "rake test" # test:scanners"
25+
sudo: false

‎Changes.textile

Lines changed: 69 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2,35 +2,80 @@ h1=. CodeRay Version History
22

33
p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
44

5+
h2. Changes in 1.1.1
6+
7+
* SQL scanner: Allow @$@ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
8+
* SQL scanner: Fix open strings [#163, thanks to Adam]
9+
* Ruby scanner: Accept number literal suffixes @r@ and @i@ (Ruby 2.1)
10+
* Ruby scanner: Accept quoted hash keys like @{ "a": boss }@ (Ruby 2.2)
11+
* Ruby scanner: Accept save navigation operator @&.@ (Ruby 2.3)
12+
* Ruby scanner: Accept squiggly heredoc @<<~@ (Ruby 2.3)
13+
* Diff scanner: Prevent running out of regexp stack.
14+
* HTML encoder: You can keep tabs intact now by setting @tab_width: false@.
15+
516
h2. Changes in 1.1
617

7-
* New scanner: Lua [#21, #22, thanks to Quintus]
8-
* New scanner: Sass [#93]
9-
* New scanner: Go [#28, thanks to Eric Guo and Nathan Youngman]
10-
* New scanner: Taskpaper [#39, thanks to shimomura]
18+
New scanners:
19+
20+
* Go [#28, thanks to Eric Guo and Nathan Youngman]
21+
* Lua [#21, #22, thanks to Quintus]
22+
* Sass [#93]
23+
* Taskpaper [#39, thanks to shimomura]
24+
25+
More new stuff:
26+
27+
* @.xaml@ file type [#121, thanks to Kozman Bálint]
28+
* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
29+
* new token kind @:id@ for CSS/Sass [#27]
30+
* new token kind @:done@ for Taskpaper [#39]
31+
* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
32+
* new token kind @:unknown@ for Debug scanner
33+
* new DebugLint encoder that checks for empty tokens and correct nesting
34+
35+
Improvements:
36+
37+
* CSS scanner uses @:id@ and @:tag@ now [#27]
1138
* Diff scanner: Highlight inline changes in multi-line changes [#99]
1239
* JavaScript scanner: Highlight multi-line comments in diff correctly
13-
* Ruby scanner: Accept keywords as Ruby 1.9 hash keys [#126]
40+
* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
1441
* HTML scanner displays style tags and attributes now [#145]
15-
* Remove double-click toggle handler from HTML table output
16-
* Fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
17-
* Fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
18-
* Added @:keep_state@ functionality to more scanners [#116]
19-
* CSS scanner uses @:id@ and @:tag@ now [#27]
20-
* Removed @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency. Nobody was using this, right?
21-
* Add .xaml file type [#121, thanks to Kozman Bálint]
22-
* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
23-
* New token type @:id@ for CSS/Sass [#27]
24-
* New token type @:done@ for Taskpaper [#39]
25-
* New token type @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and nathany]
26-
* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
27-
* Override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
28-
* Fixed @:docstring@ token type style
29-
* @Plugin@ does not warn about fallback when default is defined
30-
* @HTML@ encoder will not warn about unclosed token groups at the end of the stream
31-
* @Debug@ encoder refactored; use @DebugLint@ if you want strict checking now
32-
* @Debug@ encoder will not warn about errors in the token stream
33-
* New @DebugLint@ encoder that checks for empty tokens and correct nesting
42+
* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
43+
* Ruby scanner: Accept keywords as Ruby hash keys [#126]
44+
* performance improvements to several scanners and encoders, especially Terminal and HTML
45+
* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
46+
* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
47+
* updated and cleaned up the documentation,
48+
* documented list of TokenKinds
49+
* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
50+
* @rake generate@ supports Git now instead of Subversion
51+
52+
Removed:
53+
54+
* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
55+
* double-click toggle handler from HTML table output
56+
* @rake_helpers@, @sample@ directories and several other ancient garbage
57+
58+
Fixes:
59+
60+
* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
61+
* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
62+
* fixed @:docstring@ token type style
63+
* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
64+
* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
65+
* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
66+
* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
67+
* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
68+
* HTML encoder will not warn about unclosed token groups at the end of the stream
69+
* fixed problem with coderay/version.rb being loaded twice
70+
71+
Internals:
72+
73+
* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
74+
* The Raydebug scanner highlights unknown token kinds as @:plain@.
75+
* The Debug encoder refactored; use DebugLint if you want strict checking now..
76+
* The Debug encoder will not warn about errors in the token stream.
77+
* Plugin does not warn about fallback when default is defined.
78+
* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
3479

3580
h2. Changes in 1.0.9
3681

@@ -440,6 +485,3 @@ The helper classes were cleaned up; see above for details.
440485

441486
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
442487
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
443-
444-
445-

‎Gemfile

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ gemspec
66
# Add dependencies to develop your gem here.
77
# Include everything needed to run rake, tests, features, etc.
88
group :development do
9-
gem "bundler", ">= 1.0.0"
9+
gem "bundler"
1010
gem "rake"
1111
gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
12-
gem "term-ansicolor", '~> 1.2.2'
13-
gem "shoulda-context", "~> 1.1.2"
12+
gem "term-ansicolor"
13+
gem 'tins', '~> 1.6.0'
14+
gem "shoulda-context"
15+
gem "test-unit"
1416
gem "json" if RUBY_VERSION < '1.9'
1517
gem "rdoc"
1618
end

‎README.markdown

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ You put your code in, and you get it back colored; Keywords, strings, floats, co
1616

1717
### Dependencies
1818

19-
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0. It also runs on JRuby.
19+
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0+. It also runs on JRuby.
2020

2121
## Example Usage
2222

@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
2828

2929
## Documentation
3030

31-
See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
31+
See [rubydoc](http://rubydoc.info/gems/coderay).

‎Rakefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,4 +34,4 @@ else
3434
rd.rdoc_dir = 'doc'
3535
end
3636

37-
end
37+
end

‎bin/coderay

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ defaults:
3535
3636
common:
3737
coderay file.rb # highlight file to terminal
38-
coderay file.rb > file.html # highlight file to HTML page
38+
coderay file.rb -page > file.html # highlight file to HTML page
3939
coderay file.rb -div > file.html # highlight file to HTML snippet
4040
4141
configure output:

‎lib/coderay/encoders/debug_lint.rb

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@ class DebugLint < Debug
1818
register_for :debug_lint
1919

2020
def text_token text, kind
21-
raise Lint::EmptyToken, 'empty token' if text.empty?
21+
raise Lint::EmptyToken, 'empty token for %p' % [kind] if text.empty?
22+
raise Lint::UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
2223
super
2324
end
2425

‎lib/coderay/encoders/html.rb

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ module Encoders
2525
# == Options
2626
#
2727
# === :tab_width
28-
# Convert \t characters to +n+ spaces (a number.)
28+
# Convert \t characters to +n+ spaces (a number or false.)
29+
# false will keep tab characters untouched.
2930
#
3031
# Default: 8
3132
#
@@ -180,7 +181,7 @@ def setup options
180181

181182
@break_lines = (options[:break_lines] == true)
182183

183-
@HTML_ESCAPE = HTML_ESCAPE.merge("\t" => ' ' * options[:tab_width])
184+
@HTML_ESCAPE = HTML_ESCAPE.merge("\t" => options[:tab_width] ? ' ' * options[:tab_width] : "\t")
184185

185186
@opened = []
186187
@last_opened = nil
@@ -197,11 +198,13 @@ def finish options
197198
@last_opened = nil
198199
end
199200

200-
if options[:line_numbers]
201-
Numbering.number! @out, options[:line_numbers], options
201+
if @out.respond_to? :to_str
202+
if options[:line_numbers]
203+
Numbering.number! @out, options[:line_numbers], options
204+
end
205+
@out = Output.wrap_string_in @out, options[:wrap], @css if options[:wrap]
206+
@out = @out.sub(/(<title>)(<\/title>)/) { $1 + options[:title] + $2 } if options[:title]
202207
end
203-
@out = Output.wrap_string_in @out, options[:wrap], @css if options[:wrap]
204-
@out = @out.sub(/(<title>)(<\/title>)/) { $1 + options[:title] + $2 } if options[:title]
205208

206209
if defined?(@real_out) && @real_out
207210
@real_out << @out
@@ -287,7 +290,7 @@ def style_for_kinds kinds
287290

288291
def make_span_for_kinds method, hint
289292
Hash.new do |h, kinds|
290-
h[kinds.is_a?(Symbol) ? kinds : kinds.dup] = begin
293+
begin
291294
css_class = css_class_for_kinds(kinds)
292295
title = HTML.token_path_to_hint hint, kinds if hint
293296

@@ -299,6 +302,9 @@ def make_span_for_kinds method, hint
299302
"<span#{title}#{" class=\"#{css_class}\"" if css_class}>"
300303
end
301304
end
305+
end.tap do |span|
306+
h.clear if h.size >= 100
307+
h[kinds] = span
302308
end
303309
end
304310
end
@@ -311,8 +317,8 @@ def check_group_nesting name, kind
311317

312318
def break_lines text, style
313319
reopen = ''
314-
@opened.each_with_index do |k, index|
315-
reopen << (@span_for_kinds[index > 0 ? [k, *@opened[0...index]] : k] || '<span>')
320+
@opened.each_with_index do |kind, index|
321+
reopen << (@span_for_kinds[index > 0 ? [kind, *@opened[0...index]] : kind] || '<span>')
316322
end
317323
text.gsub("\n", "#{'</span>' * @opened.size}#{'</span>' if style}\n#{reopen}#{style}")
318324
end

‎lib/coderay/encoders/lint.rb

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@ class Lint < Debug
1717

1818
InvalidTokenStream = Class.new StandardError
1919
EmptyToken = Class.new InvalidTokenStream
20+
UnknownTokenKind = Class.new InvalidTokenStream
2021
IncorrectTokenGroupNesting = Class.new InvalidTokenStream
2122

2223
def text_token text, kind
23-
raise EmptyToken, 'empty token' if text.empty?
24+
raise EmptyToken, 'empty token for %p' % [kind] if text.empty?
25+
raise UnknownTokenKind, 'unknown token kind %p (text was %p)' % [kind, text] unless TokenKinds.has_key? kind
2426
end
2527

2628
def begin_group kind

‎lib/coderay/helpers/file_type.rb

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ def [] filename, read_shebang = false
3838
(TypeFromExt[ext2.downcase] if ext2) ||
3939
TypeFromName[name] ||
4040
TypeFromName[name.downcase]
41-
type ||= shebang(filename) if read_shebang
41+
type ||= type_from_shebang(filename) if read_shebang
4242

4343
type
4444
end
@@ -63,7 +63,7 @@ def fetch filename, default = nil, read_shebang = false
6363

6464
protected
6565

66-
def shebang filename
66+
def type_from_shebang filename
6767
return unless File.exist? filename
6868
File.open filename, 'r' do |f|
6969
if first_line = f.gets

‎lib/coderay/helpers/plugin.rb

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ module PluginHost
3030
# * a file could not be found
3131
# * the requested Plugin is not registered
3232
PluginNotFound = Class.new LoadError
33-
HostNotFound = Class.new LoadError
33+
HostNotFound = Class.new LoadError
3434

3535
PLUGIN_HOSTS = []
3636
PLUGIN_HOSTS_BY_ID = {} # dummy hash
@@ -49,8 +49,8 @@ def load_all
4949
def [] id, *args, &blk
5050
plugin = validate_id(id)
5151
begin
52-
plugin = plugin_hash.[] plugin, *args, &blk
53-
end while plugin.is_a? Symbol
52+
plugin = plugin_hash.[](plugin, *args, &blk)
53+
end while plugin.is_a? String
5454
plugin
5555
end
5656

@@ -95,7 +95,7 @@ def plugin_path *args
9595
def map hash
9696
for from, to in hash
9797
from = validate_id from
98-
to = validate_id to
98+
to = validate_id to
9999
plugin_hash[from] = to unless plugin_hash.has_key? from
100100
end
101101
end
@@ -197,22 +197,22 @@ def path_to plugin_id
197197
File.join plugin_path, "#{plugin_id}.rb"
198198
end
199199

200-
# Converts +id+ to a Symbol if it is a String,
201-
# or returns +id+ if it already is a Symbol.
200+
# Converts +id+ to a valid plugin ID String, or returns +nil+.
202201
#
203202
# Raises +ArgumentError+ for all other objects, or if the
204203
# given String includes non-alphanumeric characters (\W).
205204
def validate_id id
206-
if id.is_a? Symbol or id.nil?
207-
id
208-
elsif id.is_a? String
205+
case id
206+
when Symbol
207+
id.to_s
208+
when String
209209
if id[/\w+/] == id
210-
id.downcase.to_sym
210+
id.downcase
211211
else
212212
raise ArgumentError, "Invalid id given: #{id}"
213213
end
214214
else
215-
raise ArgumentError, "String or Symbol expected, but #{id.class} given."
215+
raise ArgumentError, "Symbol or String expected, but #{id.class} given."
216216
end
217217
end
218218

‎lib/coderay/scanners/debug.rb

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,23 @@
1+
require 'set'
2+
13
module CodeRay
24
module Scanners
35

46
# = Debug Scanner
57
#
6-
# Interprets the output of the Encoders::Debug encoder.
8+
# Interprets the output of the Encoders::Debug encoder (basically the inverse function).
79
class Debug < Scanner
810

911
register_for :debug
1012
title 'CodeRay Token Dump Import'
1113

1214
protected
1315

16+
def setup
17+
super
18+
@known_token_kinds = TokenKinds.keys.map(&:to_s).to_set
19+
end
20+
1421
def scan_tokens encoder, options
1522

1623
opened_tokens = []
@@ -21,16 +28,19 @@ def scan_tokens encoder, options
2128
encoder.text_token match, :space
2229

2330
elsif match = scan(/ (\w+) \( ( [^\)\\]* ( \\. [^\)\\]* )* ) \)? /x)
24-
kind = self[1].to_sym
25-
match = self[2].gsub(/\\(.)/m, '\1')
26-
unless TokenKinds.has_key? kind
27-
kind = :error
28-
match = matched
31+
if @known_token_kinds.include? self[1]
32+
encoder.text_token self[2].gsub(/\\(.)/m, '\1'), self[1].to_sym
33+
else
34+
encoder.text_token matched, :unknown
2935
end
30-
encoder.text_token match, kind
3136

3237
elsif match = scan(/ (\w+) ([<\[]) /x)
33-
kind = self[1].to_sym
38+
if @known_token_kinds.include? self[1]
39+
kind = self[1].to_sym
40+
else
41+
kind = :unknown
42+
end
43+
3444
opened_tokens << kind
3545
case self[2]
3646
when '<'

0 commit comments

Comments
 (0)