Skip to content

Commit ff47c7a

Browse files
committed
Merge branch 'master' into tweak-function-colors
2 parents 31c252a + 0a1f500 commit ff47c7a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+548
-3312
lines changed

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,4 @@ Gemfile.lock
1111
test/executable/source.rb.html
1212
test/executable/source.rb.json
1313
test/scanners
14-
bench/test.div.html
1514
old-stuff

.travis.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,10 @@ rvm:
22
- 1.8.7
33
- ree
44
- 1.9.3
5-
- 2.0.0
5+
- 2.0
6+
- 2.1
7+
- 2.2
8+
- 2.3.0
69
- ruby-head
710
- jruby-18mode
811
- jruby-19mode
@@ -19,3 +22,4 @@ matrix:
1922
- rvm: rbx-18mode
2023
- rvm: rbx-19mode
2124
script: "rake test" # test:scanners"
25+
sudo: false

Changes.textile

Lines changed: 69 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -2,34 +2,80 @@ h1=. CodeRay Version History
22

33
p=. _This files lists all changes in the CodeRay library since the 0.9.8 release._
44

5+
h2. Changes in 1.1.1
6+
7+
* SQL scanner: Allow @$@ signs in SQL identifiers [#164, thanks to jasir and Ben Basson]
8+
* SQL scanner: Fix open strings [#163, thanks to Adam]
9+
* Ruby scanner: Accept number literal suffixes @r@ and @i@ (Ruby 2.1)
10+
* Ruby scanner: Accept quoted hash keys like @{ "a": boss }@ (Ruby 2.2)
11+
* Ruby scanner: Accept save navigation operator @&.@ (Ruby 2.3)
12+
* Ruby scanner: Accept squiggly heredoc @<<~@ (Ruby 2.3)
13+
* Diff scanner: Prevent running out of regexp stack.
14+
* HTML encoder: You can keep tabs intact now by setting @tab_width: false@.
15+
516
h2. Changes in 1.1
617

7-
* New scanner: Lua [#21, #22, thanks to Quintus]
8-
* New scanner: Sass [#93]
9-
* New scanner: Taskpaper [#39, thanks to shimomura]
18+
New scanners:
19+
20+
* Go [#28, thanks to Eric Guo and Nathan Youngman]
21+
* Lua [#21, #22, thanks to Quintus]
22+
* Sass [#93]
23+
* Taskpaper [#39, thanks to shimomura]
24+
25+
More new stuff:
26+
27+
* @.xaml@ file type [#121, thanks to Kozman Bálint]
28+
* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
29+
* new token kind @:id@ for CSS/Sass [#27]
30+
* new token kind @:done@ for Taskpaper [#39]
31+
* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
32+
* new token kind @:unknown@ for Debug scanner
33+
* new DebugLint encoder that checks for empty tokens and correct nesting
34+
35+
Improvements:
36+
37+
* CSS scanner uses @:id@ and @:tag@ now [#27]
1038
* Diff scanner: Highlight inline changes in multi-line changes [#99]
1139
* JavaScript scanner: Highlight multi-line comments in diff correctly
12-
* Ruby scanner: Accept keywords as Ruby 1.9 hash keys [#126]
40+
* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
1341
* HTML scanner displays style tags and attributes now [#145]
14-
* Remove double-click toggle handler from HTML table output
15-
* Fixes to CSS scanner (floats, pseudoclasses)
16-
* Fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
17-
* Added @:keep_state@ functionality to more scanners [#116]
18-
* CSS scanner uses @:id@ and @:tag@ now [#27]
19-
* Removed @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency. Nobody was using this, right?
20-
* Add .xaml file type [#121, thanks to Kozman Bálint]
21-
* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
22-
* New token type @:id@ for CSS/Sass [#27]
23-
* New token type @:done@ for Taskpaper [#39]
24-
* New token type @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and nathany]
25-
* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
26-
* Override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
27-
* Fixed @:docstring@ token type style
28-
* @Plugin@ does not warn about fallback when default is defined
29-
* @HTML@ encoder will not warn about unclosed token groups at the end of the stream
30-
* @Debug@ encoder refactored; use @DebugLint@ if you want strict checking now
31-
* @Debug@ encoder will not warn about errors in the token stream
32-
* New @DebugLint@ encoder that checks for empty tokens and correct nesting
42+
* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
43+
* Ruby scanner: Accept keywords as Ruby hash keys [#126]
44+
* performance improvements to several scanners and encoders, especially Terminal and HTML
45+
* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
46+
* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
47+
* updated and cleaned up the documentation,
48+
* documented list of TokenKinds
49+
* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
50+
* @rake generate@ supports Git now instead of Subversion
51+
52+
Removed:
53+
54+
* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
55+
* double-click toggle handler from HTML table output
56+
* @rake_helpers@, @sample@ directories and several other ancient garbage
57+
58+
Fixes:
59+
60+
* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
61+
* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
62+
* fixed @:docstring@ token type style
63+
* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
64+
* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
65+
* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
66+
* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
67+
* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
68+
* HTML encoder will not warn about unclosed token groups at the end of the stream
69+
* fixed problem with coderay/version.rb being loaded twice
70+
71+
Internals:
72+
73+
* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
74+
* The Raydebug scanner highlights unknown token kinds as @:plain@.
75+
* The Debug encoder refactored; use DebugLint if you want strict checking now..
76+
* The Debug encoder will not warn about errors in the token stream.
77+
* Plugin does not warn about fallback when default is defined.
78+
* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
3379

3480
h2. Changes in 1.0.9
3581

@@ -439,6 +485,3 @@ The helper classes were cleaned up; see above for details.
439485

440486
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
441487
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
442-
443-
444-

Gemfile

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,13 @@ gemspec
66
# Add dependencies to develop your gem here.
77
# Include everything needed to run rake, tests, features, etc.
88
group :development do
9-
gem "bundler", ">= 1.0.0"
9+
gem "bundler"
1010
gem "rake"
1111
gem "RedCloth", RUBY_PLATFORM == 'java' ? ">= 4.2.7" : ">= 4.0.3"
12-
gem "term-ansicolor", '~> 1.2.2'
13-
gem "shoulda-context", "~> 1.1.2"
12+
gem "term-ansicolor"
13+
gem 'tins', '~> 1.6.0'
14+
gem "shoulda-context"
15+
gem "test-unit"
1416
gem "json" if RUBY_VERSION < '1.9'
1517
gem "rdoc"
1618
end

README.markdown

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ You put your code in, and you get it back colored; Keywords, strings, floats, co
1616

1717
### Dependencies
1818

19-
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0. It also runs on JRuby.
19+
CodeRay needs Ruby 1.8.7, 1.9.3 or 2.0+. It also runs on JRuby.
2020

2121
## Example Usage
2222

@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
2828

2929
## Documentation
3030

31-
See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
31+
See [rubydoc](http://rubydoc.info/gems/coderay).

Rakefile

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
require 'bundler/gem_tasks'
2+
13
$:.unshift File.dirname(__FILE__) unless $:.include? '.'
24

35
ROOT = '.'
@@ -32,4 +34,4 @@ else
3234
rd.rdoc_dir = 'doc'
3335
end
3436

35-
end
37+
end

bench/bench.rb

Lines changed: 31 additions & 127 deletions
Original file line numberDiff line numberDiff line change
@@ -1,142 +1,46 @@
1-
# The most ugly test script I've ever written!
2-
# Shame on me!
3-
4-
require 'pathname'
5-
require 'profile' if ARGV.include? '-p'
6-
7-
MYDIR = File.dirname(__FILE__)
8-
LIBDIR = Pathname.new(MYDIR).join('..', 'lib').cleanpath.to_s
9-
$:.unshift MYDIR, LIBDIR
1+
require 'benchmark'
2+
$: << File.expand_path('../../lib', __FILE__)
103
require 'coderay'
114

12-
@size = ARGV.fetch(2, 100).to_i * 1000
13-
14-
lang = ARGV.fetch(0) do
15-
puts <<-HELP
16-
Usage:
17-
ruby bench.rb (c|ruby) (null|text|tokens|count|statistic|yaml|html) [size in kB] [stream]
18-
19-
SIZE defaults to 100 kB (= 100,000 bytes).
20-
SIZE = 0 means the whole input.
21-
22-
-p generates a profile (slow! use with SIZE = 1)
23-
-o shows the output
24-
stream enabled streaming mode
25-
26-
Sorry for the strange interface. I will improve it in the next release.
27-
HELP
5+
if ARGV.include? '-h'
6+
puts DATA.read
287
exit
298
end
309

31-
format = ARGV.fetch(1, 'html').downcase
32-
33-
$stream = ARGV.include? 'stream'
34-
$optimize = ARGV.include? 'opt'
35-
$style = ARGV.include? 'style'
36-
37-
require 'benchmark'
38-
require 'fileutils'
10+
lang = ARGV.fetch(0, 'ruby')
11+
data = nil
12+
File.open(File.expand_path("../example.#{lang}", __FILE__), 'rb') { |f| data = f.read }
13+
raise 'Example file is empty.' if data.empty?
3914

40-
if format == 'comp'
41-
format = 'page'
42-
begin
43-
require 'syntax'
44-
require 'syntax/convertors/html.rb'
45-
rescue LoadError
46-
puts 'Syntax no found!! (Try % gem install syntax)'
47-
end
48-
end
15+
format = ARGV.fetch(1, 'html').downcase
16+
encoder = CodeRay.encoder(format)
4917

50-
def here fn = nil
51-
return MYDIR unless fn
52-
File.join here, fn
18+
size = ARGV.fetch(2, 1000).to_i * 1000
19+
unless size.zero?
20+
data += data until data.size >= size
21+
data = data[0, size]
5322
end
23+
size = data.size
24+
puts "encoding %d kB of #{lang} code to #{format}..." % [(size / 1000.0).round]
5425

55-
n = ARGV.find { |a| a[/^N/] }
56-
N = if n then n[/\d+/].to_i else 1 end
57-
$filename = ARGV.include?('strange') ? 'strange' : 'example'
58-
59-
Benchmark.bm(20) do |bm|
60-
N.times do
61-
62-
data = nil
63-
File.open(here("#$filename." + lang), 'rb') { |f| data = f.read }
64-
raise 'Example file is empty.' if data.empty?
65-
unless @size.zero?
66-
data += data until data.size >= @size
67-
data = data[0, @size]
68-
end
69-
@size = data.size
70-
71-
options = {
72-
:tab_width => 2,
73-
# :line_numbers => :inline,
74-
:css => $style ? :style : :class,
75-
}
76-
$hl = CodeRay.encoder(format, options)
77-
time = bm.report('CodeRay') do
78-
if $stream || true
79-
$o = $hl.encode(data, lang, options)
80-
else
81-
tokens = CodeRay.scan(data, lang)
82-
tokens.optimize! if $optimize
83-
$o = tokens.encode($hl)
84-
end
85-
end
86-
$file_created = here('test.' + $hl.file_extension)
87-
File.open($file_created, 'wb') do |f|
88-
# f.write $o
89-
end
90-
91-
time_real = time.real
92-
93-
puts "\t%7.2f KB/s (%d.%d KB)" % [((@size / 1000.0) / time_real), @size / 1000, @size % 1000]
94-
puts $o if ARGV.include? '-o'
95-
96-
end
26+
n = ARGV.fetch(3, 5).to_s[/\d+/].to_i
27+
require 'profile' if ARGV.include? '-p'
28+
n.times do |i|
29+
time = Benchmark.realtime { encoder.encode(data, lang) }
30+
puts "run %d: %5.2f s, %4.0f kB/s" % [i + 1, time, size / time / 1000.0]
9731
end
98-
puts "Files created: #$file_created"
9932

100-
STDIN.gets if ARGV.include? 'wait'
33+
STDIN.gets if ARGV.include? '-w'
10134

10235
__END__
103-
.ruby .normal {}
104-
.ruby .comment { color: #005; font-style: italic; }
105-
.ruby .keyword { color: #A00; font-weight: bold; }
106-
.ruby .method { color: #077; }
107-
.ruby .class { color: #074; }
108-
.ruby .module { color: #050; }
109-
.ruby .punct { color: #447; font-weight: bold; }
110-
.ruby .symbol { color: #099; }
111-
.ruby .string { color: #944; background: #FFE; }
112-
.ruby .char { color: #F07; }
113-
.ruby .ident { color: #004; }
114-
.ruby .constant { color: #07F; }
115-
.ruby .regex { color: #B66; background: #FEF; }
116-
.ruby .number { color: #F99; }
117-
.ruby .attribute { color: #7BB; }
118-
.ruby .global { color: #7FB; }
119-
.ruby .expr { color: #227; }
120-
.ruby .escape { color: #277; }
36+
Usage:
37+
ruby bench.rb [lang] [format] [size in kB] [number of runs]
12138

122-
.xml .normal {}
123-
.xml .namespace { color: #B66; font-weight: bold; }
124-
.xml .tag { color: #F88; }
125-
.xml .comment { color: #005; font-style: italic; }
126-
.xml .punct { color: #447; font-weight: bold; }
127-
.xml .string { color: #944; }
128-
.xml .number { color: #F99; }
129-
.xml .attribute { color: #BB7; }
39+
- lang defaults to ruby.
40+
- format defaults to html.
41+
- size defaults to 1000 kB (= 1,000,000 bytes). 0 uses the whole example input.
42+
- number of runs defaults to 5.
13043

131-
.yaml .normal {}
132-
.yaml .document { font-weight: bold; color: #07F; }
133-
.yaml .type { font-weight: bold; color: #05C; }
134-
.yaml .key { color: #F88; }
135-
.yaml .comment { color: #005; font-style: italic; }
136-
.yaml .punct { color: #447; font-weight: bold; }
137-
.yaml .string { color: #944; }
138-
.yaml .number { color: #F99; }
139-
.yaml .time { color: #F99; }
140-
.yaml .date { color: #F99; }
141-
.yaml .ref { color: #944; }
142-
.yaml .anchor { color: #944; }
44+
-h prints this help
45+
-p generates a profile (slow, use with SIZE = 1)
46+
-w waits after the benchmark (for debugging memory usw)

bin/coderay

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ defaults:
3535
3636
common:
3737
coderay file.rb # highlight file to terminal
38-
coderay file.rb > file.html # highlight file to HTML page
38+
coderay file.rb -page > file.html # highlight file to HTML page
3939
coderay file.rb -div > file.html # highlight file to HTML snippet
4040
4141
configure output:

lib/coderay.rb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ module CodeRay
127127

128128
$CODERAY_DEBUG ||= false
129129

130-
CODERAY_PATH = File.join File.dirname(__FILE__), 'coderay'
130+
CODERAY_PATH = File.expand_path('../coderay', __FILE__)
131131

132132
# Assuming the path is a subpath of lib/coderay/
133133
def self.coderay_path *path

0 commit comments

Comments
 (0)