Skip to content

Commit ee76cc5

Browse files
kornykorny
committed
Merge branch 'master' into bash-scanner
Conflicts: lib/coderay/helpers/file_type.rb
2 parents 4131a5f + e93aae8 commit ee76cc5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

100 files changed

+1497
-6002
lines changed

‎.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,4 @@ Gemfile.lock
1111
test/executable/source.rb.html
1212
test/executable/source.rb.json
1313
test/scanners
14-
bench/test.div.html
1514
old-stuff

‎.travis.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ branches:
1414
- master
1515
matrix:
1616
allow_failures:
17+
- rvm: ruby-head
18+
- rvm: jruby-head
1719
- rvm: rbx-18mode
1820
- rvm: rbx-19mode
1921
script: "rake test" # test:scanners"

‎Changes.textile

Lines changed: 60 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,67 @@ p=. _This files lists all changes in the CodeRay library since the 0.9.8 release
44

55
h2. Changes in 1.1
66

7-
* New scanner: Sass [#93]
7+
New scanners:
8+
9+
* Go [#28, thanks to Eric Guo and Nathan Youngman]
10+
* Lua [#21, #22, thanks to Quintus]
11+
* Sass [#93]
12+
* Taskpaper [#39, thanks to shimomura]
13+
14+
More new stuff:
15+
16+
* @.xaml@ file type [#121, thanks to Kozman Bálint]
17+
* recognize @Guardfile@, @Vagrantfile@, and @Appraisals@ as Ruby files [#121, thanks to Kozman Bálint]
18+
* new token kind @:id@ for CSS/Sass [#27]
19+
* new token kind @:done@ for Taskpaper [#39]
20+
* new token kind @:map@ for Lua, introducing a nice nested-shades trick [#22, thanks to Quintus and Nathan Youngman]
21+
* new token kind @:unknown@ for Debug scanner
22+
* new DebugLint encoder that checks for empty tokens and correct nesting
23+
24+
Improvements:
25+
26+
* CSS scanner uses @:id@ and @:tag@ now [#27]
827
* Diff scanner: Highlight inline changes in multi-line changes [#99]
9-
* Remove double-click toggle handler from HTML table output
10-
* Fixes to CSS scanner (floats, pseudoclasses)
11-
* Plugin does not warn about fallback when default is defined
12-
* Display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
13-
* Add .xaml file type [#121, thanks to Kozman Bálint]
14-
* @CodeRay::TokenKinds@ should not be frozen [#130, thanks to Gavin Kistner]
28+
* JavaScript scanner: Highlight multi-line comments in diff correctly
29+
* JSON scanner: simplify key/value heuristic, using look-ahead instead of a stack
30+
* HTML scanner displays style tags and attributes now [#145]
31+
* Ruby scanner: Accept @%i(…)@ and @%I(…)@ symbol lists (Ruby 2.0) [thanks to Nathan Youngman]
32+
* Ruby scanner: Accept keywords as Ruby hash keys [#126]
33+
* performance improvements to several scanners and encoders, especially Terminal and HTML
34+
* added @:keep_state@ functionality to more scanners so they work nicely with diff now [#116]
35+
* refactoring and cleanup to achieve better "Code Climate" ratings (but I don't really care)
36+
* updated and cleaned up the documentation,
37+
* documented list of TokenKinds
38+
* Alpha style: tweaked colors for @.binary@, @.local-variable@, and @.predefined-type@
39+
* @rake generate@ supports Git now instead of Subversion
40+
41+
Removed:
42+
43+
* @Tokens#dump@, @Tokens.load@, @Tokens::Undumping@, and @zlib@ dependency
44+
* double-click toggle handler from HTML table output
45+
* @rake_helpers@, @sample@ directories and several other ancient garbage
46+
47+
Fixes:
48+
49+
* fixes to CSS scanner (floats, pseudoclasses, nth-child) [#143]
50+
* fixed empty tokens and unclosed token groups in HTML, CSS, Diff, Goovy, PHP, Raydebug, Ruby, SQL, and YAML scanners [#144]
51+
* fixed @:docstring@ token type style
52+
* fixed several infinite Hash caches and dynamic Symbol creation that might have been exploited by an attacker [#148]
53+
* fixed HTML encoder when output is a StringIO (eg. when using @-HTML@ as a command line parameter)
54+
* TokenKinds should not be frozen [#130, thanks to Gavin Kistner]
55+
* display line numbers in HTML @:table@ mode even for single-line code (remove special case) [#41, thanks to Ariejan de Vroom]
56+
* override Bootstrap's @pre { word-break: break-all }@ styling for line numbers [#102, thanks to lightswitch05]
57+
* HTML encoder will not warn about unclosed token groups at the end of the stream
58+
* fixed problem with coderay/version.rb being loaded twice
59+
60+
Internals:
61+
62+
* The Debug scanner maps unknown token kinds to @:unknown@ (to avoid creating Symbols based on possibly unsafe input).
63+
* The Raydebug scanner highlights unknown token kinds as @:plain@.
64+
* The Debug encoder refactored; use DebugLint if you want strict checking now..
65+
* The Debug encoder will not warn about errors in the token stream.
66+
* Plugin does not warn about fallback when default is defined.
67+
* PluginHost now works with Strings instead of Symbols internally (to avoid using @#to_sym@).
1568

1669
h2. Changes in 1.0.9
1770

@@ -421,6 +474,3 @@ The helper classes were cleaned up; see above for details.
421474

422475
* *CHANGED* @Plugin@ API was simplified and stripped of all unnecessary features.
423476
* *CHANGED* Moved @GZip@ and @FileType@ libraries into @CodeRay@; cleaned them up.
424-
425-
426-

‎README.markdown

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ html = CodeRay.scan("puts 'Hello, world!'", :ruby).div(:line_numbers => :table)
2828

2929
## Documentation
3030

31-
See [http://coderay.rubychan.de/doc/](http://coderay.rubychan.de/doc/).
31+
See [rubydoc](http://rubydoc.info/gems/coderay).

‎Rakefile

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
require 'bundler/gem_tasks'
2+
13
$:.unshift File.dirname(__FILE__) unless $:.include? '.'
24

35
ROOT = '.'
@@ -32,4 +34,4 @@ else
3234
rd.rdoc_dir = 'doc'
3335
end
3436

35-
end
37+
end

‎bench/bench.rb

Lines changed: 31 additions & 153 deletions
Original file line numberDiff line numberDiff line change
@@ -1,168 +1,46 @@
1-
# The most ugly test script I've ever written!
2-
# Shame on me!
3-
4-
require 'pathname'
5-
require 'profile' if ARGV.include? '-p'
6-
7-
MYDIR = File.dirname(__FILE__)
8-
LIBDIR = Pathname.new(MYDIR).join('..', 'lib').cleanpath.to_s
9-
$:.unshift MYDIR, LIBDIR
1+
require 'benchmark'
2+
$: << File.expand_path('../../lib', __FILE__)
103
require 'coderay'
114

12-
@size = ARGV.fetch(2, 100).to_i * 1000
13-
14-
lang = ARGV.fetch(0) do
15-
puts <<-HELP
16-
Usage:
17-
ruby bench.rb (c|ruby|dump) (null|text|tokens|count|statistic|yaml|html) [size in kB] [stream]
18-
19-
SIZE defaults to 100 kB (= 100,000 bytes).
20-
SIZE = 0 means the whole input.
21-
SIZE is ignored when dump is input.
22-
23-
-p generates a profile (slow! use with SIZE = 1)
24-
-o shows the output
25-
stream enabled streaming mode
26-
27-
Sorry for the strange interface. I will improve it in the next release.
28-
HELP
5+
if ARGV.include? '-h'
6+
puts DATA.read
297
exit
308
end
319

32-
format = ARGV.fetch(1, 'html').downcase
33-
34-
$stream = ARGV.include? 'stream'
35-
$optimize = ARGV.include? 'opt'
36-
$style = ARGV.include? 'style'
37-
38-
require 'benchmark'
39-
require 'fileutils'
10+
lang = ARGV.fetch(0, 'ruby')
11+
data = nil
12+
File.open(File.expand_path("../example.#{lang}", __FILE__), 'rb') { |f| data = f.read }
13+
raise 'Example file is empty.' if data.empty?
4014

41-
if format == 'comp'
42-
format = 'page'
43-
begin
44-
require 'syntax'
45-
require 'syntax/convertors/html.rb'
46-
rescue LoadError
47-
puts 'Syntax no found!! (Try % gem install syntax)'
48-
end
49-
end
50-
51-
$dump_input = lang == 'dump'
52-
$dump_output = format == 'dump'
53-
require 'coderay/helpers/gzip_simple.rb' if $dump_input
15+
format = ARGV.fetch(1, 'html').downcase
16+
encoder = CodeRay.encoder(format)
5417

55-
def here fn = nil
56-
return MYDIR unless fn
57-
File.join here, fn
18+
size = ARGV.fetch(2, 1000).to_i * 1000
19+
unless size.zero?
20+
data += data until data.size >= size
21+
data = data[0, size]
5822
end
23+
size = data.size
24+
puts "encoding %d kB of #{lang} code to #{format}..." % [(size / 1000.0).round]
5925

60-
n = ARGV.find { |a| a[/^N/] }
61-
N = if n then n[/\d+/].to_i else 1 end
62-
$filename = ARGV.include?('strange') ? 'strange' : 'example'
63-
64-
Benchmark.bm(20) do |bm|
65-
N.times do
66-
67-
data = nil
68-
File.open(here("#$filename." + lang), 'rb') { |f| data = f.read }
69-
if $dump_input
70-
@size = CodeRay::Tokens.load(data).text.size
71-
else
72-
raise 'Example file is empty.' if data.empty?
73-
unless @size.zero?
74-
data += data until data.size >= @size
75-
data = data[0, @size]
76-
end
77-
@size = data.size
78-
end
79-
80-
options = {
81-
:tab_width => 2,
82-
# :line_numbers => :inline,
83-
:css => $style ? :style : :class,
84-
}
85-
$hl = CodeRay.encoder(format, options) unless $dump_output
86-
time = bm.report('CodeRay') do
87-
if $stream || true
88-
if $dump_input
89-
raise 'Can\'t stream dump.'
90-
elsif $dump_output
91-
raise 'Can\'t dump stream.'
92-
end
93-
$o = $hl.encode(data, lang, options)
94-
else
95-
if $dump_input
96-
tokens = CodeRay::Tokens.load data
97-
else
98-
tokens = CodeRay.scan(data, lang)
99-
end
100-
tokens.optimize! if $optimize
101-
if $dump_output
102-
$o = tokens.optimize.dump
103-
else
104-
$o = tokens.encode($hl)
105-
end
106-
end
107-
end
108-
$file_created = here('test.' +
109-
($dump_output ? 'dump' : $hl.file_extension))
110-
File.open($file_created, 'wb') do |f|
111-
# f.write $o
112-
end
113-
Dir.chdir(here) do
114-
FileUtils.copy 'test.dump', 'example.dump' if $dump_output
115-
end
116-
117-
time_real = time.real
118-
119-
puts "\t%7.2f KB/s (%d.%d KB)" % [((@size / 1000.0) / time_real), @size / 1000, @size % 1000]
120-
puts $o if ARGV.include? '-o'
121-
122-
end
26+
n = ARGV.fetch(3, 5).to_s[/\d+/].to_i
27+
require 'profile' if ARGV.include? '-p'
28+
n.times do |i|
29+
time = Benchmark.realtime { encoder.encode(data, lang) }
30+
puts "run %d: %5.2f s, %4.0f kB/s" % [i + 1, time, size / time / 1000.0]
12331
end
124-
puts "Files created: #$file_created"
12532

126-
STDIN.gets if ARGV.include? 'wait'
33+
STDIN.gets if ARGV.include? '-w'
12734

12835
__END__
129-
.ruby .normal {}
130-
.ruby .comment { color: #005; font-style: italic; }
131-
.ruby .keyword { color: #A00; font-weight: bold; }
132-
.ruby .method { color: #077; }
133-
.ruby .class { color: #074; }
134-
.ruby .module { color: #050; }
135-
.ruby .punct { color: #447; font-weight: bold; }
136-
.ruby .symbol { color: #099; }
137-
.ruby .string { color: #944; background: #FFE; }
138-
.ruby .char { color: #F07; }
139-
.ruby .ident { color: #004; }
140-
.ruby .constant { color: #07F; }
141-
.ruby .regex { color: #B66; background: #FEF; }
142-
.ruby .number { color: #F99; }
143-
.ruby .attribute { color: #7BB; }
144-
.ruby .global { color: #7FB; }
145-
.ruby .expr { color: #227; }
146-
.ruby .escape { color: #277; }
36+
Usage:
37+
ruby bench.rb [lang] [format] [size in kB] [number of runs]
14738

148-
.xml .normal {}
149-
.xml .namespace { color: #B66; font-weight: bold; }
150-
.xml .tag { color: #F88; }
151-
.xml .comment { color: #005; font-style: italic; }
152-
.xml .punct { color: #447; font-weight: bold; }
153-
.xml .string { color: #944; }
154-
.xml .number { color: #F99; }
155-
.xml .attribute { color: #BB7; }
39+
- lang defaults to ruby.
40+
- format defaults to html.
41+
- size defaults to 1000 kB (= 1,000,000 bytes). 0 uses the whole example input.
42+
- number of runs defaults to 5.
15643

157-
.yaml .normal {}
158-
.yaml .document { font-weight: bold; color: #07F; }
159-
.yaml .type { font-weight: bold; color: #05C; }
160-
.yaml .key { color: #F88; }
161-
.yaml .comment { color: #005; font-style: italic; }
162-
.yaml .punct { color: #447; font-weight: bold; }
163-
.yaml .string { color: #944; }
164-
.yaml .number { color: #F99; }
165-
.yaml .time { color: #F99; }
166-
.yaml .date { color: #F99; }
167-
.yaml .ref { color: #944; }
168-
.yaml .anchor { color: #944; }
44+
-h prints this help
45+
-p generates a profile (slow, use with SIZE = 1)
46+
-w waits after the benchmark (for debugging memory usw)

‎bin/coderay

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ when 'highlight', nil
125125
end
126126

127127
if output_file
128-
output_format ||= CodeRay::FileType[output_file]
128+
output_format ||= CodeRay::FileType[output_file] || :plain
129129
else
130130
output_format ||= :terminal
131131
end
@@ -143,7 +143,6 @@ when 'highlight', nil
143143
if output_file
144144
File.open output_file, 'w'
145145
else
146-
$stdout.sync = true
147146
$stdout
148147
end
149148
CodeRay.encode(input, input_lang, output_format, :out => file)
@@ -156,8 +155,9 @@ when 'highlight', nil
156155
puts boom.message
157156
end
158157
# puts "I don't know this plugin: #{boom.message[/Could not load plugin (.*?): /, 1]}."
159-
rescue CodeRay::Scanners::Scanner::ScanError # FIXME: rescue Errno::EPIPE
160-
# this is sometimes raised by pagers; ignore [TODO: wtf?]
158+
rescue CodeRay::Scanners::Scanner::ScanError
159+
# this is sometimes raised by pagers; ignore
160+
# FIXME: rescue Errno::EPIPE
161161
ensure
162162
file.close if output_file
163163
end

‎lib/coderay.rb

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -127,14 +127,14 @@ module CodeRay
127127

128128
$CODERAY_DEBUG ||= false
129129

130-
CODERAY_PATH = File.join File.dirname(__FILE__), 'coderay'
130+
CODERAY_PATH = File.expand_path('../coderay', __FILE__)
131131

132132
# Assuming the path is a subpath of lib/coderay/
133133
def self.coderay_path *path
134134
File.join CODERAY_PATH, *path
135135
end
136136

137-
require coderay_path('version')
137+
require 'coderay/version'
138138

139139
# helpers
140140
autoload :FileType, coderay_path('helpers', 'file_type')
@@ -166,7 +166,6 @@ class << self
166166
#
167167
# See also demo/demo_simple.
168168
def scan code, lang, options = {}, &block
169-
# FIXME: return a proxy for direct-stream encoding
170169
TokensProxy.new code, lang, options, block
171170
end
172171

0 commit comments

Comments
 (0)