Coverband 3
Coverband 3
Coverband gives deep insight into the production usage of every line of code in your Ruby app. Find dead code, find hotspots, see code paths that might be unexpectedly in high usage.
This release marks deep dives into Ruby performance, benchmarking, and driving almost all features based on improving performance. The release is a majory performance improvement, resolving some issues were Coverband 2 could have outliers that performed worse than the original Coverband 1 gem.
Me first and the Gimme Gimmes, a punk cover band, image from Wikimedia
Want to see it in action, visit the Coverband Demo Site, visit a few pages and watch the coverage report change as you use the site.
What breaking changes are there?
- drops Tracepoint
- drops Ruby <= 2.3.0
- drops JSON Gem dependency
- drops various other features not needed without Tracepoint
- memory cache, sampling, restricted to app folders, etc
What is new?
- standardizes on Coverage array format vs sparse hash
- rewrites store methods, for 60X perf!
- implemented for Redis and File store
- improved mountable web interface
What exactly is faster?
tldr;
Storing Coverage data to a data store. While this didn’t happen often when it did it could generate thousands of Redis requests on large apps, which could make for a very slow outlier request…
Long Version
The primary improvements come down to changing the storage format and moving from method of storage that required one call per file and assummed a small list of files, to a 2 pass stoage for all data. Basically, the previous storage mechanism created an N+1 type issue. Coverband 2 had solved the issue of capturing Coverage data, which was no longer a performance concern, in fact it captured all files by default as there wasn’t really a reason to filter for performance anymore… This created a new problem when trying to store coverage data. Eventually a very specific benchmark was added that simulates trying to store coverage reports for apps with 2934 files tracking coverage. In Coverband 3 by changing the storage format and the storage plan the storage benchmark improved by nearly 60X!
Simulate repeatedly storing coverage reports with Coverband 2.0.3:
rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
store_reports 1.000 i/100ms
Calculating -------------------------------------
store_reports 0.814 (± 0.0%) i/s - 13.000 in 15.982911s
Simulate repeatedly storing coverage reports with Coverband 3.0.0:
rake benchmarks:redis_reporting
runs benchmarks on reporting large sets of files to redis
Warming up --------------------------------------
store_reports 4.000 i/100ms
Calculating -------------------------------------
store_reports 47.111 (± 4.2%) i/s - 708.000 in 15.066259s
How did this impact Rails Apps
In general, depending on various settings the outliers weren’t that noticable or it made Coverband unusable for folks. I configured a Rails app with non ideal, but not terrible options, to show what this could look like and how Coverband 3 resolves the issue. If configured idealy even Coverband 2 would mostly only show in outliers, so this test is showing how even in a bad setup V3 performs with nearly no impact.
The benchmarks below are a further iteration of my AB Benchmarking Rails apps process. Now much easier to read and compare, by graphing by following this Apache Bench via Gnuplot guide.
| No Coverband | Coverband 2 | Coverband 3 |
|---|---|---|
 |
 |
 |
| mean: 27.272 [ms] | mean: 35.762 [ms] | mean: 28.460 [ms] |
| 36.67 [#/sec] (mean) | 27.96 [#/sec] (mean) | 35.14 [#/sec] (mean) |
Notice that with no Coverband and Coverband 3, the majority of requests show little variation, while Coverband 2 has frequent outliers pushing request timing all over the place. See full sized images of the benchmarks below.
What is next?
I am glad you asked, as we introduced a roadmap and set of feature discussions for upcoming Coverband improvements. Check out the changes.md for the latest, but in short here is some of what we have in the works.
- full Railties support, near zero config Rails integration
- solving the coverage drift problem…
- adding out of band data collection methods.
- droppoing simplecov dependency
- adding an Active Record adapter
Please give Coverband a shot and add any issue you have to the repo.
Footnotes
For those crazy few that really want all the data on the benchmarks…
No Coverband
|
rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)
Concurrency Level: 10
Time taken for tests: 54.543 seconds
Complete requests: 2000
Failed requests: 0
Requests per second: 36.67 [#/sec] (mean)
Time per request: 272.717 [ms] (mean)
Time per request: 27.272 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
50% 262
66% 269
75% 273
80% 276
90% 288
95% 302
98% 328
99% 350
100% 1400 (longest request)
Coverband 2
|
rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)
Concurrency Level: 10
Time taken for tests: 71.524 seconds
Complete requests: 2000
Failed requests: 0
Requests per second: 27.96 [#/sec] (mean)
Time per request: 357.619 [ms] (mean)
Time per request: 35.762 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
50% 325
66% 353
75% 378
80% 398
90% 489
95% 545
98% 622
99% 682
100% 1442 (longest request)
Coverband 3
rake benchmarks:coverband_demo_graph
Benchmarking coverband-demo.herokuapp.com (be patient)
Concurrency Level: 10
Time taken for tests: 56.919 seconds
Complete requests: 2000
Failed requests: 0
Total transferred: 16565120 bytes
HTML transferred: 14974000 bytes
Requests per second: 35.14 [#/sec] (mean)
Time per request: 284.597 [ms] (mean)
Time per request: 28.460 [ms] (mean, across all concurrent requests)
Percentage of the requests served within a certain time (ms)
50% 273
66% 282
75% 290
80% 296
90% 319
95% 344
98% 381
99% 410
100% 1358 (longest request)