Reduce allocations in `Gem::BUNDLED_GEMS.warning?` #13727

joshuay03 · 2025-06-27T14:08:16Z

When profiling our (@buildkite's) Rails application's boot, this String#split consistently showed up as taking 3-4% of allocation samples (~125k+ allocations) with a sample rate of 100. I suspect this may be related to our use of bootsnap given the comments in this method, though I haven't tested without it to confirm.

Unfortunately I can't share that specific a profile, but I tried to compare profiles using a micro test script

# build/test.rb

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "vernier"

  gem "rails", require: false
  gem "puma", require: false
  gem "sidekiq", require: false
  gem "rspec", require: false

  gem 'bootsnap', require: false
end

require "bootsnap"
Bootsnap.setup(development_mode: true, cache_dir: "tmp/cache")

Vernier.profile(out: "../alloc_build.json", allocation_interval: 10) do
  require "rails"
  require "puma"
  require "sidekiq"
  require "rspec"
end

# rm -rf ~/.rubies/ruby-master/lib/ruby/gems/3.5.0+2/gems && rm -rf tmp/cache && make && make install && ~/.rubies/ruby-master/bin/ruby ../test.rb

Viewer: https://vernier.prof/ - filter data source to allocations and invert call tree stack

~70% fewer allocations

I didn't open an issue for this, but I did come across https://bugs.ruby-lang.org/issues/20641, so it seems this method is known to be somewhat of a hotspot.

joshuay03 · 2025-06-27T14:09:33Z

lib/bundled_gems.rb

-      segments = feature.split("/")
+      segments = feature.split("/", 3)


This branch needs to have three segments at most.

Do you know why it only needs 3 segments at most? I don't doubt it, but it's not obvious (to me) from the code why that is. I think a comment explaining the number would be helpful.

Ah, because only two shifts. 🤦

Yes two shifts and then just one needed for the segments.any? return value. I'll add a comment 👍🏽

joshuay03 · 2025-06-27T14:10:40Z

lib/bundled_gems.rb

-        name = [name, segments.shift].join("-")
+        name = "#{name}-#{segments.shift}"


While I was here I thought I'd eliminate this array allocation. This was only showing up at around 0.5%, but why not 🤷🏽‍♂️

joshuay03 · 2025-06-27T14:12:46Z

I suspect this may be related to our use of bootsnap given the comments in this method, though I haven't tested without it to confirm.

Actually I did just confirm this. Without bootsnap, I see many features like psych/y, but with bootsnap they're mostly like /Users/me/Projects/buildkite/buildkite/.bundle/ruby/3.5.0+2/gems/psych-5.2.6/lib/psych/y i.e. a lot of unnecessary splitting.

nobu · 2025-06-28T10:29:35Z

lib/bundled_gems.rb

@@ -105,11 +105,14 @@ def self.warning?(name, specs: nil)
      # and `require "syslog"` to `require "#{ARCHDIR}/syslog.so"`.
      feature.delete_prefix!(ARCHDIR)
      feature.delete_prefix!(LIBDIR)
-      segments = feature.split("/")
+      # 1. A segment for the basic EXACT and SINCE check e.g. `logger`
+      # 2. A segment for the multi-part SINCE check e.g. `resolv-replace`


I don't think that resolv-replace, which provides resolv-replace.rb as the name says, is the targets of this code.
The targets are net/* family, net-http gem for require "net/http".

I suspect that this code can be removed since such libraries are no longer in SINCE table.

"This code" is joining the first two segments with -.

Thanks, I've removed that and updated the comments. I also had to remove a failing test for net/smtp. I think I can also remove this one?

ruby/spec/bundled_gems_spec.rb

Lines 347 to 364 in eab4a0b

it "Don't show warning with net/smtp when net-smtp on Gemfile" do

build_lib "net-smtp", "1.0.0" do |s|

s.write "lib/net/smtp.rb", "puts 'net-smtp'"

end

script <<-RUBY, env: { "BUNDLER_SPEC_GEM_REPO" => gem_repo1.to_s }

gemfile do

source "https://gem.repo1"

path "#{lib_path}" do

gem "net-smtp"

end

end

require "net/smtp"

RUBY

expect(err).to be_empty

end

It passes, but it's probably a false positive now.

As well as the SINCE entry here?

ruby/spec/bundled_gems_spec.rb

Line 66 in eab4a0b

Gem::BUNDLED_GEMS.const_set(:SINCE, { "openssl" => RUBY_VERSION, "fileutils" => RUBY_VERSION, "csv" => "3.4.0", "net-smtp" => "3.1.0" })

I've gone ahead and removed them.

@hsbt What do you think? Or we should keep the feature?

I don't want to remove these tests.

Okay I've put all of that back and kept the comments example-free, given the spec has good examples.

joshuay03 commented Jun 27, 2025

View reviewed changes

joshuay03 added this to Open Source Jun 27, 2025

joshuay03 moved this to In Progress / Pending Review in Open Source Jun 27, 2025

joshuay03 force-pushed the reduce-allocations-in-bundled-gems-warning branch from b454ef5 to 0d779a1 Compare June 28, 2025 00:43

joshuay03 requested a review from tenderlove June 28, 2025 00:44

nobu reviewed Jun 28, 2025

View reviewed changes

joshuay03 force-pushed the reduce-allocations-in-bundled-gems-warning branch 2 times, most recently from 66e2039 to b7bb0ab Compare June 28, 2025 12:09

joshuay03 requested a review from nobu June 28, 2025 12:11

This comment has been minimized.

Sign in to view

Reduce allocations in Gem::BUNDLED_GEMS.warning?

9cf6b95

joshuay03 force-pushed the reduce-allocations-in-bundled-gems-warning branch from b7bb0ab to 9cf6b95 Compare June 30, 2025 10:40

joshuay03 requested a review from hsbt June 30, 2025 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reduce allocations in `Gem::BUNDLED_GEMS.warning?` #13727

Reduce allocations in `Gem::BUNDLED_GEMS.warning?` #13727

Uh oh!

joshuay03 commented Jun 27, 2025 •

edited

Loading

Uh oh!

joshuay03 Jun 27, 2025

Uh oh!

tenderlove Jun 27, 2025

Uh oh!

tenderlove Jun 27, 2025

Uh oh!

joshuay03 Jun 28, 2025

Uh oh!

joshuay03 Jun 27, 2025

Uh oh!

joshuay03 commented Jun 27, 2025 •

edited

Loading

Uh oh!

nobu Jun 28, 2025

Uh oh!

nobu Jun 28, 2025

Uh oh!

joshuay03 Jun 28, 2025 •

edited

Loading

Uh oh!

joshuay03 Jun 28, 2025

Uh oh!

nobu Jun 28, 2025

Uh oh!

hsbt Jun 30, 2025

Uh oh!

joshuay03 Jun 30, 2025

Uh oh!

This comment has been minimized.

Uh oh!

		segments = feature.split("/")
		segments = feature.split("/", 3)

		name = [name, segments.shift].join("-")
		name = "#{name}-#{segments.shift}"

	it "Don't show warning with net/smtp when net-smtp on Gemfile" do
	build_lib "net-smtp", "1.0.0" do \|s\|
	s.write "lib/net/smtp.rb", "puts 'net-smtp'"
	end

	script <<-RUBY, env: { "BUNDLER_SPEC_GEM_REPO" => gem_repo1.to_s }
	gemfile do
	source "https://gem.repo1"
	path "#{lib_path}" do
	gem "net-smtp"
	end
	end

	require "net/smtp"
	RUBY

	expect(err).to be_empty
	end

Reduce allocations in Gem::BUNDLED_GEMS.warning? #13727

Are you sure you want to change the base?

Reduce allocations in Gem::BUNDLED_GEMS.warning? #13727

Uh oh!

Conversation

joshuay03 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshuay03 commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshuay03 Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

Uh oh!

Reduce allocations in `Gem::BUNDLED_GEMS.warning?` #13727

Reduce allocations in `Gem::BUNDLED_GEMS.warning?` #13727

joshuay03 commented Jun 27, 2025 •

edited

Loading

joshuay03 commented Jun 27, 2025 •

edited

Loading

joshuay03 Jun 28, 2025 •

edited

Loading