Luke Imhoff
| luke_imhoff@rapid7.com | Kronic.Deth@gmail.com | |
| @limhoff-r7 | @KronicDeth | |
| @KronicDeth |
| Loader | Extension Required | Loads Only Once | Synchrony | Loads |
|---|---|---|---|---|
Kernel.load |
Yes | No | Synchronous | File |
Kernel.require |
No | Yes | Synchronous | File |
Kernel.autoload |
No | Yes | Asynchronous | File |
ActiveSupport::Autoload#autoload |
No | Yes | Asynchronous | File |
ActiveSupport::Dependencies#autoload_paths |
No | Either | Asynchronous | Directory Tree |
A standard Rails application
daemon UI uses to run metasploit-framework
610 lines
config.autoload_paths
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and
prosvc
ackconfig.autoload_paths outside Rails
config.autoload_paths in
prosvc
config.autoload_paths works
config.autoload_paths
config.autoload_paths is mentioned in the Configuring guide
set_autoload_paths?
| Gem | Purpose |
|---|---|
| actionmailer | sending mail |
| activerecord | database records |
| activeresource | non-database records |
| bundler | dependency management |
| actionpack | ? |
| activesupport | ? |
| railties | ? |
cd ~/git
mkdir rails
cd rails
git clone git@github.com:rails/rails.git
cd rails
git checkout v3.2.2
ack set_autoload_paths
ack set_autoload_path
railties/lib/rails/engine.rb:536
_all_autoload_paths
set_autoload_paths
config.autoload_paths
config.eager_load_paths
config.autoload_once_paths
ActiveSupport::Dependencies.autoload_paths.unshift(config.autoload_paths)
Rails::Engine
set_load_path
_all_load_paths
| System | Initializers | Configuration |
|---|---|---|
| Rails |
Rails::Engine
|
Rails::Engine::Configuration
|
| prosvc |
Metasploit::Configured
|
Metasploit::Configuration
|
require 'active_support/concern'
# TODO this should be a Rails::Engine if engine is ever moved under ui
module Metasploit::Configured
extend ActiveSupport::Concern
module ClassMethods
def configuration
@configuration ||= self::Configuration.new
end
def initialize!
configuration.initializers.each do |initializer|
send(initializer)
end
end
def eager_load!
configuration.dependencies.each(&:eager_load!)
# Adapted from Rails::Engine#eager_load!
configuration.eager_load_paths.each do |load_path|
# strip the load_path itself from the name because the load_path is already add to $LOAD_PATH by
# {#set_load_path}.
# strip extension as it's not normally passed to require.
require_path_regex = /\A#{Regexp.escape(load_path)}\/(.*)\.rb\Z/
glob = "#{load_path}/**/*.rb"
Dir.glob(glob) do |path|
require_path = path.sub(require_path_regex, '\1')
require_dependency require_path
end
end
end
def root
configuration.root
end
def set_load_path
configuration.dependencies.each(&:set_load_path)
# reverse because unshift is pushing onto the front
configuration.all_autoload_paths.reverse_each do |path|
if File.directory?(path)
$LOAD_PATH.unshift(path)
end
# uniq at end in case all_autoload_paths include paths already in $LOAD_PATH
$LOAD_PATH.uniq!
end
end
def set_autoload_paths
configuration.dependencies.each(&:set_autoload_paths)
ActiveSupport::Dependencies.autoload_paths.unshift(*configuration.all_autoload_paths)
# Freeze so future modifications error out instead of being silently ignored
configuration.autoload_paths.freeze
configuration.eager_load_paths.freeze
end
end
end
require 'active_support/concern'
module Metasploit::Configured
module ClassMethods
def configuration
@configuration ||= self::Configuration.new
end
end
end
self::Configuration
|
Configuration
|
|---|---|
| Scoped to base class | Scoped to Metasploit::Configured::ClassMethods |
module Metasploit::Configured
module ClassMethods
def initialize!
configuration.initializers.each do |initializer|
send(initializer)
end
end
end
end
initialize! to match Rails convention
# Load the rails application
require File.expand_path('../application', __FILE__)
# Initialize the rails application
Pro::Application.initialize!
module Metasploit::Configured
module ClassMethods
def eager_load!
configuration.dependencies.each(&:eager_load!)
# Adapted from Rails::Engine#eager_load!
configuration.eager_load_paths.each do |load_path|
# strip the load_path itself from the name because the load_path is
# already add to $LOAD_PATH by {#set_load_path}.
# strip extension as it's not normally passed to require.
require_path_regex = /\A#{Regexp.escape(load_path)}\/(.*)\.rb\Z/
glob = "#{load_path}/**/*.rb"
Dir.glob(glob) do |path|
require_path = path.sub(require_path_regex, '\1')
require_dependency require_path
end
end
end
end
end
module Metasploit::Configured
module ClassMethods
def root
configuration.root
end
end
end
module Metasploit::Configured
module ClassMethods
def set_load_path
configuration.dependencies.each(&:set_load_path)
# reverse because unshift is pushing onto the front
configuration.all_autoload_paths.reverse_each do |path|
if File.directory?(path)
$LOAD_PATH.unshift(path)
end
# uniq at end in case all_autoload_paths include paths
# already in $LOAD_PATH
$LOAD_PATH.uniq!
end
end
end
end
module Rails
class Engine < Railtie
def ordered_railties
railties.all + [self]
end
def initializers
initializers = []
ordered_railties.each do |r|
if r == self
initializers += super
else
initializers += r.initializers
end
end
initializers
end
end
end
module Metasploit::Configured
module ClassMethods
def set_load_path
configuration.dependencies.each(&:set_load_path)
# reverse because unshift is pushing onto the front
configuration.all_autoload_paths.reverse_each do |path|
if File.directory?(path)
$LOAD_PATH.unshift(path)
end
# uniq at end in case all_autoload_paths include paths
# already in $LOAD_PATH
$LOAD_PATH.uniq!
end
end
end
end
module Rails
class Engine < Railtie
initializer :set_load_path, :before => :bootstrap_hook do
_all_load_paths.reverse_each do |path|
$LOAD_PATH.unshift(path) if
File.directory?(path)
end
$LOAD_PATH.uniq!
end
end
end
module Metasploit
class Configuration
def all_autoload_paths
all_autoload_paths = autoload_paths + eager_load_paths
unique_autoload_paths = all_autoload_paths.uniq
unique_autoload_paths
end
def autoload_paths
@autoload_paths ||= []
end
def dependencies
@dependencies ||= []
end
def eager_load_paths
@eager_load_paths ||= []
end
def initializers
# XXX not sure I like that initializer methods that only exist in
# Metasploit::Configured are used here.
@initializers ||= [
:set_load_path,
:set_autoload_paths,
:eager_load!
]
end
attr_reader :root
end
end
module Metasploit
class Configuration
def initializers
# XXX not sure I like that initializer methods that only exist in
# Metasploit::Configured are used here.
@initializers ||= [
:set_load_path,
:set_autoload_paths,
:eager_load!
]
end
end
end
$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), 'lib'))) # engine/lib
$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), '..', 'msf3', 'lib'))) # msf3/lib
# ...
require_relative '../ui/config/initializers/carrierwave' # ui/config/initializers
#
# Load specific patches for system libraries
#
if arch == "win32"
require "win32/registry" # engine/lib
require "patches/win32_registry" # engine/lib
end
# ...
#
# Require our basic libraries and start loading
#
require 'metasploit_data_models' # (gem)
include MetasploitDataModels
# Figure out the rails path
rails_app_path = File.expand_path(File.dirname(__FILE__)) + "/../ui/app"
# Bring all Mdm::* models into view
Dir.glob("#{rails_app_path}/models/mdm/*.rb").each do |mdm_model_path| # ui/app/models
require mdm_model_path
end
# ---- BEGIN - Load all things SocialEngineering from Rails ----
module LiquidTemplating; end
liquid_files = Dir.glob("#{rails_app_path}/../lib/liquid_templating/*.rb") # ui/lib
liquid_files.sort.each do |liquid_path|
require liquid_path
end
# TODO: any better way to have both Rails and prosvc happy w/ talking to License?
require "#{rails_app_path}/models/license.rb" # ui/app/models
# Declare the module manually - Rails does this from directory structure
module SocialEngineering; end
require "#{rails_app_path}/uploaders/social_engineering/campaign_file_uploader.rb" # ui/app/uploaders
manually_loaded_models = ["human_target", "email", "web_page_attack_config_interface"]
skippable_files = ["campaign_task"]
require "#{rails_app_path}/models/social_engineering/human_target.rb" # ui/app/models
# avoid chicken-egg
require "#{rails_app_path}/models/social_engineering/web_page_attack_config_interface.rb" # ui/app/models
# avoid chicken-egg
se_files = Dir.glob("#{rails_app_path}/models/social_engineering/*.rb")
se_files.sort.each do |se_model_path|
file_name = File.basename(se_model_path).split('.').first
next if skippable_files.include? file_name
next if manually_loaded_models.include? file_name
require se_model_path # ui/app/models
end
# ---- END - Load all things SocialEngineering from Rails ----
require 'msf/ui' # msf3/lib
require 'rex' # msf3/lib
# Metasploit Core API
require 'msf/core/rpc/v10/service' # msf3/lib
# Pro API
require 'pro/filters' # engine/lib
require 'pro/config' # engine/lib
require 'pro/rpc/v10/rpc_pro' # engine/lib
# Pro Mixins
require 'pro/mixins' # engine/lib
# Pro Hooks
require 'pro/hooks' # engine/lib
# Pro Client
require 'pro/client' # engine/lib
# Background Daemon
require 'pro/bgdaemon' # engine/lib
# NginX
require "pro/nginx" # engine/lib
require 'logger' # (standard library)
# ...
# Load individual namespaces and models
require_relative "#{rails_app_path}/models/user_session.rb" # ui/app/models
require_relative "#{rails_app_path}/models/task_chain.rb" # ui/app/models
require_relative "#{rails_app_path}/models/scheduled_task.rb" # ui/app/models
require_relative "#{rails_app_path}/models/campaign.rb" # ui/app/models
require_relative "#{rails_app_path}/models/attachment.rb" # ui/app/models
require_relative "#{rails_app_path}/models/email_template.rb" # ui/app/models
require_relative "#{rails_app_path}/models/email_address.rb" # ui/app/models
require_relative "#{rails_app_path}/models/web_template.rb" # ui/app/models
# Load classes in lib root
ui_lib_path = File.expand_path(File.dirname(__FILE__)) + "/../ui/lib"
require "#{ui_lib_path}/sender.rb" # ui/lib
lib_files = Dir.glob("#{ui_lib_path}/*.rb")
lib_files.sort.each do |lib_class|
require lib_class # ui/lib
end
engine/libmsf3/libui/config/initializersui/app/modelsui/libui/app/uploaders
module Metasploit::Framework
class Configuration < Metasploit::Configuration
def initialize
@root = Metasploit::Pro.root.join('msf3')
lib_pathname = root.join('lib')
lib_path = lib_pathname.to_s
autoload_paths << lib_path
base_pathname = root.join('lib', 'base')
base_path = base_pathname.to_s
autoload_paths << base_path
end
end
end
module Metasploit::Pro::Engine
class Configuration < Metasploit::Configuration
def initialize
@dependencies = [
Metasploit::Framework,
Metasploit::Pro::UI
]
@root = Metasploit::Pro.root.join('engine')
lib_pathname = root.join('lib')
lib_path = lib_pathname.to_s
autoload_paths << lib_path
end
end
end
module Metasploit::Pro::UI
class Configuration < Metasploit::Configuration
def initialize
@root = Metasploit::Pro.root.join('ui')
lib_pathname = root.join('lib')
lib_path = lib_pathname.to_s
autoload_paths << lib_path
# Some models reference the controllers,
# so app/controllers needs to be added to autoload paths
controllers_pathname = root.join('app', 'controllers')
controllers_path = controllers_pathname.to_s
autoload_paths << controllers_path
uploaders_pathname = root.join('app', 'uploaders')
uploaders_path = uploaders_pathname.to_s
autoload_paths << uploaders_path
models_pathname = root.join('app', 'models')
models_path = models_pathname.to_s
autoload_paths << models_path
end
end
end
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and
prosvc
modules directorymsfconsoleuse command can instantiate a Metasploit Moduleset command configures the instance of the Metasploit Modulerun launch the instance of the Metasploit Modulemsfpromsfconsole with access to Metasploit Pro features
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
namename.presenceklass_nameklass_name.to_sklass_name.to_s.scannestingModule#name
Module#name containing
::
Metasploit3?
MetasploitN
N is Metasploit3 Ruby Classes/ModulesMetasploit4 Ruby Classes/Modulesrequire_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new usages
Module.new Usages
Msf::ModuleManager#reload_module
Msf::ModuleManager#load_module_from_file
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
Module.new
name is
/ separated (
auxiliary/pro/social_engineering/web_phish)
String#camelize will convert
/ separated to
Module#name
>> "auxiliary/pro/social/engineering/web_phish".camelize
=> "Auxiliary::Pro::SocialEngineering::WebPhish"
Object?
Hex-escaping
child_name not passed to Module.new
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
#cache_entries,
#rebuild_cache,
#refresh_cache)
Msf::Module instances (
#create)
#demand_load_module,
#failed,
#has_module_file_changed?,
#load_module_from_file,
#load_module_source,
#load_modules,
#load_modules_from_directory,
#on_module_load)
#auxiliary,
#encoders,
#exploits,
#init_module_set,
#module_names,
#module_set,
#module_types,
#nops,
#payloads,
#post)
#add_module_path,
#remove_module_path)
#reload_module,
#reload_modules)
#add_module,
#auto_subscribe_module,
#register_type_extension)
Msf::ModuleManager#add_module
#auto_subscribe_moduleframework.events.on_module_loadProbably loading related...
Called from #on_module_load, so definitely loading
Msf::ModuleManager#register_type_extension
Dead Code?
def = dead
include Ruby Module in base Ruby Class/Module
module Msf
class ModuleManager
require 'msf/core/module_manager/cache'
include Msf::ModuleManager::Cachet
end
end
| Internal | External | |
|---|---|---|
| Single | add_module | |
| autosubscribe_module | ||
| demand_load_module | ||
| failed | ||
| has_module_file_changed? | ||
| load_module_from_file | ||
| load_module_source | ||
| on_module_load | ||
| Multiple | load_modules | load_modules_from_directory |
Msf::ModuleSet#add_module
Msf::ModuleManager::Loading#auto_subscribe_module too due to usage
#load_module_from_file in directory category, so extract to Msf::Modules::Loader::Directory
Check Usages to confirm category
Msf::ModuleSetframework.modules.demand_load_moduleKeep in Msf::ModuleManager::Loading
modules_failed attribute.Find usages to check for public interface
framework.modules.failed, so part of public interfaceKeep in Msf::ModuleManager::Loading
on_* implies callbackMsf::ModuelManager or Msf::ModuleSet stateKeep in Msf::ModuleManager::Loading
class Msf::Modules::Loader::Directory < Msf::Modules::Loader::Base
protected
# Yields the module_reference_name for each module file found under the directory path.
#
# @param [String] path The path to the directory.
# @yield (see Msf::Modules::Loader::Base#each_module_reference_name)
# @yieldparam [String] path The path to the directory.
# @yieldparam [String] type The type correlated with the directory under path.
# @yieldparam module_reference_name (see Msf::Modules::Loader::Base#each_module_reference_name)
# @return (see Msf::Modules::Loader::Base#each_module_reference_name)
def each_module_reference_name(path)
::Dir.foreach(path) do |entry|
if entry.downcase == '.svn'
next
end
full_entry_path = ::File.join(path, entry)
type = entry.singularize
unless ::File.directory?(full_entry_path) and
module_manager.type_enabled? type
next
end
full_entry_pathname = Pathname.new(full_entry_path)
# Try to load modules from all the files in the supplied path
Rex::Find.find(full_entry_path) do |entry_descendant_path|
if module_path?(entry_descendant_path)
entry_descendant_pathname = Pathname.new(entry_descendant_path)
relative_entry_descendant_pathname = entry_descendant_pathname.relative_path_from(full_entry_pathname)
relative_entry_descendant_path = relative_entry_descendant_pathname.to_path
# The module_reference_name doesn't have a file extension
module_reference_name = module_reference_name_from_path(relative_entry_descendant_path)
yield path, type, module_reference_name
end
end
end
end
end
.rb files
Msf::Modules::Loader::Base#load_module
#module_path#read_module_content#namespace_module_transaction
# Returns the full path to the module file on disk.
#
# @param (See Msf::Modules::Loader::Base#module_path
# @return [String] Path toe the module file on disk.
def module_path(parent_path, type, module_reference_name)
file_name = module_reference_name + MODULE_EXTENSION
type_directory = DIRECTORY_BY_TYPE[type]
full_path = File.join(parent_path, type_directory, file_name)
full_path
end
# Load the module content from the on disk file.
#
# @param (see Msf::Modules::Loader::Base#read_module_content)
# @return (see Msf::Modules::Loader::Base#read_module_content)
def read_module_content(parent_path, type, module_reference_name)
full_path = module_path(parent_path, type, module_refrence_name)
File.read(full_path)
end
#load_module
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
Congratulation! The test case works, but... does the test case cover all ways that all 1851 Metasploit Modules are actually written?
NOPE!
| Module Creation | #create_module_namespace |
|---|---|
| Module Evaluation | #module_eval_with_lexical_scope |
Module#module_eval work?Msf from lexical scope
Module or Class
module Msf::ModuleManager
puts Module.nesting.inspect # [Msf::ModuleManager]
end
module Msf
module ModuleManager
puts Module.nesting.inspect # [Msf::ModuleManager, Msf]
end
end
[Msf::ModuleManager, Msf]
[Msf::Modules::Loader::Base]require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
Msf
Msf namespace, which is invalid for metasploit-framework (Metasploit::Framework is correct)Module#module_eval with block doesn't capture lexical scope; have to use Stringrequire_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
Msf
Msf in lexical scope using Module#module_eval(String)
wrapper_module (bugged)
namespace_module_names (fixed)
module_eval
require_relative in prosvcrequires in
prosvc
autoload_paths between UI and prosvc
Module.new usages
Module.new in Msf::ModuleManager#load_module_from_file
Module.new in Msf::ModuleManager#reload_module
Msf in lexical scope using Module#module_eval(String)
Module.new creates anonymous Ruby Modules
Module#name when a constant is set to the ModuleActiveSupport::Dependencies’s const_missingClass or Module per fileincluded ModulesClass hierarchiesmodule_eval
module_eval captures the lexical scope only with Stringmodule declaration has a different lexical scope than :: separated namesModule.nestingmodule_eval allows debugging in string code.I’d like to thank James “Egypt” Lee for imparting his knowledge of how to use the various msfconsole commands to ensure my changes didn’t break anything, fixing my lambda vs proc bug, and Ruby 1.8-incompatibility. I’d like to thank Samuel “Shuckins” Huckins for testing these changes against Metasploit Pro. I’d like to thank HD Moore for spotting when I missed automatic namespace names colliding with real Ruby Modules. I’d like to thank Trevor Rosen for allowing me to spend weeks to fix this the right way. I’d like to thank Regina Imhoff and Sonny Gonzalez for reviewing the accompanying article. I'd like to thank the Metasploit team for reviewing this presentation.