Protocol Buffers (Protobufs) are a great way to pass information between your apps. Written by Google, Protobufs give you the readability of JSON but the structure of something like XML. Using the protoc
tool you can compile the Protobufs into a language of your choosing.
One use case for Protobufs is defining a shared domain model. All the applications in your ecosystem will then use the shared Protobufs. However, this is a challenge with Ruby on Rails. Unlike Go, Ruby does not have a great way to import a shared protobuf library.
One way is to copy and paste the shared Protobuf repo into your Ruby on Rails codebase. Unfortunately, this means that every app will have to handle updating the Protobuf repo whenever there is a change. Instead of committing the entire Protobuf repo, we can instead commit only the compile Protobufs. This is where the ruby_proto_compiler
gem comes in.
This gem contains a rake task that will pull down a versioned Protobuf repository, unpack it, compile the files, and include it in your Rails app. Here's a break down of the rake task and the design decisions I made.
Breakdown of the Rake task
Download Tar
Firstly, we need to download the archived Protobuf file from Github. Version matters, you have to have tagged your Protobuf repository. The download_url
will be a Github link e.g: https://github.com/jonathanyeong/2019-blog/archive/1.0.0.tar.gz
. The dest_folder
is where you want to place your downloaded tar file.
task :download_tar, [:dest_folder, :download_url] do |t, args|
begin
Zlib::GzipWriter.open(args[:dest_folder]) do |local_file|
open(args[:download_url]) do |remote_file|
puts "Downloading TAR: #{args[:download_url]}"
local_file.write(Zlib::GzipReader.new(remote_file).read)
end
end
rescue OpenURI::HTTPError => e
File.delete(args[:dest_folder])
abort "Error: downloading tar with this URL: #{args[:download_url]} caused this error: #{e}"
end
puts "Succesfully download the TAR file found here: #{args[:dest_folder]}"
end
Unpack Tar
This is largely copied from another blog post. The biggest change is setting the unzipped_folder
instance variable. This will be used later to tell protoc
where to find the Protobufs.
task :unzip, [:tar_path] do |t, args|
puts "Unzipping tar at #{args[:tar_path]}"
unzipped_folders = []
Gem::Package::TarReader.new( Zlib::GzipReader.open(args[:tar_path])) do |tar|
dest = nil
tar.each do |entry|
if entry.full_name == TAR_LONGLINK
dest = File.join(DEFAULT_DOWNLOAD_DIR, entry.read.strip)
next
end
dest ||= File.join DEFAULT_DOWNLOAD_DIR, entry.full_name
if entry.directory?
unzipped_folders << dest.gsub('tmp/', '')
FileUtils.rm_rf dest unless File.directory? dest
FileUtils.mkdir_p dest, :mode => entry.header.mode, :verbose => false
elsif entry.file?
FileUtils.rm_rf dest unless File.file? dest
File.open dest, "wb" do |f|
f.print entry.read
end
FileUtils.chmod entry.header.mode, dest, :verbose => false
elsif entry.header.typeflag == '2' #Symlink!
File.symlink entry.header.linkname, dest
end
dest = nil
end
end
puts "Finished unzipping tar at #{args[:tar_path]}"
@unzipped_folder = unzipped_folders.first.chomp('/')
end
Compile Protos
This hooks into your system to run the protoc
tool. It will find the Protobufs using the @unzipped_folder
variable and compile them to whatever your args[:output_dir]
is.
task :compile_protos, [:output_dir] do |t, args|
puts 'Generating Protos'
FileUtils.mkdir_p(args[:output_dir])
system("protoc tmp/#{@unzipped_folder}/*/*.proto --ruby_out=#{args[:output_dir]} -I tmp/#{@unzipped_folder}")
end
Require compiled Protos (if Rails)
In Rails you will need to include the compiled Protobuf files when you load your app. This rake task will create a new initializer file and add the corresponding requires to your compiled Protobufs.
task :include_protos, [:output_dir] do |t, args|
puts "Adding initializers/protobufs.rb file"
File.open(Rails.root.join('config/initializers/protobufs.rb'), 'w') do |f|
f.puts('Dir["#{Rails.application.config.root}/' + "#{args[:output_dir]}" + '*/*.rb"].each { |file| require file }')
end
end
Putting it all together
The generate task is the entry point to the whole rake task. It will assign some defaults, do some simple input validation, and then call the components mentioned above.
desc 'Generate the Protos'
task :generate, [:release, :github_archive_url, :output_dir] do |task, args|
args.with_defaults(:output_dir => DEFAULT_OUTPUT_DIR)
abort("Error: No Release Specified\n\n" + help_text) if args[:release].nil?
abort("Error: wrong output folder format #{args[:output_dir]} requires trailing /") unless args[:output_dir][-1].eql?('/')
destination_path = DEFAULT_DOWNLOAD_DIR + args[:release] + TAR_EXT
download_url = args[:github_archive_url] + args[:release] + TAR_EXT
Dir.mkdir(DEFAULT_DOWNLOAD_DIR) unless Dir.exist?(DEFAULT_DOWNLOAD_DIR)
Rake::Task['ruby_proto_compiler:download_tar'].invoke(destination_path, download_url)
Rake::Task['ruby_proto_compiler:unzip'].invoke(destination_path, download_url)
Rake::Task['ruby_proto_compiler:compile_protos'].invoke(args[:output_dir])
Rake::Task['ruby_proto_compiler:include_protos'].invoke(args[:output_dir]) if defined?(Rails)
end
What's next
There's a lot of improvements that I could make:
- There's no validation on the URL - but if it fails then no tar folder is generated and the rake task stops.
- The help text format is pretty bad.
- I don't like how arguments are passed into a rake task. For example, if you have spaces between the arguments then it won't run. Ideally, I would use
OptParse
to have a more robust parsing function. - The rake task itself is pretty messy. I'd invest more time figuring out how to make each task run individually.
- It'd be great to have a cleanup task. That will undo all the changes for you.
If you made it this far thanks for reading! Checkout the Github project to learn more about the gem.