daengine - Digital Asset File parsing and search for OFI

daengine is a rails engine plugin gem that provides access to digital assets, aka SSC documents/pdfs such as fund documents from the rails applications. It also includes the parser code that reads digital-asset metadata produced by the Teamsite legacy CMS and ingests it into a MongoDB instance for use by the client API..

Installation

To install the gem for use as a server process

$ gem install daengine

To install within a rails app as an engine (for client API use)

# Use the digital-asset api
gem "daengine"

daengine Server Process

The server process of daengine is packaged as an executable inside the gem called process_assets. You execute the server portion of the gem on a single-instance where the Teamsite tuple .xml files are accessible and you must provide a configuration .yml file to the executable that specifies the location of the tuple files and the MongoDB connection settings ala...

$ cat config_prod.yml
hosts: [[prod-db1,22785], [prod-db3,22785]]
database: the_database_where_you_want_stuff_to_go
assets_path: /tmp/some/teamsite/tuple/file/dir/

You then execute the server process as so:

$ process_assets config_prod.yml

daengine Client API

The data produced by the daengine server process can be consumed by any rails app that includes the daengine gem and has an existing Mongoid connection to the MongoDB database that the producer side writes to. Client access is either directly thru the DigitalAssets model class, or via the engine-provided REST API ie http://localhost:3000/digital_assets.json

Querying for Digital Assets

Querying assets via the Model Class...

DigitalAsset.guid_is('/asdfasdflasdfalasdfasdfasdf.asfdasdf.foo')
DigitalAsset.path_is('/digitalAssets/some-crazy-guid.pdf')
DigitalAsset.sami_is('FOOBAR.001')

Query via the REST API...

get 'http://localhost:3000/digital_assets/search?sami=FOOBAR.001'
get 'http://localhost:3000/digital_assets/search?funds=12345'

Scoped Query Test for finding Digital Assets by fund code

Point your dummy rails app to the correct database based on your setting in #RAILS_ROOT\spec\dummy\config\mongoid.yml. Seed the database with DigitalAssets and TaxonomyTerms

  1. Go to your dummy folder cd #RAILS_ROOT\spec\dummy

  2. Start the rails console jruby -S rails c

  3. Create an instance of AssetLookupService klass = AssetLookupService.new assets = klass.find_documents_by_fund_code("1365")

To test the app locally as a project from irb console

Update your #RAILS_ROOT/client/config.yml

  1. Ensure mongo is running locally
  2. Provide a database name. Asset metadata records will be stored in edist database in mongo.
  3. Ensure mongo port is correct in config.yml
  4. Ensure assets_path info is correct. [Sample deploy files provided under #RAILS_ROOT/client/digital-assets/*.xml]
  5. Set paths for daengine.yml, daengine.log in config.yml

%%% Sample config.yml %%% host: localhost database: edist port: 27017 assets_path: c:/tmp/daengine/digital_assets daengine_yml_file: c:/tmp/daengine/daengine.yml logfile_location: c:/tmp/daengine.log digital_assets_file_directory: V:/sscqa/STAGING/ #location of digital asset files in TeamSite Staging content_service_url: http://ofiwsqa.den.ofi.com/ taxonomy_xml_filepath: c:/tmp/taxonomyengine/taxonomy.xml taxonomyengine_yml_file: c:/tmp/taxonomyengine/taxonomyengine.yml taxonomyengine_logfile_location: c:/tmp/taxonomyengine/taxonomyengine.log content_service_logfile_location: c:/tmp/daengine/content_service.log

  1. Go to your project root (#RAILS_ROOT)

Process assets

process_assets

Process taxonomy

process_taxonomy

"Optionally", you can start the irb using the command from your project root directory:

C:\dev\daengine> irb -Ilib -rdaengine irb(main): config = YAML.load_file('#RAILS_ROOT\client\config.yml') irb(main): Daengine.execute(config)

At the end of this, "digital_assets" collection will be created with all asset metadata records in "edist" database.