Class: Measures::CqlLoader
- Inherits:
-
BaseLoaderDefinition
- Object
- BaseLoaderDefinition
- Measures::CqlLoader
- Defined in:
- lib/measures/loading/cql_loader.rb
Overview
Utility class for loading CQL measure definitions into the database from the MAT export zip
Class Method Summary collapse
-
.generate_single_code_references(elms, all_codes_and_code_names, user) ⇒ Object
Add single code references by finding the codes from the elm and creating new ValueSet objects With a generated GUID as a fake oid.
-
.get_files_from_zip(zip_file, out_dir) ⇒ Object
Opens the zip and grabs the cql file contents, the ELM contents (XML and JSON) and hqmf_path.
-
.get_value_set_oid_version_objects(value_sets, single_code_references) ⇒ Object
returns a list of objects that include the valueset oids and their versions.
- .load(file, user, measure_details, vsac_options, vsac_ticket_granting_ticket) ⇒ Object
- .load_mat_cql_exports(user, zip_file, out_dir, measure_details, vsac_options, vsac_ticket_granting_ticket) ⇒ Object
- .mat_cql_export?(zip_file) ⇒ Boolean
-
.modify_value_set_versions(elms) ⇒ Object
Adjusting value set version data.
-
.process_cql(files, main_cql_library, user, vsac_options, vsac_ticket_granting_ticket, measure_id = nil) ⇒ Object
Manages all of the CQL processing that is not related to the HQMF.
-
.replace_codesystem_oids_with_names(elms) ⇒ Object
Replace all the code system ids that are oids with the friendly name of the code system TODO: preferred solution would be to continue using OIDs in the ELM and enable Bonnie to supply those OIDs to the calculation engine in patient data and value sets.
- .set_data_criteria_code_list_ids(json, cql_artifacts) ⇒ Object
Methods inherited from BaseLoaderDefinition
Class Method Details
.generate_single_code_references(elms, all_codes_and_code_names, user) ⇒ Object
Add single code references by finding the codes from the elm and creating new ValueSet objects With a generated GUID as a fake oid.
233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 |
# File 'lib/measures/loading/cql_loader.rb', line 233 def self.generate_single_code_references(elms, all_codes_and_code_names, user) single_code_references = [] # Add all single code references from each elm file elms.each do | elm | # Check if elm has single reference code. if elm['library'] && elm['library']['codes'] && elm['library']['codes']['def'] # Loops over all single codes and saves them as fake valuesets. elm['library']['codes']['def'].each do |code_reference| code_sets = {} # look up the referenced code system code_system_def = elm['library']['codeSystems']['def'].find { |code_sys| code_sys['name'] == code_reference['codeSystem']['name'] } code_system_name = code_system_def['id'] code_system_version = code_system_def['version'] code_sets[code_system_name] ||= [] code_sets[code_system_name] << code_reference['id'] # Generate a unique number as our fake "oid" based on parameters that identify the DRC code_hash = "drc-" + Digest::SHA2.hexdigest("#{code_system_name} #{code_reference['id']} #{code_reference['name']} #{code_system_version}") # Keep a list of generated_guids and a hash of guids with code system names and codes. single_code_references << { guid: code_hash, code_system_name: code_system_name, code: code_reference['id'] } all_codes_and_code_names[code_hash] = code_sets # code_hashs are unique hashes, there's no sense in adding duplicates to the ValueSet collection if !HealthDataStandards::SVS::ValueSet.all().where(oid: code_hash, user_id: user.id).first() # Create a new "ValueSet" and "Concept" object and save. valueSet = HealthDataStandards::SVS::ValueSet.new({oid: code_hash, display_name: code_reference['name'], version: '' ,concepts: [], user_id: user.id}) concept = HealthDataStandards::SVS::Concept.new({code: code_reference['id'], code_system_name: code_system_name, code_system_version: code_system_version, display_name: code_reference['name']}) valueSet.concepts << concept valueSet.save! end end end end # Returns a list of single code objects and a complete list of code systems and codes for all valuesets on the measure. return single_code_references, all_codes_and_code_names end |
.get_files_from_zip(zip_file, out_dir) ⇒ Object
Opens the zip and grabs the cql file contents, the ELM contents (XML and JSON) and hqmf_path.
271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 |
# File 'lib/measures/loading/cql_loader.rb', line 271 def self.get_files_from_zip(zip_file, out_dir) Zip::ZipFile.open(zip_file.path) do |file| cql_entries = file.glob(File.join('**','**.cql')).select {|x| !x.name.starts_with?('__MACOSX') } zip_xml_files = file.glob(File.join('**','**.xml')).select {|x| !x.name.starts_with?('__MACOSX') } elm_json_entries = file.glob(File.join('**','**.json')).select {|x| !x.name.starts_with?('__MACOSX') } begin cql_paths = [] cql_entries.each do |cql_file| cql_paths << extract(file, cql_file, out_dir) if cql_file.size > 0 end cql_contents = [] cql_paths.each do |cql_path| cql_contents << open(cql_path).read end elm_json_paths = [] elm_json_entries.each do |json_file| elm_json_paths << extract(file, json_file, out_dir) if json_file.size > 0 end elm_json = [] elm_json_paths.each do |elm_json_path| elm_json << open(elm_json_path).read end xml_file_paths = extract_xml_files(file, zip_xml_files, out_dir) elm_xml_paths = xml_file_paths[:ELM_XML] elm_xml = [] elm_xml_paths.each do |elm_xml_path| elm_xml << open(elm_xml_path).read end files = { :HQMF_XML_PATH => xml_file_paths[:HQMF_XML], :ELM_JSON => elm_json, :CQL => cql_contents, :ELM_XML => elm_xml } return files rescue Exception => e raise MeasureLoadingException.new "Error Parsing Measure Logic: #{e.}" end end end |
.get_value_set_oid_version_objects(value_sets, single_code_references) ⇒ Object
returns a list of objects that include the valueset oids and their versions
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 |
# File 'lib/measures/loading/cql_loader.rb', line 180 def self.get_value_set_oid_version_objects(value_sets, single_code_references) # [LDC] need to make this an array of objects instead of a hash because Mongo is # dumb and *let's you* have dots in keys on object creation but *doesn't let you* # have dots in keys on object update or retrieve.... value_set_oid_version_objects = [] value_sets.each do |vs| value_set_oid_version_objects << {:oid => vs.oid, :version => vs.version} end single_code_references.each do |single_code| # Only add unique Direct Reference Codes to the object unless value_set_oid_version_objects.include?({:oid => single_code[:guid], :version => ""}) value_set_oid_version_objects << {:oid => single_code[:guid], :version => ""} end end # Return a list of unique objects only value_set_oid_version_objects end |
.load(file, user, measure_details, vsac_options, vsac_ticket_granting_ticket) ⇒ Object
86 87 88 89 90 91 92 |
# File 'lib/measures/loading/cql_loader.rb', line 86 def self.load(file, user, measure_details, , vsac_ticket_granting_ticket) measure = nil Dir.mktmpdir do |dir| measure = load_mat_cql_exports(user, file, dir, measure_details, , vsac_ticket_granting_ticket) end measure end |
.load_mat_cql_exports(user, zip_file, out_dir, measure_details, vsac_options, vsac_ticket_granting_ticket) ⇒ Object
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
# File 'lib/measures/loading/cql_loader.rb', line 25 def self.load_mat_cql_exports(user, zip_file, out_dir, measure_details, , vsac_ticket_granting_ticket) measure = nil cql = nil hqmf_path = nil # Grabs the cql file contents, the elm_xml contents, elm_json contents and the hqmf file path files = get_files_from_zip(zip_file, out_dir) # Load hqmf into HQMF Parser hqmf_model = Measures::Loader.parse_hqmf_model(files[:HQMF_XML_PATH]) # Get main measure from hqmf parser main_cql_library = hqmf_model.cql_measure_library cql_artifacts = process_cql(files, main_cql_library, user, , vsac_ticket_granting_ticket, hqmf_model.hqmf_set_id) # Create CQL Measure hqmf_model.backfill_patient_characteristics_with_codes(cql_artifacts[:all_codes_and_code_names]) json = hqmf_model.to_json json.convert_keys_to_strings # Set the code list ids of data criteria and source data criteria that use direct reference codes to GUIDS. json['source_data_criteria'], json['data_criteria'] = set_data_criteria_code_list_ids(json, cql_artifacts) # Create CQL Measure measure = Measures::Loader.load_hqmf_cql_model_json(json, user, cql_artifacts[:all_value_set_oids], main_cql_library, cql_artifacts[:cql_definition_dependency_structure], cql_artifacts[:elms], cql_artifacts[:elm_annotations], files[:CQL], nil, cql_artifacts[:value_set_oid_version_objects]) measure['episode_of_care'] = measure_details['episode_of_care'] measure['type'] = measure_details['type'] # Create, associate and save the measure package. measure.package = CqlMeasurePackage.new(file: BSON::Binary.new(zip_file.read())) measure.package.save measure end |
.mat_cql_export?(zip_file) ⇒ Boolean
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
# File 'lib/measures/loading/cql_loader.rb', line 5 def self.mat_cql_export?(zip_file) # Open the zip file and iterate over each of the files. Zip::ZipFile.open(zip_file.path) do |zip_file| # Check for CQL, HQMF, ELM and Human Readable cql_entry = zip_file.glob(File.join('**','**.cql')).select {|x| !x.name.starts_with?('__MACOSX') }.first elm_json = zip_file.glob(File.join('**','**.json')).select {|x| !x.name.starts_with?('__MACOSX') }.first human_readable_entry = zip_file.glob(File.join('**','**.html')).select { |x| !x.name.starts_with?('__MACOSX') }.first # Grab all xml files in the zip. zip_xml_files = zip_file.glob(File.join('**','**.xml')).select {|x| !x.name.starts_with?('__MACOSX') } if zip_xml_files.count > 0 xml_files_hash = extract_xml_files(zip_file, zip_xml_files) !cql_entry.nil? && !elm_json.nil? && !human_readable_entry.nil? && !xml_files_hash[:HQMF_XML].nil? && !xml_files_hash[:ELM_XML].nil? else false end end end |
.modify_value_set_versions(elms) ⇒ Object
Adjusting value set version data. If version is profile, set the version to nil
215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 |
# File 'lib/measures/loading/cql_loader.rb', line 215 def self.modify_value_set_versions(elms) elms.each do |elm| if elm['library']['valueSets'] && elm['library']['valueSets']['def'] elm['library']['valueSets']['def'].each do |value_set| # If value set has a version and it starts with 'urn:hl7:profile:' then set to nil if value_set['version'] && value_set['version'].include?('urn:hl7:profile:') value_set['profile'] = URI.decode(value_set['version'].split('urn:hl7:profile:').last) value_set['version'] = nil # If value has a version and it starts with 'urn:hl7:version:' then strip that and keep the actual version value. elsif value_set['version'] && value_set['version'].include?('urn:hl7:version:') value_set['version'] = URI.decode(value_set['version'].split('urn:hl7:version:').last) end end end end end |
.process_cql(files, main_cql_library, user, vsac_options, vsac_ticket_granting_ticket, measure_id = nil) ⇒ Object
Manages all of the CQL processing that is not related to the HQMF.
95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 |
# File 'lib/measures/loading/cql_loader.rb', line 95 def self.process_cql(files, main_cql_library, user, , vsac_ticket_granting_ticket, measure_id=nil) elm_strings = files[:ELM_JSON] # Removes 'urn:oid:' from ELM for Bonnie and Parse the JSON elm_strings.each { |elm_string| elm_string.gsub! 'urn:oid:', '' } elms = elm_strings.map{ |elm| JSON.parse(elm, :max_nesting=>1000)} elm_annotations = parse_elm_annotations(files[:ELM_XML]) # Hash of define statements to which define statements they use. cql_definition_dependency_structure = populate_cql_definition_dependency_structure(main_cql_library, elms) # Go back for the library statements cql_definition_dependency_structure = populate_used_library_dependencies(cql_definition_dependency_structure, main_cql_library, elms) # Add unused libraries to structure and set the value to empty hash cql_definition_dependency_structure = populate_unused_included_libraries(cql_definition_dependency_structure, elms) # fix up statement names in cql_statement_dependencies to not use periods <<WRAP 1>> # this is matched with an UNWRAP in MeasuresController in the bonnie project Measures::MongoHashKeyWrapper::wrapKeys cql_definition_dependency_structure # Depening on the value of the value set version, change it to null, strip out a substring or leave it alone. modify_value_set_versions(elms) # Grab the value sets from the elm elm_value_sets = [] elms.each do | elm | # Confirm the library has value sets if elm['library'] && elm['library']['valueSets'] && elm['library']['valueSets']['def'] elm['library']['valueSets']['def'].each do |value_set| elm_value_sets << {oid: value_set['id'], version: value_set['version'], profile: value_set['profile']} end end end # Get Value Sets value_set_models = [] # Only load value sets from VSAC if there is a ticket_granting_ticket. if !vsac_ticket_granting_ticket.nil? value_set_models = Measures::ValueSetLoader.load_value_sets_from_vsac(elm_value_sets, , vsac_ticket_granting_ticket, user, measure_id) else # No vsac credentials were provided grab the valueset and valueset versions from the 'value_set_oid_version_object' on the existing measure db_measure = CqlMeasure.by_user(user).where(hqmf_set_id: measure_id).first unless db_measure.nil? measure_value_set_version_map = db_measure.value_set_oid_version_objects measure_value_set_version_map.each do |value_set| query_params = {user_id: user.id, oid: value_set['oid'], version: value_set['version']} value_set = HealthDataStandards::SVS::ValueSet.where(query_params).first() if value_set value_set_models << value_set else raise MeasureLoadingException.new "Value Set not found in database: #{query_params}" end end end end # Get code systems and codes for all value sets in the elm. all_codes_and_code_names = HQMF2JS::Generator::CodesToJson.from_value_sets(value_set_models) # Replace code system oids with friendly names # TODO: preferred solution would be to continue using OIDs in the ELM and enable Bonnie to supply those OIDs # to the calculation engine in patient data and value sets. replace_codesystem_oids_with_names(elms) # Generate single reference code objects and a complete list of code systems and codes for the measure. single_code_references, all_codes_and_code_names = generate_single_code_references(elms, all_codes_and_code_names, user) # Add our new fake oids to measure value sets. all_value_set_oids = value_set_models.collect{|vs| vs.oid} single_code_references.each do |single_code| # Only add unique Direct Reference Codes unless all_value_set_oids.include?(single_code[:guid]) all_value_set_oids << single_code[:guid] end end # Add a list of value set oids and their versions value_set_oid_version_objects = get_value_set_oid_version_objects(value_set_models, single_code_references) cql_artifacts = {:elms => elms, :elm_annotations => elm_annotations, :cql_definition_dependency_structure => cql_definition_dependency_structure, :all_value_set_oids => all_value_set_oids, :value_set_oid_version_objects => value_set_oid_version_objects, :single_code_references => single_code_references, :all_codes_and_code_names => all_codes_and_code_names} end |
.replace_codesystem_oids_with_names(elms) ⇒ Object
Replace all the code system ids that are oids with the friendly name of the code system TODO: preferred solution would be to continue using OIDs in the ELM and enable Bonnie to supply those OIDs
to the calculation engine in patient data and value sets.
201 202 203 204 205 206 207 208 209 210 211 212 |
# File 'lib/measures/loading/cql_loader.rb', line 201 def self.replace_codesystem_oids_with_names(elms) elms.each do |elm| # Only do replacement if there are any code systems in this library. if elm['library'].has_key?('codeSystems') elm['library']['codeSystems']['def'].each do |code_system| code_name = HealthDataStandards::Util::CodeSystemHelper.code_system_for(code_system['id']) # if the helper returns "Unknown" then keep what was there code_system['id'] = code_name unless code_name == "Unknown" end end end end |
.set_data_criteria_code_list_ids(json, cql_artifacts) ⇒ Object
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
# File 'lib/measures/loading/cql_loader.rb', line 61 def self.set_data_criteria_code_list_ids(json, cql_artifacts) # Loop over data criteria to search for data criteria that is using a single reference code. # Once found set the Data Criteria's 'code_list_id' to our fake oid. Do the same for source data criteria. json['data_criteria'].each do |data_criteria_name, data_criteria| unless data_criteria['code_list_id'] if data_criteria['inline_code_list'] # Check to see if inline_code_list contains the correct code_system and code for a direct reference code. data_criteria['inline_code_list'].each do |code_system, code_list| # Loop over all single code reference objects. cql_artifacts[:single_code_references].each do |single_code_object| # If Data Criteria contains a matching code system, check if the correct code exists in the data critera values. # If both values match, set the Data Criteria's 'code_list_id' to the single_code_object_guid. if code_system == single_code_object[:code_system_name] && code_list.include?(single_code_object[:code]) data_criteria['code_list_id'] = single_code_object[:guid] # Modify the matching source data criteria json['source_data_criteria'][data_criteria_name + "_source"]['code_list_id'] = single_code_object[:guid] end end end end end end return json['source_data_criteria'], json['data_criteria'] end |